When Shift Happens - Confounding Is to Blame
Authors: Abbavaram Gowtham Reddy
Celia Rubio-Madrigal
Rebekka Burkholz
Krikamol Muandet
Venue: ICLR 2026 Poster (2026)
Keywords: Explainability
OOD Generalization
Confounding shifts
Primary area: interpretability and explainable AI
TL;DR: Hidden confounding can make all-feature models outperform causal ones under distribution shifts. Learning environment-specific patterns and using confounder proxies improves OOD robustness beyond standard causal approaches.

Abstract: Distribution shifts introduce uncertainty that undermines the robustness and generalization capabilities of machine learning models. While conventional wisdom suggests that learning causal-invariant representations enhances robustness to such shifts, recent empirical studies present a counterintuitive finding: (i) empirical risk minimization (ERM) can rival or even outperform state-of-the-art out-of-distribution (OOD) generalization methods, and (ii) OOD generalization performance improves when all available covariates, including non-causal ones, are utilized. We present theoretical and empirical explanations that attribute this phenomenon to hidden confounding. Shifts in hidden confounding induce changes in data distributions that violate assumptions commonly made by existing approaches. Under such conditions, we prove that generalization requires learning environment-specific relationships, rather than relying solely on invariant ones. Furthermore, we explain why models augmented with non-causal but informative covariates can mitigate the challenges posed by hidden confounding shifts. These findings offer new theoretical insights and practical guidance, serving as a roadmap for future research on OOD generalization and principled covariate-selection strategies.

Published as a conference paper at ICLR 2026
WHEN SHIFT HAPPENS CONFOUNDING Is TO BLAME
Abbavaram Gowtham Reddy, Celia Rubio-Madrigal, Rebekka Burkholz, Krikamol Muandet" CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
ABSTRACT
Distribution shifts introduce uncertainty that undermines the robustness and gen- eralization capabilities of machine learning models. While conventional wisdom suggests that learning causal-invariant representations enhances robustness to such shifts, recent empirical studies present counterintuitive finding: (i) em- pirical risk minimization (ERM) can rival or even outperform state-of-the-art -istriiiittiittiionn (OOD) generalization methods, and (ii) OOD generalization performance improves when all available covariates, including non-causal ones, are utilized. We present theoretical and empirical explanations that attribute this phe- nomenon to hidden confounding. Shifts in hidden confounding induce changes in data distributions that violate assumptions commonly made by existing approaches. Under such conditions, we prove that generalization requires learning environment- specific relationships, rather than relying solely on invariant ones. Furthermore, explain why models augmented with non-causal but informative covariates mitigate the challenges posed by hidden confounding shifts. These findings offer new theoretical insights and practical guidance, serving as roadmap for future research on OOD generalization and principled covariate-selection strategies.
INTRODUCTION
Generalization the ability to draw reliable conclusions about unseen data based observed data is central to numerous scientific fields. In medicine and the social sciences, it is embodied as external validity, ensuring findings from one population are applicable a different population (Campbell & Stanley, 2015); in ecology, it supports spuce-for-time substitutions, where spatial variation is used as proxy for temporal change to infer long-term ecological patterns (Pickett, 1989); and in engineering, it drives robust control, where models must maintain performance in the presence of unmodeled disturbances (Khalil al., 1996). In recent years, machine learning has become a powerful tool for learning generalizable models (Zhou et al., 2022; Wang et al., 2022; Liu et al., 2021).
To generalize well, machine learning models must robust distribution shifts between training and test data. For example, model trained to predict food stamp recipiency based on household attributes in one region should capable of adapting and performing well when deployed in another region. model is said to achieve nol-of-distribution (OOD) generulization when it maintains its performance on both in-distribution (ID) data (from which training data are sampled) and OOD test data, Over the last decade, various families of methods such domain generalization (Muandet al., 2013; Heinze- Deml et al., 2018; Zhao et al., 2022: Singh et al., 2024), domain adaptation (Zhao et al., 2019; Long et al., 2018; Xu al., 2020; Sun & Saenko, 20) and robust learning (Levy al., 2020; Sagawa et al., 2019) have been proposed to achieve provable OOD generalization under specific assumptions. However, under careful model selection, models based on standard empirical risk minimization (ERM) (Vapnik, 1999) often achieve competitive OOD generalization performance across a range of real-world applications (Gulrajani & Lopez-Paz, 2021; Krueger et al., 2021; Liu et al., 2023; Nastl Hardt, 2024; Rosenfeld et al., 2022; Vedantam et al., 2021). Moreover, a recent empirical study (Nastl & Hardt, 2024) on 16 real-world tabular datasets has concluded that incorporating all available covariates when predicting the outcome, regardless of whether they directly affect the outcome, can improve OOD generalization performance. These findings challenge prevailing assumptions in the field and motivate a deeper investigation into the mechanisms underlying OOD generalization.
Distribution shifts are commonly observed when data originate from different environ- ments (Schölkopf, 2022). For example, food stamp recipiency may differ across states, because each
'These authors share senior authorship.

---

Published as a conference paper at ICLR 2026
state operates under different eligibility rules, leading to distinct, environment- distributions. In such data, certain statistical relationships between covariates and the outcome may stay consistent across all environments, referred to as invariant relationships (Arjovsky et al., 2019). Identifying and learning such invariant relationships ensures OOD generalization (Arjovsky et al., 2019; Peters al., 2016; 6; Rojas-Carulla et al., 2018; Muandet al., 2013; Quinzan et al., 2024). A special kind of invariance is known as causal invariance, where the causal relationships in the data stay invariant across environments (Schölkopf, 2022: Peters et al., 2016; Arjovsky al., 2019). However, in many real-world settings, not all relevant covariates required for predicting an outcome are observed, due to limitations in data collection, privacy constraints, or measurement errors (Carroll et al., 2006; Louizos et al., 2017; Dwork et al., 2014; Little & Rubin, 2019). Their absence disrupts the invariant relationships needed for models to achieve generalization. This issue is further compounded when the unobserved variables are confounders that influence both the observed covariates and the outcome. In practice, such hidden confounders are pervasive, and shifts in their distributions correspond to distinct environments. Ignoring these shifts not only undermines generalization performance (Landeiro & Culotta, 2018; Alabdulmohsin et al., 2023; Tsai et al., 2024; Prashant et al., 2025), but can also lead to learning incorrect relationships between observed covariates and the outcome.
Despite its importance and the recent progress on achieving generalization under hidden confounding shifts (Alabdulmohsin et al., 2023; Tsai et al., 2024; Prashant et al., 2025), how hidden confounding shift affects generalization and how informative covariates help in generalization remains poorly understood. Our work aims to bridge this gap by providing theoretical and empirical explanations.
Our contributions are as follows:
We motivate the need for studying confounding shift in OOD generalization from a causal perspec- tive ($ 3) and explain why adding informative, non-causal covariates can improve performance.
While invariant representations alone, while sufficient, are challenging to achieve under the hidden confounding shift. We show that maximizing predictive information between model predictions and true outcomes demands explicitly learning environment-specine relationships. ($ 4.1, $ 4.2).
We demonstrate that variables informative of either outcome or hidden confounders help in im- proving predictive information between model predictions and true outcomes. This explains the importance of principled covariate selection in the presence of hidden confounding shift ($ 4.3).
Our experiments both real-world and synthetic datasets provide evidence that (i) hidden con- founding is prevalent in real-world tabular benchmark data, (ii) learning environment-specife relationships correlates positively with ID and OOD test accuracy, and (iii) incorporating informa- tive, non-causal covariates improves generalization ($ 5).
2 RELATED WORK
duttif-distribution (OOD) generalization encompasses various facets, with notable examples in- cluding domain generalization (Zhou et al., 2022; Wang et al., 2022), domain adaptation (Zhao et al., 2019; Sun & Saenko, 2016b; Long et al., 2018; Xu et al., 2020), robust learning (Levy et al., 2020; Sagawa et al., 2019), federated learning (Li et al., 2023), and OOD detection (Lee et al., 2018; Hendrycks & Gimpel, 2017). Domain adaptation assumes access to unlabeled data from the test set, whereas domain generalization relies on environment labels during training. Federated learning adopts a collaborative learning framework, tackling constraints such as communication efficiency and privacy when data originates from multiple environments. OOD detection focuses on identifying samples that differ significantly from the training distribution. A common goal of many of these methods is learn invariant relationships. However, recent work suggests that additional inductive biases beyond invariance are required for improved generalization (Lin et al., 2022; Schrouff et al., 2022; Ye et al., 2021). Ye et al. (2021) argue that invariance of features is necessary but not sufficient for generalization and discuss the importance of informativeness of features for generalization. We explain how informativeness plays a crucial role in generalization under hidden confounding shifts.
Proxy variable adjustment: When confounding variables are observed during training and unob- served at test time, Landeiro & Culotta (2018) propose adjusting for confounding shifts to improve classifier performance. Building on proxy-based adjustment methods for causal effect estima- tion (Miao et al., 2018; Kuroki & Pearl, 2014), recent domain adaptation methods rephrase the problem of unknown distribution shifts causal effect identification problem (Alabdul
2

---

Published as a conference paper at ICLR 2026
et al., 2023; Tsai al., 2024). Recently, OOD generalization under hidden confounding shift has been considered under the assumption of overlapping confounder support (Prashant et al., 2025). In contrast to these approaches, we explain how proxy variables enhance OOD generalization.
All variable models vs causal models. Recently, Nastl & Hardt (2024) introduced a benchmark study where covariates categorized into four groups: causal (conservatively chosen), arguably causal, anti-causal, and other spurious covariates. They show that across 16 benchmark datasets, models using all covariates Pareto-dominate those using only causal arguably causal subsets on both ID and OOD data. However, there is limited theoretical work explaining these results. We present scenarios and arguments to explain their experimental findings. For linear causal models, anchor regression (Rothenhäusler et al., 2021) introduces a framework that balances between two estimation paradigms: models that include all observed covariates and models that focus solely on causal covariates. We aim to explain the impact of adding more covariates that are not necessarily causal under hidden confounding shift. Eastwood et al. (2023) show that unstable covariates can boost performance when they carry information about the label, provided they are conditionally independent the stable covariates given the label. They propose to adjust the distribution shift by looking at the test domain without labels. However, when applied to medical real-world dataset not constructed for this particular problem (Bandi et al., 2019), ERM still remains competitive with their method, in line with the findings of Nastl & Hardt (2024). This reflects the broader insight that, under well-specified covariate shifts, maximum likelihood estimation (MIH) achieves minimax optimality for OOD generalization (Ge al., 2024). Yet, real-world settings are rarely well-specified due to hidden confounding shift, which is the main focus of this work.
MANIFESTATIONS OF HIDDEN CONFOUNDING SHIFT
We now provide background on hidden confounding shift and motivate the need address it.
Types of distribution shifts. For covari- ates X and target Y, one may observe sev- eral distribution shifts between two environ- ments e and e' as shown in Table 1. These distribution shifts usually result from shift in the distribution (U) of an unobserved covariate U that causes either X or Y both (see Figure 1). The shifts in PUU) lead shifts in observed distributions involving only X and Y. For instance, we can write: (XX) = Σ (X U). Thus, when (U))) + P (U) and U + X, we observe P
Table 1: A summary of different distribution shifts.
Type of Shift
Mathematical Expression
Label
PO(Y) * Pd (Y)
Covariate
P*(X) / P (X)
Conditional Covariate
P=(X Y) / (X
Concept
P"(Y X) PY (Y X)
Dataset
P Y) P (X,Y)
X
+ Ipl' X). Similar arguments can be made about
the distribution shifts of PYY), P(X| Y), P(Y
X),
P P(
Existing methods for OOD generalization assume certain causal relationships among U,X.Y Y that guarantee specific invariances. For instance, when U + +Y X (Figure (a)), since P(X, Y) = (P | Y), a shift IPe (X, + pe' (X. Y) is observed due label shift, i.e., (Y) + IP (Y), but conditional covariate distribution stays
U
U
X
X
(a) Label shift
(b) Covariate shift
(c) Confounding shift
Figure 1: Causal graphs underlying distribution shifts.
invariant i.e., P Y)= = ppe (X Y) because P(X[Y U) = P(X(Y) (Wu et al., 2021: Tachet des Combes al., 2020; Garg al., 2020; Alexandari et al., 2020). Similarly, when U X Y (Figure (b)), shift (X, Y) A pe' (X, is observed due to covariate shift, i.e., IP (X) X pe' (X),
but the conditional distribution P(Y X) stays invariant i.e., (P X) Pp PY X) (Schneider et al., 2020; Sugiyama & Kawanabe, 2012; Gretton et al., 2009). These invariances may not hold in many scenarios because U usually causes both and Y (Figure (c)) (Liu et al., 2023; Alabdulmohsin et al., 2023; Landeiro & Culotta, 2018; Tsai et al., 2024; Prashant et al., 2025; Reddy et al., 2022: Reddy & N Balasubramanian, 2024). In this case, when P(U) shifts between environments, label shift, covariate shift, conditional covariate shift, and concept shift can all occur simultaneously. Nevertheless, confounding shift induces invariance: Pe Px(X Y U)
Tp' (X,Y Y
3

---

Published as a conference paper at ICLR 2026
Structural Equations:
La Hoar 4
LA Uaie &-F
Lik, Uske X.l LA Unie 3. (Drecte)
[Hidden Confounderi Affee 0.5%) He E (-2,
Thain data
Test data
Me (0. for 0OD
Predictions train data
(Input) X 0.750 + 1.55)
Pre dictiiss on test data
(Outcome) (Informative) x,- 0.6./ + c.3.0.3P)
0.5²)
We evaluate four linear regression (L.R.) models in an OOD setting characterized by hidden confounding shifts and minimal environment overlap (i.e., distant (c)). (i) A model trained solely on X learns an incorrect relationship with Y, illustrating Simpson's 's paradox. (ii) Using environment-speccinn summary statistics of X, denoted as E, recovers the correct relationship but remains limited in representation power. (iii Using an informative covariate X, for U improves OOD generalization. (iv) The oracle model is trained on X and U.
Figure 2:
U) referred to as stable confounding shift (Tsai et al., 2024; Alabdulmohsin et al., 2023). Since U is unobserved, this invariance is absent from the observed data.
Information-theoreti measures: Following Federici et al. (2021), we use mutual information to quantify distribution shifts. That is, we use 7 T(X; E), T(Y; E), I(X; E,Y), T(Y; E|X), (X,Y; F) to measure the shifts in P(X), PY()), P(X Y), PY(Y X). P(X,Y Y) respectively across the environments E. As discussed earlier, existing methods for OOD generalization often assume either T(X; E Y) = O through the label shift assumption or I(Y; E X) = O through the covariate shift assumption. However both of them may be nonzero under hidden confounding shifts.
To illustrate the challenges posed by hidden confounding shift OOD generalization, we present the experiment shown in Figure 2. The outcome Y depends on X and a hidden confounder U. Enviromment--pecifn shifts in the mean of U, denoted by Pes induce distinct environments. A linear regression model trained solely on observed covariate X infers an incorrect relationship between X and Y. as shown in the first subplot. This exemplifies the Simpson's 's paradox (Simpson, 2018), where Y increases with X within each environment, yet the model captures the opposite trend. We observe similar behavior from the models designed for OOD generalization, such as IRM (Arjovsky et al., 2019), VREX (Krueger et al., 2021), and Group DRO (Sagawa et al., 2019). In such scenarios, we theoretically show in 4 that the optimal strategy for generalization involves learning environment-specifee relationships, which enables recovery of the correct relationship between X and Y. This is illustrated in the second subplot, where catrromment-specifie summary statistics of X, such as mean, standard deviation, and quantiles, help uncover the true X-Y relationship. This mirrors the backdoor adjustment criteria in causal effect estimation (Pearl, 2009), where anviroomenttspeccin confounders. However, as shown in the second subplot, environment-specific information encoded in observed covariates alone may be insufficient due to the limited representational capacity. However, as we show in $ 4.3, additional informative covariates serving as proxies for unobserved confounders can further improve generalization performance, as shown in the third subplot. For comparison, we include an oracle model trained on X and U, which achieves the best- best-possible mean squared error.
Challenges with hidden confounding shift:
information acts as a proxy for adjusting for
IMPACT OF HIDDEN CONFOUNDING SHIFTS ON OOD GENERALIZATION
In this section, theoretically explain how hidden confounding shifts impact various aspects of OOD generalization. Let ý (f ( (X) be the predicted label for input X, where f is classifier and Φ is a feature extractor or transformation function. H (X) = E [log(P(X))] denotes entropy F(X,Y)
and V) Fx,y [log P[X]F(Y)] denotes mutual information. The risk of the predictor (ʃ o b) in an environment e is defined as: Rf o Φ) = E(x.y>~Xx, [e(( o (X), Y)] where E(.,) is a loss function. The goal in OOD generalization is to learn a predictor (f o Φ) that performs well on both ID and OOD data. To this end, various objectives have been considered the literature. Robust iilimization-basee methods (Ben-Tal & Nemirovski, 2002; Sinha et al., 2017: Sagawa et al., 2019) aim to minimize the worst risk across all training environments: Rob o Φ) = max Rf o o) where Err denotes the set of training environments. Invariant risk minimization (IRM) (Arjovsky et al., 2019) aims to minimize Eeee, R' o o) with the constraint that f is a simultancously optimal classifier across all environments. Empirical risk minimization (ERM) based methods simply
4

---

Published as a conference paper at ICLR 2026
pool data from all training environments and minimize the empirical risk on the pooled data (Arjovsky et al., 2019; Krueger et al., 2021). That is, ERM minimizes RERM (f の) = Leee, Rn Φ).
To understand how hidden confounding shift impacts traditional objective functions, explore different aspects of maximizing predictive information: I(;; Y) where y (f (X). I(Y; Y) quantifies how informative the prediction Y' is about the true label Y. making it a natural and mean- ingful objective for many tasks. We start by defining two key properties of 6 (no))iiformaaivveess and invariance-bott of which are crucial for OOD generalization (Ye et al., 2021).
Definition 4.1 (Informativeness and Conditional Informativeness), The informativeness of fea- tures (XX), for predicting Y is defined as IN)( Y) 1(((X); Y. The conditional informa- tiveness of features (XX) for predicting Y conditioned on environment variable is defined as (INF E) = (()()))
Conditional informativeness measures the information ((X) provides about Y within each envi- ronment. Minimizing CINF implies information loss, and maximizing CINF can be undesirable in applications such as algorithmic faimess, as ((X) may exploit sensitive or biased information within environments, leading unfair predictions. While such biased representations can be avoided by minimizing I (x( E), it may reduce predictive performance, as we show in $ 4.2. Considering the setting where no sensitive information is associated with E, we adopt the perspective that maximizing CINF can improve generalization performance, a view we follow and substantiate in $ 4.1.
Definition 4.2 (Variation and Invariance). For a given label Y, the variation in the features o(X) across environments E is defined as VAR(o(X), Y, E) [(x();; E\Y). For a given label Y, the invariance of the features <(X) across environments E is defined as the negative of the variation i.e. INV( Y, E) VAR(Ø(X),Y,
II ((x E
Invariance quantifies how consistent the representation 6(X) is across different environments for a given label Y. Minimizing invariance can lead to overfitting by preserving environment--seeciti information in ()X), whereas maximizing invariance helps eliminate environment-specifcc dependen- cies, promoting invariant learming. Extending the information -thooretteecc measures from 3, we use I (o(()) E) and [Y; E (X)) to quantify feature shift and concept shift respectively, Depending on the context, we use the term concept shift to denote either T(Y; E X) or I(Y: E <(X)).
4.1 A GENERAL DECOMPOSITION OF PREDICTIVE INFORMATION
We begin with a few causal-graph preliminaries. causal graph G consists of nodes representing random variables, and directed edges indicating direct causal influences between nodes. A path between two nodes X, and Xj is a sequence of unique nodes connected by edges. A directed path from X; to X2 with
j one where all arrows point toward Xj. i.e., X; X1+1 X,-1 + X,. In such a directed path, X; is called the parent of X,+1, X is the child of X,, X; an ancestor of X,, and X, a descendant of X,. Paths decompose into three fundamental structures: a chain X; + X, + X a fork X; + X, - Xx., and collider X; + X, + X₁. In both chains and forks, X; and Xx are marginally dependent yet become conditionally independent upon conditioning on the intermediate node Xj. i.e. X; X₂ and X; II Xx Xj. In collider, X; and X₃ are marginally independent but become conditionally dependent when conditioning on X, or any of its descendants, i.c. X; JL X and X, L Xx Xz. 3. A path between X; and X₁ is said to be blocked by a conditioning set S if and only if either (i) the path contains chain or fork whose middle node lies in S, or (ii) it contains collider such that neither the collider nor any of its descendants belongs to S. all paths from X; to X₁ are blocked by S, then X; IL X₂ S. A path is open if is not blocked.
Now consider the predictive information [(Y; ł), where the predictions y = (ʃ (X) are based on a learned representations of (). ). We model the under- lying causal relationships among X, Y, U, ((X). y and E shown in Figure 3. Here, U is hidden con- founding variable. Given either or o(X). Y is re- dundant for reasoning about Y. That is, Y LYYI X and Y .L Y ((X). E denotes an environment vari- able that captures shifts in IU.).. That is, between
E
U
(X)
X
Figure 3: Bi-directed arrow between X and Y indicate that some covariates cause Y. and some are caused by Y.
5

---

Published as a conference paper at ICLR 2026
any two environments, there is shift in P(U). We first present more general decomposition of predictive information without explicit consideration of how U inffuences X, Y. A formal treatment U's inftuence on X and Y is presented in 4.2.
Theorem 4.1. For covariate vector X. label Y, with causal structure X ++ Y, i.e., some covariates cause Y and some covariates are caused by Y, environment variable E, a feature extructor 4,, and prediction Y, the predictive information I(Y;)) decomposed as follows:
Variation
Label shif
Fenture shilt
Concept shift
Cond. informatis
Residual
((
T(Y: F)
[((X);
T(Y; El((x))
I(((X); Y
2
2
(I)
where (()) is the residual information in o(X) () for inferring Y that is not captured by the prediction Y. The decomposition above also holds when 6(X) is replaced with X.
Proofs are presented in Appendix $ A. Existing methods can be viewed as methods that explicitly minimize maximize certain terms in Equation 1. For instance, IRM (Arjovsky et al., 2019) enforces that a fixed classifier remains the same across different environments. This is directly related to maximizing - -T(((X) (X); E because if 6(X) were to carry extra environment-spern
information conditional on Y, the fixed classifier would no longer be optimal in all environments. DANN (Ganin al., 2016) and the independence criterion for fair classification (Federici al., 2021) aims to minimize environment-specifee information in (X) by minimizing (x();; E). In the context of domain adaptation, certain properties of features are learned invariant to domains (Sun & Saenko, 2016b; Xu et al., 2020). While minimizing [ X); E) may be good for fair classification, it may degrade predictive performance (Johansson et al., 2019; Federici et al., 2021). This can be seen from Equation I, where minimizing (ox(;;; E)) can reduce I(Y: Y). In CDAN (Long et al., 2018), the objective is to obtain ()))),, Y) = F (6(X).Y). By enforcing this equality, E becomes independent of the pair (e(X), Y). Consequently, IP(Y ((X), F)) P(Y ((X) = T(Y; F <(X)) = O. Thus, although CDAN does not explicitly include T(Y; E x(X)) ) in its loss, its distiddisttibution alignment objective effectively drives T(Y; F ((X)) zero. While T(Y; E) is constant, there exist methods that are robust to label shift (Sagawa et al., 2019).
Recall that Equation is derived without explicitly considering how U influences X, Y. Knowing how U influences X and Y can further guide better understanding of the terms in Equation 1. That is, if + Y.U A X, Y + X. then I [X:: E YY 0, suggesting that minimizing I ((x(X); E Y) is principled objective. Similarly, if U A YUU + X. X Y. then EY X) = O, thereby motivating the minimization of T(Y;
(X)). However, if U - Y and U - X, the interaction
between the terms in Equation becomes non-trivial, and it remains unclear what constitutes an ideal strategy for addressing them. We answer this question in the next section.
4.2 PREDICTIVE INFORMATION UNDER HIDDEN CONFOUNDING SHIFT
To understand how hidden confounding variable U that cause both X and Y, impact the predictive information, we consider two special cases of the relationship between X and Y: (i) X + and (ii) + X. When X + Y, we obtain the inequalities in 2 because conditioning on opens the path (X) + + Y + U + E' from o(X) to E' at the collider node Y which results in additional information flow from (X) to E via Y. In contrast, conditioning on <(X) partially blocks the information flow from Y to E at the node X as long as o(X) encodes some information about X.
Variation
Feature
Label shift
Concept shift
(i) (((();; ElY) 2 T((();; E)
(ii) T(Y; E) 2 T(Y; E()()))
(2)
Similarly, when Y + X, we obtain the incqualities in 3 because conditioning on Y blocks the information flow from ((X) E the node Y and conditioning on o o(X) opens the path + X + U + E from Y to E at the node X because Φ(X) is the child of X and conditioning on the child of a collider opens a path via that collider (Pearl, 2009) (recall the preliminaries in $ 4.1).
Feature shift
Variation
Concept
Label shift
(i) [(x(X;; E) N E()
(it) I(Y: (X)) 2 EY:
(3)
Using the inequalities in 2, 3, we refine the predictive information decomposition as follows.

---

Published as a conference paper at ICLR 2026
Theorem 4.2. For covariate vector X. label Y, environment variable E, feature extractor ø. the prediction Y, and unobserved confounding variable U that cause both X and Y, and if either (i) X Y or (ii) + X, the predictive information I I(Y;Y) Y) can be decomposed as follows:
Cond. Informativeness
Residual
I( ((x(: T YIE) - I(6(X);))))
In Theorem 4.2, we consider the two cases: X + Y and Y X separately. However, the causal relationship X + Y is more prevalent in real-world tabular prediction tasks. For instance, 11 out of 16 datasets considered in recent benchmark studies (Nastl & Hardt, 2024; Liu et al., 2023; Gardner et al., 2023) follow the causal structure X - Y. In the remaining 5 out of 16 datasets, the number of covariates that are caused by Y much smaller than the number of covariates that cause Y. We present the key understanding from Theorem 4.2 below.
Compared to the decomposition in 1, the decomposition in 4 (which is obtained for the special cases: (i) U + X, U Y,X - Y, (ii) U - X, - Y, Y - X) is free from the terms: variation, label shift, feature shifi, and concept shift. That is, under hidden confounding shift, when either X + Y or Y + X, the predictive information is equal to the difference: conditional informativeness residual.
Remarks on conditional informativeness: In very recent work, Prashant et al. (2025) proposed mixture of experts (MoE) model for OOD generalization under hidden confounding shift, where each expert corresponds to specific hidden confounder assignment. That is, each expert focuses on maximizing the performance within the environment corresponding to a particular value of the hidden confounder. Thus, learning MoE models aligns with the goal of maximizing the conditional informativeness I (x(();: E). Equation 4 provides theoretical justification for the use MoE-type models. Beyond supporting such models, the Equation also highlights the need for methods that operate under more general confounding shift settings. For instance, the method proposed by Prashant al. (2025) assumes overlapping confounding support i.e., supp(IP(UT (()) supp(P(U((.r)) and the proxy variables to be discrete-valued.
The results in Figure 2 illustrate that, in a simple setting without confounding overlap and with continuous- proxy, a linear regression model can successfully recover the correct causal relationship between observed covariates and the target across environments when provided with environment-speccin information. We theoretically explain in $ 4.3 why environment-specific informative covariates help improve generalization performance.
Remarks on residual: The residual term I(o(X;; Y) quantifies how much additional infor-
mation o(X) provides about the true label Y beyond what is already contained in the prediction
Y. The residual term can be expressed as: (x();; Y) H( () H!Y ((X), Y). The
conditional entropy H(Y Y) is related to the expected cross-entropy loss as: By y (f Y)) =
H f)+ + calibration error (Bröcker, 2009; Berta et al.. 2025). Here H(Y Y) is known as the refinement error that measures the model's 's ability to distinguish between classes.
4.3 IMPACT OF ADDITIONAL INFORMATIVE COVARIATES
In $ 4.2, we present the desiderata for generalization under hidden confounding shifts. In practice, we sometimes have access to proxies for the hidden confounding variable (Alabdulmohsin et al., 2023; Tsai al., 2024; Prashant et al., 2025), which can be leveraged to substitute for the hidden confounding variable. We now explain how adding such non-causal but informative covariates improves predictive information.
Definition 4.3 (Informative Covariates). A set of covariates X, that are not causally related to Y i.e., neither ancestors nor descendants Y are in X,, are said to be informative to Y if X and Y are not independent of each other given other causally related covariates X and F. i.e., Y 1 X, X, F.
Since + Y, any covariate that is informative to U, is also informative to and vice-versa. Causal graphs that show informative covariates X, have the structure U < X,;Y X, in addition
7

---

Published as a conference paper at ICLR 2026
Table 2: Quantifying distribution shifts. standad deviation is computed over 10 random subsets of 40,000 samples. Statistical significance against mean of zero is assessed via one-sample t-tests (a 0.05), confirming all measures are significantly different from zero (p- (p-value = 0).
Dataset
Conditional Covariate Shift T(X; FlY)
Label shift T(Y; E)
Covariate Shift T(X; F)
Concept shift T(Y; F|X)
Readmission
0.107 + 0.002
0.068 1 0.002
0.097 + 0.002
2.032 + 0.000
Food stamps
0.126 + 0.004
0.030 + 0.003
0.108 0.001
2.118 1 0.001
Income
0.168 0.002
0.075 0.003
0.147 + 0.001
2.059 0.002
Public coverage
0.231 + 0.002
+
0.222 + 0.002
1.945 +
Unemployment
0.117 + 0.001
0.019 + 0.002
0.114 0.002
2.010 + 0.003
Diabetes
0.032 + 0.002
0,048 + 0.001
0.022 + 0.002
2.132 = 0.001
Hypertension
0.090 = 0.002
0,183 + 0.003
0.037 0.001
1.883 = 0.004
ASSISTments
0.293 0.002
0.260 0.001
0.306 + 0.002
0.367 + 0.004
to the causal relationships shown in Figure 3. From the predictive information decomposition in Equation 1, maximize T(Y: y), it is required minimize the concept shift T(Y;; F d(X)). Liu et al. (2023) suggest based on their empirical analysis that collecting additional covariates X, such that (P XX Xy) is more stable across environments i.e., reducing concept shift, improves the OOD) test accuracy. We theoretically show that utilizing more informative variables helps maximizing predictive information by maximizing minimizing certain terms in Equation 1.
Proposition 4.1. Øf (+), Φ2(-) are invertible functions and if X2 are informative variables such that Y X, X, E, then we have the following inequalities:
(i) ((02(((XX))) X )); E) > (( (X);
(ii) [(
(X))(X); I(6: (X); (5)
(iit) I(Y; Bl (XXX))) < I(Y; (())( (iv) ((
Proposition 4.1 shows that adding informative covariates increases conditional informativeness (5.i) and feature shift (5.ii,, while reducing concept shift (5.iii). However, adding informative covariates also amplifies variation (5.iv), which may necessitate dedicated strategies control the variation. However, from our experimental results, we observe that the reduction of concept shift leads to significant improvement in the performance of models compared to minimizing variation.
EXPERIMENTAL RESULTS
We conduct experiments on both real-world and synthetic datasets to analyze: (i) the presence of hidden confounding shift in real-world data, (ii) how the components of the decomposition in affect OOD generalization under hidden confounding shifts, and (iii) the role of informative covariates in improving generalization. We consider eight real-world tabular benchmark datasets: Food stamps, Readmission, Income, Public coverage, Unemployment, Diabetes, Hypertension, and ASSISTments. These datasets and corresponding domain splits are adopted from TableShifi benchmark (Gardner al., 2023). We ID test and OOD test accuracies to measure the performance of models. We perform experiments on two ERM-based methods: XGBoost (XGB) (Chen & Guestrin, 2016) and multi-layer perceptron (MLP), two domain generalization methods: IRM (Arjovsky al., 2019), VREX (Krueger et al., 202 I), and one robust learning based method: Group DRO (GDRO) (Sagawa al., 2019).
We use Non-Parametrie Entropy Estimation toolbox (NPEET) for estimating mutual information (MI) via Traskov-Stögbauce Stögbauer-Grassberger (KSG) estimator (Kraskov et al., 2004: Steeg Galstyan, 2011; 2013). Following (Gardner et al., 2023), we evaluate mutual information using both in in-distribution (ID) test data (from training domains) and out-of-disti (OOD) test data (from test domains). Additional details of the experimental setup provided in Appendix B. The code and instructions to reproduce the results are provided the supplementary material.
Hidden confounding shift: Recall that hidden confounding shifts induce observable shifts in the distributions IP, P(X PYY)), and IPY X). Results in Table 2 show that label shift, covariate shift, conditional covariate shift, and concept shift are all present in real-world datasets, indicating the presence of hidden confounding shifts. For the qualitative understanding, we query GPT-4o (Achiam al., 2023) list potential hidden confounders for several benchmark datasets (Gardner et al., 2023;
8

---

Published as a conference paper at ICLR 2026
Table For all methods, sign consistency value is high for concept shift (CS). However, the sign consistency metric for conditional informativeness (CI) is crucial for generalization according to Theorem 4.2, and XGB excels at this. Res: residual, Var: variation, FS; feature shift, C: causal, AC: arguably causal, A: all.
Sign Consistency Metric (1)
Test Accuracy (f)
OOD-Test Accuracy (f)
Method
CI
Var
FS
CS
Res
AC
A
AC
A
XGB
0.92
0.56
0.35
0.79
0.15
78.91
81.96
82.31
64.35
72.80
72.90
MLP
0.65
0.60
0.21
0.85
0.42
77.56
78.86
80.16
62.03
67.92
66.93
GDRO
0.69
0.50
0.25
0.90
0.35
77.87
80.15
76.20
61.95
66.64
65.87
IRM
0.71
0.58
0.23
0.88
0.31
61.68
63.38
64.67
61.14
61.18
62.75
VREX
0.52
0.71
0.19
0.85
0.58
58.74
64.69
62.75
60.40
65.57
65.21
Nastl & Hardt, 2024). These results are presented in Appendix E. For instance, in the food stamps dataset, unmeasured factors such as economic policies specific to each state may influence both household income and food stamp recipiency. GPT-4o returned results are solely meant for semantic insight and are not implicitly or explicitly used in other experiments.
Conditional informativeness vs. accuracy: From Theorem 4.2, under hidden confounding shift, maximizing the difference conditional informativenesss residual is essential for maximizing predic- tive information. We observe that the difference conditional informativeness - residual is positively correlated with ID test and OOD test accuracies, as measured using the Spearman rank correlation coefficient (p) between accuracy and the difference: conditional informativeness residual. For the results on five methods and eight datasets, obtain p 0.93 with respect to ID test accuracy, and = 0.80 with respect to OOD test accuracy. As shown in the dataset-wise results in Appendix C, we observe that sum variation / label shift / 2 feature shift / 2 concept shift 2 is closer zero under due to hidden confounding shift (Theorem 4.2).
Informative covariates accuracy: We now study how inclusion of informative covariates helps in achieving better OOD generalization. To this end, we adopt the covariate partitioning from (Nastl & Hardt, 2024), where all available covariates are grouped into three nested covariate subsets: causal covariates S arguably causal covariates C all covariates (see Appendix $ B). We evaluate how the terms in the decomposition in affect ID and OOD performance when going from one covariate subset to another covariate subset. To this end, we use what we call the sign consistency metric, which works as follows. When we move from one covariate set to a larger covariate set, for any term in with a positive coefficient, such as conditional informativeness, we count how often its value rises when accuracy improves. For negatively weighted terms, such as concept shift, we count how often its value decreases when accuracy improves. The metric is the fraction of observations where a term's 's change aligns with its beneficial direction, thus capturing how reliably it contributes to better generalization. See Appendix $ B for the formal definition. Table shows that both conditional informativeness and concept shift exhibit high sign consistency with accuracy gains. While reducing concept shift is crucial, XGB further enhances conditional informativeness compared to other methods, yielding additional performance improvements. Dataset-specific results are presented Appendix $ C.
Note that in real-world datasets, it is often challenging to verify whether all available covariates include every relevant sufficiently informative variable. To further test our hypothesis that informa- tive covariates enhance generalization, we conduct experiments structure: + X, Y, X Y. and U + X,, where X, denotes informative covariates caused by the hidden confounder U. We observe that including X, leads to improvements in conditional informativeness and feature shift, while reducing concept shift, as shown in Figure 4. The synthetic data generation process and additional results are provided in Appendix D.
synthetic data with a known causal
6 DISCUSSION, LIMITATIONS, AND FUTURE WORK
We decompose the predictive information between model outputs and true outcomes to explain the factors limiting OOD generalization under the hidden confounding shift. We explain why simple methods such as XGBoost work better than invariance-bart OOD generalization methods. We also explain how the addition of non-causal informative covariates helps improve the generalization performance of any method. Our goal is to explain these phenomena but not to provide any solution

---

Published as a conference paper at ICLR 2026
Mean Squared Error
Cond. informatieeness & Varlation
Feature Shift & Concept Shift
Train
Test
Train
Test
Train
Best
10
0.0
20
10
[X]
LeserRegrsssinn
X68
Cond. Informetvvness
Figure 4: Adding more proxy variables X, of U that are informative to Y helps in reducing MSE, increasing conditional informativeness and feature shift while reducing concept shift.
to the problems posed by the hidden confounding shift. By highlighting the inevitability of hidden confounding and the need to address it directly, our work lays a foundation for future work: (i) understanding the role of environments in maximizing conditional informativeness (ii) quantifying the cost-accuracy trade-off of acquiring non-causal informative covariates, (iii) handling entangled shifts without relying on untestable proxy assumptions. and (iv) modeling ambiguity from unobserved confounders to inspire new OOD-robust paradigms.
ETHICS AND REPRODUCIBILITY STATEMENT
All authors have read and agree to adhere to the ICLR Code of Ethics. This work complies with all ethical guidelines outlined therein. Proofs of the theoretical results are presented in the appendix. The code and instructions to reproduce the results are available at https: /ttp::///ttie /tt///gttth//tt com/ gautam0707 202007/Cooo htt///ooooooinnnn _eto__is_________o___ La__tto_________
ACKNOWLEDGMENTS
The authors thank Anurag Singh and Rajeev Verma for proofreading the paper, which improved its readability. Moreover, authors gratefully acknowledge the Gauss Centre for Supercomputing e. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). We also gratefully acknowledge funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116395 SPARSE-ML.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference machine learning, pp. 69. PMLR, 2018.
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain- adversarial neural networks. arXiv preprint 4rXiv:1412.44440 2014.
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D' Amour, Arthur Gretton, Sanmi Koyejo, Matt J. Kusner, Stephen Pfohl, Olawale Salaudeen, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 9637-9661. PMLR, 25-27 Apr 2023.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat label shift adaptation. In International Conference on Machine Learning, pp. 222-232. PMLR, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Pazz Invariant risk minimization. arXiv preprint arXiv: (el::0:009000000 2019.
10

---

Published as a conference paper at ICLR 2026
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, Quanzheng Li, Farhad Ghazvinian Zanjani, Svitlana Zinger, Keisuke Fukuta, Daisuke Komura, Vlado Ovtcharov, Shenghua Cheng, Shaoqun Zeng, Jeppe Thagaard, Anders B Dahl, Huangjing Lin, Hao Chen, Ludwig Jacobsson, Martin Hedlund, Melih Cetin, Eren Halici, Hunter Jackson, Richard Chen, Fabian Both, Jorg Franke, Heidi Kusters-Vandeveld Vandevelde, Willem Vreuls, Peter Bult, Bram van Ginneken, Jeroen van der Laak, and Geert Litjens. From detection of individual metas- tases to classification of lymph node status at the patient level: The CAMELYONI challenge. IEEE Trans Med Imaging, 38, 2019.
Aharon Ben- Tal and Arkadi Nemirovski. Robust opiimizationn-mchhooooogg and applications. Math- ematical programming. 92:453 480, 2002.
Eugene Berta, David Holzmüller, Michael Jordan, and Francis Bach. Rethinking early stopping: Refine, then calibrate. arXiv preprint urXiv:2501.19195 2025.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society: A journal of the atmospheric sciences, applied meteorology and physical oceanography, 135:1512-1519 2009.
Donald Campbell and Julian C Stanley, Experimental and quasi-experimental designs for research. Ravenio books, 2015.
Raymond Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. Measurement error in nonlinear models: modern perspective. Chapman and Hall/CRC, 2006.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant causal mechanisms through distribution matching. arXiv preprint arXiv:2206.. 1646, 2022.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations differential privacy. Foundations and Trends in Theoretical Computer Science, 9:211-407, 2014.
Cian Eastwood, Shashank Singh, Andrei L Nicolicioiu, Marin Vlastelica Pogančić, Julius von Kügelgen, and Bernhard Schölkopf. Spuriosity didn't todn't kill the classifier: Using invariant predictions to harness spurious features. Advances in Neural Information Processing Systems, 2023.
Marco Federici, Ryota Tomioka, and Patrick Forré. An information-theoree approach to distribution shifts. A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversart training of neural networks. Journal of machine learning research, 17:1-35, 2016.
Joshua P Gardner, Zoran Popovi, and Ludwig Schmidt. Benchmarking distribution shift in tabular data with tableshift. In -hirth-seventh Conference on Neural Information Processing Systems Dalasets and Benchmarky Track, 2023.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances Neural Information Processing Systems, 33:3290-3300, 2020.
Jiawei Ge, Shange Tang, Jianqing Fan, Cong Ma, and Chi Jin. Maximum likelihood estimation is all you need for well-specified covariate shift. In The Twelfth International Conference Learning Representations, 2024.
Yury Gorishniy. Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in neural information processing systems, 34:1 2021.
11

---

Published as a conference paper at ICLR 2026
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, Bernhard Schölkopf, et al. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4): 5, 2009.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Christina Heinze- Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference, 6:201 70016, 2018.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outtt--distri examples in neural networks. In International Conference on Learning Representations, 2017.
Fredrik D Johansson, David Sontag, and Rajesh Ranganath. Support and invertibility in domain- invariant representations. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 527-536, 2019.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017.
IS Khalil, JC Doyle, and K Glover. Robust and optimal control, volume 2. Prentice hall New York, 1996.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical Review E, 2004.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International conference on machine learning, 2021.
Manabu Kuroki and Judea Pearl. Measurement bias and effect restoration in causal inference. Biometrika, 101, 2014.
Virgile Landeiro and Aron Culotta. Robust text classification under confounding shift. Journal of Artificial Intelligence Research, 63:391-419, 2018.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting cut-of-distributton samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Daniel Levy, Yair Carmon, John Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization. Advances in Neural Information Processing Systems, 33:8847- 2020.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang. and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018.
Ying Li, Xingwei Wang, Rongfei Zeng, Praveen Kumar Donta, Ilir Murturi, Min Huang, and Schahram Dustdar. Federated domain generalization: A survey. arXiv preprint arXiv:2306.01334 2023.
Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 35:24529-244 24542,, 2022.
Roderick JA Little and Donald B Rubin. Statistical analysis with missing data. John Wiley & Sons, 2019.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2 108. /06./3 2021.
Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongscok Namkoong. On the need for language describing distribution shifts: Illustrations on tabular datasets. In Thirty-sevente Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
12

---

Published as a conference paper at ICLR 2026
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Christos Louizos, Uri Shalit, Joris M Mooij, David Sontag, Richard Zemel, and Max Welling. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
Wang Miao, Zhi Geng, and Eric Tchergen Tchetgen. Identifying causal effects with proxy variables of an unmeasured confounder. Biometrika, 105:987-993. 2018.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 10-18, 2013.
Vivian Yvonne Nastl and Moritz Hardt. Do causal predictors generalize better to new domains? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78:947-1012, 2016.
Steward TA Pickett. Space-for-time substitution as alternative to long-term studies. In Long-tern term studies in ecology: approaches and alternatives, pp. 110-135. Springer. 1989.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for deep learning on tabular data. arXiv preprint arXiv:1909.06312 2019.
Parjanya Prajakta Prashant, Seyedch Baharan Khatami, Bruno Ribeiro, and Babak Salimi. Scalable out-of-distribution robustness in the presence of unobserved confounders. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
Francesco Quinzan, Cecilia Casolo, Krikamol Muandet, Yucen Luo, and Niki Kilbertus. Learning counterfactually invariant predictors. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Abbavaram Gowtham Reddy and Vineeth N Balasubramanian. Detecting and measuring confounding using causal mechanism shifts. Advances in Neural Information Processing Systems, 61699, 2024.
Abbavaram Gowtham Reddy, Benin L Godfrey, and Vineeth N Balasubramanian. On causally disentangled representations. Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pp. 8089-8097, 2022.
Mateo Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 19:1-34, 2018.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint 2.Xi:22002.06866. 2022.
Dominik Rothenhäusler, Nicolai Meinshausen, Peter Bühlmann, and Jonas Peters. Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83:215-2466 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.087. 2019.
Steffen Schneider. Evgenia Rusak, Luisa Eck, Oliver Bringmann. Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33:1 1539-115511 2020.
13

---

Published as a conference paper at ICLR 2026
Bernhard Schölkopf. Causality for machine learning. In Probabilistic and causal inference: The works of Judea Pearl, pp. 765-804, 2022.
Jessica Schrouff, Natalie Harris, Oluwasanmi O Koyejo, Ibrahim Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alexander Brown, Subhrajit Roy, Diana Mincu, Chrsitina Chen, Awa Dieng, Yuan Liu, Vivek Natarajan, Alan Karthikesalingam, Katherine Heller, Silvia Chiappa, and Alexander D' Amour. Diagnosing failures of faimess transfer across distribution shift in real- wolld medical settings. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
E. H. Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society: Series B (Methodological), 13(2):238-22) 2418, 2018.
Anurag Singh, Siu Lun Chau, Shahine Bouabid, and Krikamol Muandet. Domain generalisation via imprecise learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 45544-45570, PMLR, 2024.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv: 1710. 10571, 2017.
Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C Bayan Bruss, and Tom Goldstein. Saint: Improved neural networks for tabular data via row attention and contrastive pre-training arXiv preprini arXiv:2106.01342 2021.
Greg Ver Steeg and Aram Galstyan. Information transfer in social media, 2011.
Greg Ver Steeg and Aram Galstyan. Information-theoretic measures of influence based on content dynamics, 2013.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT press, 2012.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer visVon-ECCV 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16. 2016, proceedings, part III 14, pp. 443-450. Springer, 2016a.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer vision-ECCV 2016 workshops: Amsterdam, the Netherlands, October 8-10 and 15-16, 2016, proceedings. part III 14, pp. 443-450. Springer, 2016b.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Information Processing Systems, 33: 19286-9929, 2020.
Katherine Tsai, Stephen R Pfohl, Olawale Salaudeen, Nicole Chiou, Matt Kusner, Alexander D' Amour Sanmi Koyejo, and Arthur Gretton. Proxy methods for domain adaptation. In In- ternational Conference on Artificial Intelligence and Statistics, pp. 3961-3969. PMLR, 2024.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Shanmukha Ramakrishna Vedantam, David Lopez-Paz, and David J. Schwab. An empirical investi- gation of domain generalization with empirical risk minimizers. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yigiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing unseen domains: A survey on domain generalization. IEEE transactions on knowledge and data engineering, 35:8052-800 8072, 2022.
Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances in Neural Information Processing Systems, 34:1 1 1340-11135 11351, 2021.
Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6502-6509, 2020.
14

---

Published as a conference paper at ICLR 2026
Shen Yan, Huan Song, Nanxiang Li. Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677, 2020.
Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards theoretical framework of out-of-distribution generalization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Jingzhao Zhang, Aditya Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. arXiv preprint arXiv:2010.1222 12230, 2020.
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On learning invariant representations for domain adaptation. In International conference machine learning, pp- 7523-7532, 2019.
Han Zhao, Chen Dan, Bryon Aragam, Tommi S Jaakkola, Geoffrey J Gordon, and Pradeep Ravikumar. Fundamental limits and tradeoffs in invariant representation learning. In Journal of machine learning research, volume 23, pp. 1-49, 2022.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 194396-4411, 2022.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOFS OF THEORETICAL RESULTS
Theorem 4.1. For a covariate vector X, label Y, with causal structure X + Y, i.e., some covariates cause Y and some covariates caused by Y, environment variable E, feature extractor o, and prediction Y, the predictive information I(Y;Y) is decomposed follows:
Vartation
ship
Festure shift
Concent shifs
Cond. informatiss
Revidial
[(
I(Y;E)
I(x(X):
I(Y: E((()) -(((X)
I(Y;)) I(o(X):YIE)
2
2
(1)
where I ( is the residual information in ((X) for inferring Y that is not captured by the prediction Y. The decomposition above also holds when ((X) is replaced with X.
Proof. From the causal graph shown in Figure of the main paper, in the causal substructure: Y ((X) - ++ Y. Y provides no additional information in predicting Y given either Φ(X) or X. This is because, the path y x- (XX) f- X + Y, between Y and Y is blocked by either (o(X)
or (X}. This implies Y Y (X) and Y T Y X. Now consider the expansion of ;(Y;Y). Y))
I(Y; ((Y: 9) 1(0(X):
(Introduce (xX))
= =(((()) + I(Y; (0(8) - -I(6(X);X11)
(Expand and use Y IL Y 6(X))
- ((()))))-(((()::)))))) 1(6(x);111)
(Introduce E)
= E(Y;E) + (((()((()))))))) I(Y: Elo(()) - T((X): (Expand)
(6)
Similarly,
I(::)) I(Y: ((Y:6(X))))))) -()
(Introduce o(X))
= -(((0);)) + 101; - -(((
(Expand and use y Il Y ((X))
T(Y, ( T(x( BlY) T(((();;11) (Introduce E)
= =[((X); E) + (x()); E| - 1(6(X); E|Y) - (()(X);
(Expand)
Summing Equations and equation 7 and grouping the terms gives the desired expression:
T(Ø(X) E\)) + I(Y; E)
T(:;)) = I(O(X);((E))
2 + I(o(X): E) I(Y; ((()) I(O(
2
2
To get the similar decomposition with X instead of ((X), use y 1 Y X and introduce X into the expansion instead of ((X).
Theorem 4.2. For a covariate vector X. label Y, an environment variable E, a feature extractor a. the prediction Y, and an unobserved confounding variable U that cause both X and Y, and if either (i) or (ii) Y + X. the predictive information 1 I(Y; Y) can decomposed as follows:
Cond. informativeness
Revidual
I(Y; = (X) Y|E) - -(((());
Proof. Following the causal graph shown in Figure 3 of the main paper, we prove the theorem two separate cases: X Y and Y - X.
16

---

Published as a conference paper at ICLR 2026
Case 1: X ++Y, Y. In this case, we have the following inequality because conditioning Y opens the path o(x) t X Y + U between ((X) and at the collider node Y:
[( Z [()
(8)
Additionally, considering the causal substructure: Y + X + U inequality because conditioning on ((X) partially blocks the information flow from Y E the node X as long ((X) encodes some information about X:
++ E, we have the following
T(Y; F.) 2 T(Y; (X)(
Now consider the following expansion from Equation 7 in the proof of Theorem 4.1.
I(Y;Y) T(x(X): F) + (() T(x(X);
BJY) ((X());
(Equation 7)
((x()); E - ((( I((())
(Using 8)
I(((X); Y| - I(6(X);
Similarly, consider the following expansion from Equation 6 from the proof of Theorem 4.1:
i( = I(Y; + + (X -(X; - - (Equation 6)
2 I(Y; / I(Y; E) (((()) (Using 9)
That is, we have -
S I(Y;)
Case 2: Y + X. In this scenario, we have the following inequality because conditioning on c(X) opens the path E U Y between and the collider node X because ((X) is child of X (Pearl, 2009).
I(Y; (I)( I(Y; E)
(10)
Additionally, we have the following inequality because conditioning on blocks the path <(X) X Y U <> E at Y and hence less information flow between ((X) and E.
I(o(X): E) ((((X)E)
(11)
Now consider the following expansion from Equation 6 from the proof of Theorem 4.1.
I(Y:)) = x(
((((X): (X)( I(Y; F()())())) (((X); (Using 10)
(Equation 6)
< (((( ((((
= =((((X); - X(1
Similarly, consider the following expansion from Equation from the proof of Theorem 4.1.
v T(o(X); F.) + T(<X); (((X) (((())
(Equation 7)
2 [(((X); E) + ((((); ( -1
(Using 11)
I(((X): Y I(6(X);YI))
That is, we have I(o(X); (( I(O(X;; YI < I(Y; < -
Since the upper bound and lower bounds are the same both cases, we have the desired result I(Y; 8) (((();; - 1(((X);
Proposition 4.1. of ())()), (-),z() (2(-) are invertible functions and X, are informative variables such that
X, X, E, then we have the following inequalities:
(i) ((xx((X > ( (Mi) (XX ElY; (iv)
(ii)
(5)
(())
17

---

Published as a conference paper at ICLR 2026
Proof. (i) By the chain rule of mutual information, have: I(X, Xx; = (()
I[x;; EX We have 1(X;; Y, > O because Y I X X, E. Hence, we have I(X,;;; Y I(X; E). Since o1 (+), (-(-) are invertible functions and mutual information is invariant to invertible transformations, we have (( U x;)); F)) > T(á1(X) F))).
(ii) We consider the causal graph shown in Figure 3. From the chain rule of mutual information, we have: I(X, x;; E)= = [(X: E) + y(X;; X). Since E ++ U - Y, any covariate that is informative to Y is also informative
U. Since E U, we have [(x;; E|X) > O, and hence X;; E) > I (X; E). Since é1 (-), (-)-) are invertible functions and mutual information is invariant to invertible transformations, we have [( X))); > [(0r (X): E).
(iii) Consider [Y;; E, X, X). It can be expressed in two ways:
E, X) = I[;; E X) + XY; E,X) I(Y;X, X) + [X
Equating and rearranging the terms gives the following.
I(Y; E X) - I(Y; Xr, X) = I(;; X, X) I(;; E,
Since Y I X, E, we have [(Y; X, X) IY: E, X) because conditioning usually reduces mutual information unless the additional conditioning variable opens any collider paths. Here E neither collider not descendant of any collider. Thus, I(Y; X) I(; X). Since s (1()), (2(-) are invertible functions and mutual information is invariant to invertible transformations, we have I(Y; E Φ1 (X)) > ((Y;B) E ( U X)))..
(iv) By the chain rule of mutual information, we have: 1(X, E Y) I(x;E Y) T( F| Y). Since E ++ U - Y, any covariate that is informative to is also informative to U. Since E <+ U, we have (Xy; E Y) > O, and hence [(X, Xy; E Y) > [Xx YY. Since (1()), (1()) are invertible functions and mutual information is invariant to invertible transformations, have ((x2((X x;); E E Y) I(d1 (X); E Y).
B EXPERIMENTAL SETUP
In this section, we detail our experimental setup. The impact of random seeds on the results is statistically insignificant in many settings (Gardner et al., 2023). For the results in the main paper, we report the mean and standard deviation across five random hyperparameter values (sampled from their respective domains in Table B1) for each dataset-method combination. Since we evaluate models on ID test and OOD test data, to evaluate mutual information terms, we use the nonparametric entropy estimation toolbox (Kraskov et al., 2004; Steeg & Galstyan, 2011; 2013). Following (Gardner et al., 2023), we evaluate mutual information using both in-distribution (ID) test data (from training domains) and out-of-distribution (OOD) test data (from test domains).
For consistency, we select 20,000 random sam- ples from each ID and OOD test data, though some datasets contain fewer than 20,000 sam- ples. Throughout our experiments, the ple size has a relatively insignificant effect on the evaluated mutual information terms. For instance, as shown in Figure B1, the terms in Equation| computed for the representations of the XGB model on the income dataset stayed relatively consistent as we vary sample size. Fea- tures ((X) are extracted from the layer preced- ing the classification head; for XGBoost, we combine output margins and SHAP values from the model. All experiments were conducted on single NVIDIA RTX A6000 GPU.
Sample Size
Sumple Size
Res
Vir
CS
Figure B1: Evaluated terms of predictive informa- tion are relatively consistent across various sample sizes on the income dataset.
In our experiments studying the impact of informative covariates on predictive information, we follow the categorization introduced by Nastl & Hardt (2024), partitioning covariates into: causal, arguably causal, and all covariates. Causal covariates have clearly established influence on the target, with well- supported directionality and unlikely reverse causation. However, strict reliance causal variables
18

---

Published as a conference paper at ICLR 2026
Table Bl: Hyperparameters and their possible values considered in this paper.
Model
Hyperparameter
Search values
Learning rate
[0.1, 0.2, 0.3]
Maximum tree depth
4, 6]
Minimum child weight
0.1,, 1, 10]
XGBoost
Gamma (min. loss reduction) Subsample ratio
0.001,,0001,0.00 0.01]
0.5, 0.6, 0.7]
Column subsample per LI regularization (a)
0.5, 0.7]
1.0001,00011, 0.01
L2 regularization (A)
0.000,,0.001,1 0.01
Number of layers
2, 4]
Hidden layer size
[256, 512, 1024]
Dropout rate
0.00, 0.1,
MLP
Learning rate
00.01, 0.02, 0.05]
Weight decay (L2)
00.0001, 0.001, 0.01] [096] [1,2,3]
Batch size
Number of epochs
Number of layers
[2,3, 4]
Hidden layer size
2256, 512, 1024]
troup-weights step size
0.01, 0.05, 0.1]
GroupDRO
Dropout rate
0.1, 0.2]
Learning rate
0.01, 0.02, 0.05]
Weight decay (L.2)
0..0001, 0.001, 0.01]
Batch size
[4096] [1,2,3]
Number of epochs
Number of layers
[2,3, 4]
Hidden layer size
256,, 512, 1024]
Dropout rate
0.0, 0.1, 0.2]
IRM penalty weight (A)
00.01, 0.05, 0.1]
IRM
IRM penalty anneal iterations
1,2,3]
Learning rate
0.01, 0.02, 0.05]
Weight decay (L2)
0.0001,0.001 0.01]
Batch size
4096]
Number of epochs
1,2,3]
Number of layers
[2,3, 4) 256,, 512, 1024]
Hidden layer size
VREX penalty anneal iterations
1,2,3]
VREX penalty weight (A)
0.1, 10, 100]
VREX
Dropout rate
0.00, 0.1, 0.2
Learning rate
0.01, 0.02, 0.05]
Weight decay (L2)
0.0001, 0.001, 0.01] 4096] [1,2,3]
Batch size
Number of epochs
risks omitting relevant parents due to knowledge gaps. Arguably causal covariates have uncertain causal relationships, meeting at least one of: (1) being known causal, (2) having plausible but poten- tially bidirectional influence, or (3) likely (but unconfirmed) causal effect. These groups approximate true causal parents based available knowledge, though relationships may be confounded. Some datasets additionally contain anti- anti-causal covariates where the target likely affects them but not vice versa. The all covariates includes all observed variables regardless of causal status. The sign con- sistency metric introduced in the main paper $ is defined as follows. For each (m, am) E M where M {(conditional informativeness, +1), (variation, -1), (label shift, +1), (feature shift, +1),
19

---

Published as a conference paper at ICLR 2026
Readmisston
Food Stamps
Dlabetes
In
01300
0.99
6:30
6:35
0:13
0099
6:34
0028
-0025
6:00
XG8 MLPODRO VREX
GDRO IAM
XGB
MLP IAM VIRE
Unemployment
ASSISTments
Public Coverage
Hypertenson
6:04
6:0
-4.5
-41
-42
.0.00
XGB GDRO IMM VREX
XGB MLPGDRO IRM VIRE
XGB MLP GDRO IMM VREX
XGB MLP GDRO IRM VIRX
CI Res
Var
LS
C5
Figure C2: Dataset specific results on data.
(concept shift, 1), and (residual, -1)}, for specific dataset, compute:
C. 2|Pl
1
[T (om (17(m) (m)) > 0) + +I (am (1)(mm 1/5 (1(mm) > 0)] (a.))P
Σ
Where P is the set of covariate set pairs ts with t s. For example, can be causal covariates and can arguably causal covariates. There is factor of in that accounts for both train and test data. / (m) is the mean of measure m for covariate set in training/testing data. T(-) is the indicator function. |P| for covariate settings (causal, arguably causal, all). To get final sign
consistency metric value for each m, we average across datasets: C. 1 is the number of datasets.
D EBED C. where D]
C ADDITIONAL RESULTS ON REAL-WORLD DATASETS
In this section, present additional results on real-world datasets. Figures C2, C3 show the com- parison the terms: conditional informativeness - residual and -variation / label shift /2 feature shift / 2 concept shift /2 for each dataset. The key takeaway from these results is that, from Theorem 4.1, the sum of blue and orange terms is positively correlated with the overall model perfor- mance. At the same time, due potential hidden confounding shift in real-world data, the contribution of the difference conditional informativeness residual is higher towards the overall predictive in- formation when compared to the contribution -variation / 2 + label shift / + feature shift / concept shift/ / 2. This serves as motivation to maximize conditional informativeness for better OOD generalization under hidden confounding shift.
Table C2: Comparison of ID and OOD test accuracies of models.
Method
Readmission
Food stamps
Diabetes
OOD
ML.P GDRO
65.78 + 0.14 60.47
61.75 + 0..22
84.74 0.03
82.00 + 0.06
87.66 + 0.02
83.22 + 0.05
1.08
57.60
0.42
84.47
0.04
81.41
87.47 0.10
82.90 0.10
IRM
50.28 1 7.35
50.85
1 1.74
80.91 1 0.00
78.01 1 0.00
42.54 11 36.52
43.48 13.99231.92
Method
Income
Unemployment
MLP
82.92 + 0.05
81.46
0.01
97.27 + 0.02
96.04 04 0.05
GDRO
82.70 0.05
80.16 0.45
97.28
96.10 0.00
IRM
67.91 1 0.00
60.20 0.00
96.61 1. 0.00
94.84 1: 0.00
20

---

Published as a conference paper at ICLR 2026
Environments
Readmission
604
8450-
6:35
035
Unemployment
ASSIS Timents
Public Coverage
Hypertenson
0.3
3.10
0.879
0.230
0.00
0.9
0.000
.030
-0.15
GDRO
-1.10
XG8 GDRO VREX
-0.75
XGB MLPGDRO
-0.5
XGB MLPGDRO VREX
CI
-Res
Var
C5
Figure C3: Dataset-specific results on OOD data.
In-domain: Avg. Over Datasets
Out-of-domainn Avg. Over Datasets
Method
Avg. Test Acc.
Avg. OOD Test Acc.
MLP GDRO
83.67
$0.89
82.48
79 64
MLP
GBRD
IRM
MEP
GDRD
IRM
Predictive Information
Serour
Conditional informativeness
Figure C4:
The difference conditional informativeness residual in the plots is positively correlated with the average ID and OOD test accuracy over five datasets shown in the table on the right. The contribution of the difference conditional informativeness residual is higher towards the overall predictive information when compared to the contribution of variation / label shift 2 feature shift / concept shift 22
Readmission
Food Stamps
Diabetes
Income
Unemployment
Trah Instralments
S0R0
1080 Stamps
PCP
5242
0282 Unemployment
Readmission
Thit Enottommam
500
informativeness
Marlytiie
Label siilt
Fastara chift
Concept
Figure C5: Dataset specific results. Comparison of predictive information among models.
21

---

Published as a conference paper at ICLR 2026
Figures C6- C15 further show the comparison of various terms of the information decomposition in for each method and dataset. From these results, we observe that the values plotted in the second column (variation) and third column (feature shift) are similar in magnitude, and based on Equation 1, their difference is close to zero, contributing very little to the overall predictive decomposition. Similarly, observe that the values plotted in the fourth column (label shift) and fifth column (concept shift) are similar in magnitude, and based on Equation 1, their difference is close to zero, contributing very little the overall predictive decomposition. almost all settings, the difference conditional informativenss residual is the majority contributor for the predictive information.
We also perform hyperparameter tuning on model parameters. We run a grid search over hyperparameter values and select the best hyperparameter values based on ID validation accuracy (not OOD validation accuracy). We run 1500 experiments datasets, 3 models, and 100 hyperparameter choices sampled from grid of hyperparameter values). In this case, to get the mean and standard deviation results for accuracies and mutual information terms, we run each experiment for 5 random seeds, even if the effect of random seeds is insignificant in many cases, Results are shown in Figure C4. From Theorem 4.1, the sum blue and orange terms is positively correlated with the overall model performance. At the same time, due potential hidden confounding shift in worldw data, the contribution the difference conditional informativeness residual is higher towards the overall predictive information when compared to the contribution of -variation / 2 + label shift 2 + feature shift / - concept shift / 2. Dataset-specific results shown in Figure C5 and Table C2 also show similar trends.
Hype Hyperparameter tuning:
Potential limitations with predictive information decomposition analysis: While predictive information I(Y;Y) Y) measures statistical dependence between predictions and labels, it does not guarantee accuracy. For instance, a binary classifier that systematically predicts the wrong label (e.g., y when Y 0) achieves maximal T(Y;Y) 1 bit (for binary variables) due to perfect anti coti-correlatio,, yet yields 0% accuracy. This occurs because mutual information quantifies reduction in uncertainty rather than correctness. Other decomposition terms provide further insight: conditional informativeness I (x);;; Y|E) may mask harmful variation [(((X); EJY), while feature shift (Xx F) can inflate T(Y: Y) through spurious correlations. Nevertheless, this decomposition remains valuable for analyzing reasonably performant models, as our experiments demonstrate- it reveals whether predictive information stems from generalizable patterns environmment-speciiic artifacts, enabling targeted improvements while maintaining interpretabiliity.
22

---

Published as a conference paper at ICLR 2026
Train Shifts XGB
Inform.
Variatinn
Feature
Concept
Recidral
Thit ACC.
Figure C6: Decomposition of information metrics on train data for XGBoost and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
23

---

Published as a conference paper at ICLR 2026
Test Data Shifts XGB
A.C
A/C
AC
A.C
A.C
A/C
AC
A.C
A.C
AC
AC
A.C
A.C
A.C
Figure C7: Decomposition of information metrics on test data for XGBoost and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
24

---

Published as a conference paper at ICLR 2026
Train Data Shifts MLP
Cond,
Varlation
Feature Shift
Label
Concept Shift
Residual
Thxt
A.C
A.C
AC
A.C
A.C
A/C
A.C
A.C
A/
AC
A.C
AC
AC
AC
A.C
AC
A.C
4.
AC
A/C
AC
AC
A.C
A.C
AC
AC
A.C
AC
AC
A.C
A6
A6
AC
Figure C8: Decomposition of information metrics on train data for MLP and groups of features: (C)
Causal, (A.C) Arguably causal, (A) All.
25

---

Published as a conference paper at ICLR 2026
Test Data Shifts MLP
A.C
A/C
AC
A.C
A.C
A/C
AC
Starees
A.C
ACC
AC
PAINIE Coverreee
Preseere
A.C
A.C
ACC
AC
Figure C9: Decomposition of information metrics on test data for MLP and groups of features: (C)
Causal, (A.C) Arguably causal, (A) All.
26

---

Published as a conference paper at ICLR 2026
Train Data Shifts GDRO
A.C
ACC
A.C
AC
A/C
A.C
A.C
A.C
ACC
ACC
Figure C1O: Decomposition of information metrics on train data for GDRO and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
27

---

Published as a conference paper at ICLR 2026
Test Data Shifts GDRD
A.C
A/C
AC
A.C
A.C
A/C
AC
Stames
4.C
A.C
A.C
AC
Preseere
A.C
A.C
AC
AC
Figure C11: Decomposition of information metrics on test data for GDRO and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
28

---

Published as a conference paper at ICLR 2026
Train Data Shifts IRM
A.C
A/C
AC
A.C
A.C
A/
AC
A.C
AC
AC
ACC
AC
A.C
A3C
Figure C12: Decomposition of information metrics on train data for IRM and groups of features: (C)
Causal, (A.C) Arguably causal, (A) All.
29

---

Published as a conference paper at ICLR 2026
Test Data Shifts IRM
A.C
A/
A
AC
A.C
A.C
A/6
AC
A.C
AC
A.C
AC
4.C
A.C
A.C
AC
Figure C13: Decomposition of information metrics on test data for IRM and groups of features: (C)
Causal, (A.C) Arguably causal, (A) All.
30

---

Published as a conference paper at ICLR 2026
Train Data Shifts VREX
Reatmbisie
A.C
A.C
AC
A.C
AC
A/C
A.C
AC
ACC
ACC
Figure C14: Decomposition of information metrics on train data for VREX and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
31

---

Published as a conference paper at ICLR 2026
Test Shifts
Cond, Inform.
Vartation
Fasture Shift
Concapt
Residua!
ACC.
Aic
AC
ACC
Figure C15: Decomposition of information metrics on test data for VREX and groups of features:
(C) Causal, (A.C) Arguably causal, (A) All.
32

---

Published as a conference paper at ICLR 2026
D RESULTS ON SYNTHETIC DATASETS
We conduct several experiments synthetic datasets to investigate: (i) the impact of different informativeness criteria, (ii) how the level of confounder overlap across environments affects model performance, and (iii) the influence of the number of causal versus anti-causal variables the performance. The data-gener process follows the structural equations in 12, with the causal structure U + X, U > Y, U X1, and X + Y. where U is an unobserved confounder, X is an observed covariate, Y is the target variable, and X, represents additional informative covariates.
U U~N(P X, Xx
X - fx( + NNua,, + (xx,))))+N((
(12)
x,
We explore different functional forms for fx. fis and fY, with the corresponding results presented in analysis. Domain shifts induced by systematically varying the environment-specifee mean parameter μ4, of the confounder distribution. This allows us to examine how different degrees of distributional shift affect model performance across environments.
Informativeness vs. accuracy: In this set of experiments, we begin with linear structural equations: 0.1U, and fy X 2U. For these experiments 14 2- for training
fx = 0.3U, environments and 14 E {0, 4} for test environments. For the results presented in Figure 4 of the main paper. we consider one hidden confounding variable with X 20 informative covariates. As we increase the number of informative covariates from 0 to 20, we observe: (i) a reduction in mean squared error, (ii) improved conditional informativeness, (iii) enhanced feature shift, while (iv) decreased concept shift. We extend these findings by examining two additional informativeness criteria. First, we analyze the case with one hidden confounder and a single informative covariate Figure D16 demonstrates that noise
(Xx 1) across varying noise levels σ; E 0.1.. 0. 2.0 decreases from 2.0 to 0.0, we observe: (i) reduced mean squared error, (ii) improved conditional informativeness, (iii) enhanced feature shift, alongside (iv) decreased concept shift. Consistent with other synthetic experiments, the variation term remains zero throughout. Finally, we investigate a setting with 20 hidden confounders and corresponding 20 informative covariates, using modified structural equations fx = 0.3U, f; = 0.2U, and fy = X 2U. Figure D17 shows that as we introduce informative covariates corresponding to hidden confounders, we again observe: (i) reduced mean squared error, (ii) improved conditional informativeness, (iii) enhanced feature shift, while (iv) decreased concept shift. These experiments demonstrate how different informativeness criteria help in achieving OOD generalization.
Mean Squared Error
Cond, Informativeness Variation
Feature Shift
Concept Shift
Train
Toet
Taan
Twxt
Train
Test
0.s
0.5
0.5
0.0
Libear Regross on
-
XGB
GDRD
+
1000
VIEX
Cond, Informaivanass
Var alion
- Feature Stift
--- Concap:: Shift
Figure D16: Adding proxy variable with low noise helps in reducing MSE, increasing conditional informativeness and feature shift while reducing concept shift.
Mean Squared Error
Cond. Informativeneus Variation
Feature Shift & Concept Shift
Train
1.0
1.6
10
1.0
20
20
1.0
20
100
20
10
20
Regression on
GDRD
Cand. Informsvvenes
Var Jtinn
Feature
Concapt Shift
Figure D17: Adding more proxy variables X, of set of hidden confounding variables U that are informative to Y helps in reducing MSE, increasing conditional informativeness and feature shift while reducing concept shift.
33

---

Published as a conference paper at ICLR 2026
UaLE Tour 26.8
HIL 2- MeR:
NLL OATE A.N.
Structural Equations:
(Hidden Confounder) U~ U-.VVe. 0.5?) E 2.2) for ID
(0, for DOD
(npuut 0.75 .TD.
(Dutcome) Y- 3U+ A10, 0.5%) (Informative) X, 0.6U+++++++ ARO, 0.39)
6
40
Train data
data
Predictions en train data
Predictions on test
Figure D18: Performance of MLP on the low overlap non-linear setting, for sets of features: (i) X (ii) X and E (environment statistics) (iii) X and X, (iv) X and U (Oracle).
NLP Using Teatl 18511 2
Structural Equations:
Text
Text MStI 6.B.
Text
(Hidden Confounder) Alp, 0.52) Ae ( -1, 1.0) for ID Me (-0.5, 0.5) for 00D
(input) X 0,750 A10, 1.5%)
(Dutcome) x (Informative) X 0.6 -NTO, 0.3%)
-MD, 0.52)
X
X
Train data
20
Test data
Predictions on train data
Predictions on test date
Figure D19: Performance of MLP on the high overlap non-linear setting, for sets of features: (i) X (ii) X and E (environment statistics) (iii) X and X; (iv) X and U (Oracle).
Overlapping confounder support vs. accuracy: As established in the main paper, recent work by Prashant al. (2025) proposed an OOD generalization method for hidden confounding shift that assumes test confounder support remains within training support. While our results demonstrated successful learning of correct relationships in synthetic linear data when sufficient information about hidden confounders exists in observed covariates, we now extend this analysis to nonlinear data using MLP and XGB models under both low and high confounding support conditions (Figures D18- D21). In low confounding overlap settings (characterized by distant μ4 between ID and OOD data), models relying solely on X learn an inadequate global function that fails to distinguish environments. Performance improves when incorporating environment cerreiiiet--ppccfii summary statisties of observed covariates, which helps capture environment-specific relationships, but the most sigmificant gains occur only when leveraging additional informative covariates, underscoring their importance. For completeness, we include both oracle model results and U-Y relationship scatter plots to elucidate the learned input-output mappings. These experiments confirm better performance under high confounding overlap as expected, but more importantly, reveal promising results in the low-overlap regime- previously unstudied scenario that challenges the common support assumption in the literature.
Number of causal vs anti-causal variables: Recent benchmarks (Gardner et al., 2023) show that bascline methods (e.g., XGB, MLP) consistently match or outperform OOD generalization techniques (e.g., GDRO, IRM, VREX) in real-world datasets, coinciding with the prevalent causal structure X + Y in tabular data. To test if baseline superiority stems from the causal structure:
X + Y, we vary the causal-to-anti-causs. ratio p = X {0.0.... 1.03 with X = 50 and X X UXA (e.g., P 0.5 gives [Xcl X 25). Results (Figure D22) show baseline
resilience even under the causal structures X + Y + X though no method dominates universally (Figures D23, D24). Baselines achieve superior PMA-OOD scores (Gardner et al., 2023) (fraction of maximum OOD accuracy). We evaluate: (1) ERM- BRsd methods (XGBoost (Chen & Guestrin, 2016), LightGBM (Ke al., 2017), ML.P, ResNet (Gorishniy et al., 2021), SAINT (Somepalli et al.,
34

---

Published as a conference paper at ICLR 2026
KORUSNI That
Structural Equations:
Confeunder) U-NNee. 0.5%) Me 2.2)
20
Ue ((
(Ipput) X 0.75 U NNO, 1.5%)
(Dutcome) x= 3 AÑO, 0.5?) (Iformativve X, 0.6.U++ A10, 0.3%)
Test data
data
Predictions on test data
Figure D20: Performance of XGB the low overlap non-linear setting, for sets of features: (i) X (ii) and (enviroument statistics) (iii) and X; (iv) X and U (Oracle).
Usiing
20B Usinn
Uhing X (Dracl))
Structural Equations:
(Hidden Me (-1.
H4 (-0.5.0.5)
(Input) X 0.750 + -70, 1.5%)
(Outcome) X= 33 -T0, 0
(Iformative)
28
Test data
10
Predictions
Predictions on test data
Figure D21: Performance of XGB the low overlap non-linear setting, for of features: (i) X (ii) and (environment statistics) (iii) X and X; (iv) X and U (Oracle).
35

---

Published as a conference paper at ICLR 2026
OOD Test Accuracy Model On Linear Synthetic Dataset
000
Domain Coneraa lz Methods
Domain Adaptation Mathods
Robust Learring Metho
ERMM-Based Methods
Figure D23: OOD test accuracy on the linear setting per model for different values of p.
2021), Tabrransformer, NODE (Popov et al., 2019), FT-Transformer, ExpGrad (Agarwal et al., 2018)); (2) OOD generalization methods (IRM (Arjovsky et al., 2019), IB-IRM, IB-ERM (Gulrajani & LLope--Paza Paz, 2021), CausIRL (Chevalley et al., 2022), DANN (Ajakan et al., 2014), MMD (Li et al., 2018), DeepCORAL (Sun & Saenko, 2016 V-REx (Krueger et al., 2021), Domain Mixup (Xu et al., 2020; Yan et al., 2020)); and (3) Robust optimization methods (DRO (Levy et al., 2020), GroupDRO (Sagawa et al., 2019), Label DRO, Adversarial Label DRO (Zhang et al., 2020)).
Percentage of Max Accuracy (PMA-OOD) on Synthetic Datasets Across All ratios (p= IX]
Left: Data With Linear Equations, Right: Data With Nonlinear Equations [X]
)
XGB
Adv. Label DRO
dfrmer
Explerad
Label apRo
6
Causil COKALL
DANN
7
Label GORK
8
Adw Labul DRO
8
IN INN
Uppccas
9
SAINT
10
HLP
10
Milp
11
XGB
11
Noot
88
12
Explered
12
Tab DnTranntommer
13
saia!
13
IRN
14
Cautid wikio
14
MLP
15
15
Frirandoomer
16
16
DeeSCORAL
17
Is Ban
17
DaNn
18
HixUp
Reaflitt
19
Causifi CORAL
I8 ERH
20
RRD
VIEE
DeapCORAL
MicUp
TIEX
Cauaii MPR
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0,4
0.6
0.8
1.0
Percentage of Max Accuracy (PMA-COD)
Percentage of Max Accuracy (MAAODD)
BRM-Based Methods
Robust Leaming Methods
Domain Adaptation Methods
Domain General Methods
Figure D22: ERM- based methods achieve better percentage of maximum OOD (PMA-OOD) test accuracy on the synthetic datasets, in line with the world results of (Gardner et al., 2023).
36

---

Published as a conference paper at ICLR 2026
OOD Test Accuracy by Model On Nonlinear Synthetic Dataset
2000
2002
000
0207
D'013
000
Domain Generalization Methods
Domain Adaptation Methods
Robust Learrnng Methods
ERM Based Methods
Figure D24: OOD test accuracy on the non
non-linee linear setting per model for different values of p
37

---

Published as a conference paper at ICLR 2026
E QUALITATIVE ANALYSIS OF HIDDEN CONFOUNDING SHIFT IN REAL-WORLD DATASETS
In this section, qualitatively study the presence of hidden confounding shifts in real-world datasets. Tables E3-E18 present potential hidden confounders influencing various observed covariates in several real-world datasets (Nastl Hardt, 2024). For additional details regarding these datasets, we refer to (Gardner et al., 2023; Nastl & Hardt, 2024). We use GPT4o (Achiam et al., 2023) for this task, and we acknowledge that the list of unobserved confounding variables provided is not exhaustive. This study supports the arguments for the existence and impact of hidden confounding shifts across domains in real-world datasets. These GPT-4o responses are solely meant for semantic insight and are not implicitly or explicitly used in other experiments.
Prompt: We use the prompt below to query GPT4o to get possible hidden confounders.
For target variable, you will be given lists of causal, arguably causal, anti-causal, and spurious covariates. Also, you will be provided with envi ronment variables such that different datasets are collected in environments induced by these environment variables. Provide possible hidden confounders that cause any of the target, causal, arguably causal, spurious, and anti-causal covariates. That is, the hidden confounders should influence the distribution of their children in different demains. Ensure hidden confounders are not a part the given variables, and your explanation should include how hidden confounders influence their children in different environments.
Table E3: Dataset: Food stamps. Target: Food stamp recipiency in the past year for households with child. Environments: regions in the United States.
Hidden confounder
Affected variables
Reason confounding
Local economic conditions
Food stamp recipiency, Age, Sex, Race, Marital status
Poor economic conditions increase food insecurity (raising food stamp usage), while also influencing population demo- graphics due to migration, employment patterns, and household formation.
State or regional public as- sistance policies
Food stamp recipiency, Marital status, Number of children, Household income
Generous or restrictive assistance policies directly affect food stamp eligibility and indirectly influence household structure and income distribution.
Cultural attitudes towards welfare and public assis- tance
Food stamp recipiency, Marital status, Disahility, Race, Ethnicity
Social stigma support for welfare af- feets both the uptake of food stamps and social norms around marriage, disability reporting. and identity categories.
Regional health care condi- tions
Food stamp recipiency, Dis- ability, Cognitive difficulty, Hearing difficulty, Vision difficulty
Regions with limited healtheare access may show higher disability prevalence and greater reliance on food stamps due to in- creased financial and care burdens.
38

---

Published as a conference paper at ICLR 2026
Table B4: Dataset: Income. Target: Total person's income 2 56K for employed adults. Environ- ments: regions in the United States.
Hidden confounder
Affected variables
Reason for confounding
Local economic conditions
Income, Occupation, Edu- cational attainment, Marital status
Stronger local cconomies offer better- paying jobs and education access, increas- ing income; they also influence occupa- tional choices and household formation patterns.
State or regional tax and la- bor policies
Income, Class of worker, Educational attainment, Marital status
Tax incentives and lahor protections af- feet wages and employment types, while also shaping decisions about education and family due financial security.
Cost of living and regional affordability
Income, Occupation, Mar- ital status, Educational tainment
Higher living costs necessitate higher in- comes and may drive occupational ed- ucational shifts; they also influence mar- riage or cohabitation decisions.
Cultural and gional economic history
Income, Educational attain- ment, Occupation
Cultural values and historical industry presence shape education levels and joh opportunities, which in turn affect income distributions.
Table ES: Dataset: Public coverage.
Target: Coverage of non-Medicare eligible low-income
individuals. Environments: Disability statuses.
Hidden confounder
Affected variables
Reason for confounding
State or regional healthcare policies
Public health coverage, Marital status, Employ- ment status, Income
Differences in Medicaid expansion and public insurance eligibility influence health coverage; these policies also affect economic stability, employment, and fam- ily dynamics.
Economic conditions and poverty levels
Public health coverage, Em- ployment status, Income, Marital status
Poorer regions tend to have lower employ- ment rates and incomes, which increase reliance on public health coverage and fluence marriage and household composi- tion.
Cultural attitudes towards disability and hcalthcare
Public health coverage, Dis- ability status, Cognitive dif- ficulty. Hearing difficulty, Vision difficulty
Stigma or support for disability and public care varies culturally, affecting both the reporting of disabilities and the likelihood of seeking receiving public coverage.
Urban vs. rural divide
Public health coverage, Ed- ucational attainment, Mari- tal status, Occupation
Urban areas typically offer better health- care access, education, and jobs, all of which affect coverage likelihood and cioeconomic indicators.
39

---

Published as a conference paper at ICLR 2026
Table E6: Dataset: Unemployment. Target: Classify whether a person is unemployed. Environ
ments: Educational attainments.
Hidden confounder
Affected variables
Reason for confounding
Local cconomic conditions
Employment status, Occu- pation, Marital status, Mo- bility status
Regional cconomic strength affects job availability and unemployment rates: it also shapes occupation types, migration decisions, and houschold stability.
State or regional labor mar- ket policies
Employment status, Occu- pation, Educational attain- ment, Marital status
Labor regulations and unemployment ben- efits vary region, affecting hiring prac- tices, education incentives, family structures.
Cultural and social norms around employment
Employment status, Dis- ability status, Marital status
Cultural attitudes toward work and depen- dency influence unemployment reporting and societal roles around disability and family responsibilities.
Health and disability status
Employment status, Dis- ability status, Cognitive dif- ficulty, Hearing difficulty. Vision difficulty
Poor health or disabilitics reduce the abil- ity to work, directly increasing unem- ployment and shaping associated health- related variables.
Table E7: Dataset: Voting. Target:
Classify whether person voted in the U.S. presidential election. Environments: United States census regions.
Hidden confounder
Affected variables
Reason for confounding
Political, family, and peer influences
Voted in national election, Party identification, Politi- cal participation
Social networks shape political idcology and engagement, influencing both voting likelihood and party alignment.
Media consumption habits
Voted in national election, Party identification, Politi- cal knowledge, Voting be- havior
Media exposure affects awareness of po- litical issues and biases, influencing party affiliation, political knowledge, and voting participation.
Social capital
Voted in national election, Political participation, In- terest in elections
Strong community ties and civic networks increase political interest and participa- tion, Icading to higher voter turnout.
Civic education and politi- cal engagement programs
Voted in national election, Interest elections, Politi- cal knowledge
Educational programs raise political awareness and civic responsibility, influ- encing both knowledge levels and voting decisions.
Historical cultural con-
Voted in national election, Party identification, Interest in elections
Historical events regional political cul- ture affect interest in elections and party alignment, which in turn influence voting behavior.
text
40

---

Published as a conference paper at ICLR 2026
Table E8: Dataset: Hypertension. Target: Whether a person has hypertension. Environments:
Body Mass Index (BMI) values.
Hidden confounder
Affected variables
Reason for confounding
Sociocconomic status
Income, Employment sta- tus, Smoking habits, Alco- hol consumption, Physical activity, Healthcare access, Medical cost, High blood pressure diagnosis
Lower sociocconomic status reduces ac- cess healthcare healthy lifestyle op- tions, leading to poor diet, limited activity, and delayed diagnosis, which jointly influ- ence both BMI and hypertension risk.
Access to healthcare ser- vices
Healthcare access, Medical Smoking habits, Al- cohol consumption, Phys- ical activity. High blood pressure diagnosis
Limited healthcare access results in under- diagnosis and unmanaged hypertension, while also affecting lifestyle choices that vary with BMI, confounding the relation- ship between BMI and hypertension.
Genetic predisposition
Age group. Race, Sex, Smoking habits, Diabetes, High blood pressure diag- nosis
Genetic risk factors for hypertension may co- co-vary with demographic attributes and influence both hypertension prevalence and BMI distribution across subpopula- tions.
Psychosocial stress
Smoking habits, Alcohol consumption, Physical ac- tivity, High blood pressure diagnosis, Diabetes, Age group
Chronic stress alters behavior (c.g., smok- ing, inactivity) and physiological sponses, contributing to both increased BMI and elevated blood pressure, thus confounding the BMI hypertension link.
Environmental factors
Physical activity, Diet, Smoking habits, High blood pressure diagnosis, Diabctes, BMI category
Living environments affect ational spaces, food quality, and pollution exposure. influencing both BMI and hy- pertension risks. These vary across BMI categories, creating confounding.
Dietary habits beyond fruits and vegetables
Alcohol consumption, Smoking habits. Physi- cal activity. High blood pressure diagnosis, BMI
High-sodium or processed-foo diets raise both BMI and hypertension risk. Variation in such unmeasured dictary habits across BMI categories creates spurious associa- tions with hypertension.
category
41

---

Published as a conference paper at ICLR 2026
Table E9: Dataset: College Scorecard. Target: Predict completion rate for first-time, full-time students at four-year institutions. Environments: Based Carnegie Classifications.
Hidden confounder
Affected variables
Reason for confounding
Institutional funding and sources
Accreditor, Control in- stitution, Highest degree awarded, In-state tuition, Out-of-state tuition, Cost of attendance, SAT scores
Wealthier institutions can offer better aca- demic support, facilities, and programs, leading to higher completion rates and more selective admission profiles. This varies across Carnegie classifications.
Regional socio-economic factors
Region, Poverty rate. Unemployment rate, SAT scores
Economic conditions across regions affect affordability, student preparedness, and institutional support levels, all influencing both enrollment outcomes graduation likelihood.
Demographic factors
HBCU flag, Federal loan re- cipient rate, Pell grant recip- ient rate, ACT scores, Un- dergraduate enrollment
Student demographics shape financial aid needs, academic preparation, and gradua- tion rates. Their effect differs by institu- tion type and selectivity under Carnegie categories.
Community support and en- gagement
Distance-oducation flag, Federal loan recipient rate, Pell grant recipient rate, SAT scores, Undergraduate enrollment
Supportive institutional communitics im- prove retention and completion. Variation in engagement
institution types and student aid profiles induces confounding.
Admission selectivity
Admission rate, SAT (read- ing/math) midpoints, ACT midpoint, Undergraduate cnrollment
Selective admissions correlate with hetter- prepared students and higher completion rates, and vary with institutional prestige and classification.
State and local policies
Region, Poverty rate, Un- employment rate, Cost of attendance
Differences in education funding and pub- lic policy affect cost structures and com- pletion outcomes, interacting with insti- tutional classification and regional demo- graphics.
42

---

Published as a conference paper at ICLR 2026
Table E10: Dataset: ASSISTments. Target: Predict whether student solves a problem correctly on the first attempt in an online learning tool. Environments: Different schools.
Hidden confounder
Affected variables
Reason for confounding
Institutional teaching qual- ity
Hint count, Attempt count, Skill ID, Problem type, Tu- tor mode, Position, Type, First action, Milliseconds first response, Overlap time, Average confidence
Variation instructional quality and ped- agogy across schools affects how effec- tively students engage with content, lead- ing differences in problem-solving strategies, response hehavior, and emo- tional states.
Student motivation
Hint count, Attempt count, Skill ID, Problem type, First action. Milliseconds to first response, Overlap time, Average confidence
Differences in intrinsic motivation across schools influence students' willingness persevere, seek help, or give up quickly, affecting interaction and performance.
Classroom environment
Hint count, Attempt count, Tutor mode, Position, Type, First action. Average confi- dence
Peer dynamics, classroom culture, and noise levels vary by school and affect how confidently and independently students solve problems.
School technology infras- tructure
Hint count, Attempt count, Tutor mode, Position, Type, First action. Milliseconds first response, Overlap time, Average confidence
Access to reliable devices and fast inter- net differs by school, inftuencing response time, tool usage, and student experience.
Teacher-student interaction
Hint count, Attempt count, Tutor mode, Position, Type, First action. Milliseconds to first response, Overlap time, Average confidence
The level of teacher guidance and feed- back shapes how much support students require during problem-solving. affecting engagement and confidence.
Previous academic perfor- mance
Hint count, Attempt count, Skill ID, Problem type, First action, Average con- fidence
Students' prior achievement affects how easily they solve problems, their need for assistance, and their confidence, all of which vary across schools.
43

---

Published as a conference paper at ICLR 2026
Table E11: Dataset: ICU. Target: Predict whether the patient will stay in the ICU for longer than
days. Environments: Insurance types.
Hidden confounder
Affected variables
Reason for confounding
Socioeconomic
status
Age. Gender. Ethnicity, Hoight, Weight, Bicarbon- atc, CO,, pCO,, pO,, Lac- Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, etc.
SES influences access to healthcare, pre- ventive services, and overall hcalth status. Differences in SES across insurance types lead to variability in pre-ICU health, physi- ological indicators, and ICU stay duration.
(SES)
Hospital resources and quality
Age, Gender, Ethnicity, Height, Weight, Bicarbon- ate. CO2, pCO,, pO2, Lac- tate. Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, Heart rate, etc.
Hospital infrastructure and standards affect monitoring, intervention speed, and clinical decisions. These factors vary by insurance coverage and influence ICU out- comes and vitals.
Comorbidities
Age. Gender, Ethnicity, Height. Weight, Bicarbon- ate, CO2. pCO2, pO2. Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, Heart rate, etc.
Presence of chronic conditions (e.g., dia- betes, cardiovascular discase) affects both need for prolonged ICU care and phys- iological measurements. The distribution of comorbidities differs across insurance types.
Insurance-related treatment variability
Age, Gender, Ethnicity, Height, Weight, Bicarbon- ate. CO2, pCO,, pO2, Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, etc.
Differences in treatment timing, intensity, and access to specialists based on insur- ance policies affect ICU stay duration and clinical metrics.
Genetic factors
Age, Gender, Ethnicity, Height, Weight, Bicarbon- ate, CO2, pCO,, pO2, Lac- tate, Sodium, Hemoglobin, etc.
Inherited traits influence predisposition organ failure, metabolic responses, recovery trajectories. These effects are partially mediated by ethnicity and age distributions, which vary across insurance groups.
44

---

Published as a conference paper at ICLR 2026
Table E12: Dataset: Hospital mortality. Target: Classify whether an ICU patient expires in the hospital during their current visit. Environments: Insurance types.
Hidden confounder
Affected variables
Reason for confounding
Socioeconomic
status
Age. Gender, Ethnicity, Height, Weight, Bicar- bonate. Lactate. Sodium, Hemoglobin, Oxygen saturation, Respiratory rate, Systolic blood pressure, White blood cell count, etc.
SES shapes access to timely high- quality care, preventive services, and gen- eral health status. Patients with higher SES often have better insurance and out- comes, leading confounding with mor- tality risk.
(SES)
Comorbidities
Age, Gender, Ethnicity, Height, Weight, Bicarbon- ate, CO2, pCO;, pO2, Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respi- ratory rate, Heart rate, Sys- tolic blood pressure, etc.
Pre-existing conditions such diabetes heart disease increase mortality risk influence physiological features. Their prevalence differs by insurance type, cre- ating confounding.
Hospital resources and quality
Age. Gender. Ethnicity, Height, Weight, Bicarbon- ate. CO2. pCO2, pO2, Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respi- ratory rate, Heart rate, Sys- tolic blood pressure, White blood count, etc.
Access advanced treatments. trained staff, and timely interventions influences survival rates. These factors correlate with insurance coverage, confounding mortal- ity outcomes.
Genetic factors
Age. Gender. Ethnicity, Height, Weight, Bicarbon- atc, CO2, pCO2, pO2, Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, Heart rate, etc.
Genetic predispositions affect disease coptibility and treatment responses. Varia- tions in genetic risk factors may correlate with demographic traits across insurance types.
Lifestyle and hehavioral
Age, Gender, Ethnicity, Height, Weight, Bicarbon- ate. CO2, pCO,, pO,, Lac- tate, Sodium, Hemoglobin, Oxygen saturation, Respira- tory rate, Heart rate, Tem- perature, Systolic blood pressure, White blood cell count, etc.
Behaviors such as smoking, diet, and phys- ical activity affect long-term health and mortality risk. These behaviors vary sys- tematically with SES and insurance erage, influencing both target and physio- logical features.
factors
45

---

Published as a conference paper at ICLR 2026
Table E13: Dataset: Childhood lead. Target: Predict blood lead levels above CDC blood level
reference value. Environments: Poverty-incone ratios.
Hidden confounder
Affected variables
Reason for confounding
Environmental Exposure
Blood lead levels, Country of birth, Age, Race and His- panic origin
Environmental exposure to lead influences blood lead levels, and this varies signifi- cantly
socio-economic groups. Peo- ple in lower PIR groups are more likely to live areas with higher lead contami- nation, which contributes to higher blood lead levels. Furthermore, environmental factors may affect the demographic distri- bution (e.g., country of birth, race).
Access to Hcalthcare
Blood lead levels, Age, Gender, Race and Hispanic origin, Marital status, Edu- cation
Limited access to healthcare, especially in lower PIR groups, means fewer opportuni- ties detection and treatment of lead poi- soning. This results higher blood lead levels, with disparities also influencing de- mographic variables like age, gender, and education. Additionally, healthcare access varies insurance and socio-cconomic status, further confounding the relation- ships.
Diet Nutrition
Blood lead levels, Age, Gender, Race and Hispanic origin
Dietary factors, such as poor nutrition in lower PIR groups, can exacerbate lead absorption. Malnutrition increases the body's susceptibility to lead poisoning, raising blood lead levels. In contrast, higher groups may have better access nutritious foods, lowering lead absorp- tion, thus creating a confounding effect in how socio-econen economic status and race influ- ence lead toxicity.
Housing Conditions
Blood lead levels, Country of birth, Race and Hispanic origin, Marital status
Older housing conditions, which are more prevalent in lower PIR groups, contribute significantly elevated lead exposure (c.g., lead paint, poor plumbing). These living conditions directly influence blood lead levels and can also correlate with de- mographic factors like country of birth, race, and marital status. This introduces confounding, socio-economic status im- pacts both exposure the demographics of affected individuals.
Occupation
Blood lead levels, Age, Race Hispanic origin, Marital status, Education
Certain occupations, which more com- mon among lower PIR groups, involve higher lead exposure (e.g.. construction, manufacturing). Occupational lead expo- sure directly impacts blood lead levels and is often correlated with education, mari- status, and nocio-cconomic status. The varying prevalence of lead exposure by occupation introduces confounding, espe- cially across different PIR groups.
46

---

Published as a conference paper at ICLR 2026
Table E14: Dataset: Diabetes. Target: Predict diabetes. Environments: Preferred race categories.
Hidden confounder
Affected variables
Reason for confounding
Genetic predisposition
Diabetes, BMI High blood pressure, High blood cholesterol
Genetic factors and family history con- tribute to both the onset of diabetes and comorbid conditions like obesity, hyper- tension, and high cholesterol. These ge- netic predispositions can make individuals more susceptible to diabetes, leading confounding as they correlate with other health indicators.
Access to hcaltheare
Diabetes, Physical health, BMI, Hcalthcare coverage, Health checkups
Limited or unequal access to healthcare, especially in marginalized racial groups, leads disparities in diabetes diagno- sis, management, and comorbidity treat- ment. It also influences the frequency of health checkups and access to medica- tions, which can confound the relationship between diabetes status and other health metries.
Dietary habits and food availability
Diabetes, BMI, Physical health, Alcohol consump- tion, Fruit and vegetable in-
Dictary habits, often shaped by socio- economic status and local food environ- ments, influence weight, health behaviors (such alcohol consumption), and dia- betes risk. People lower sociaeconomic strata may have limited access to healthy food options, leading to higher BMI and increased diabetes risk, creating confound- ing effects on health outcomes.
Psychosocial stress and mental health factors
Diabetes, Mental health, Physical health, BMI, Phys- ical activity, Doctor visits
Chronic stress and mental health issues, often higher in marginalized groups, tribute diabetes development and com- plicate its management. These factors also affect physical health (e.g.. weight gain due to stress) and health-seekin seeking behav- iors (e.g.. fewer doctor visits), leading confounding influencing both diahetes risk and its associated variables.
Socioeconomic status be- yond income
Diabetes, Income. Physi- cal health, Healtheare cov- orage, Education level
Socioeconomic factors, such occupa- tion, education, and neighborhood wealth, influence access to healthcare, nutrition, and overall health behaviors. These fac- tors can confound the relationships be- tween diabetes and other vocio-cconomic variables like income and education, they shape opportunities prevention and treatment.
47

---

Published as a conference paper at ICLR 2026
Table E15; Dataset: Sepsis. Target: Predict, from set of fine- fine-grained ICU data, whether a patient will experience sepsis onset within the next 6 hours. Environments: Lengths of ICU stay.
Hidden confounder
Affected variables
Reason for confounding
Infection prevalence in ICU
SepsisLabel, Temperature (Temp), Leukocyte count (WBC), Heart rate (HR),
Higher infection rates in certain ICU units lead to higher probability of sepsis onset (SepsisLabel), These infection rates influence biomarkers such as WBC, HR, and BUN, creating confounding because the unit's 's infection environment affects both the likelihood of sepsis and the ob- served clinical measures.
Blood urea nitrogen (BUN)
Quality of ICU Care
SepsisL.bbc, Fibrinogen concentration (Fibrinogen), Leukocyte count (WBC), Platelet count (Platelets)
Higher-quality care in certain ICUs may lead to earlier identification and treatment sepsis, resulting in more accurate Sep- sisLabel predictions. Additionally, better care could affect biomarkers like fibrino- gen, WBC, and platelets, which are crit- ical in sepsis detection and progression, thereby confounding the relationships be- tween these variables and outcome.
Severity of underlying con- ditions
SepsisLabel, Blood urea ni- trogen (BUN), Creatinine, Lactate, Calcium
chronic conditions (e.g., kidney disease, cardiovascular is- sues) are at a higher risk sepsis and may show abnormal levels in biomark- ers like BUN, creatinine, lactate, and cal- cium. These underlying conditions tribute the SepsisLabel outcome and confound the relationship between the biomarkers and the likelihood of sepsis, varying across ICU units depending on pa- tient population.
Patients with
Patient's socio-cconomic status
Age (Ago), Gender (Gen- der). Leukocyte count (WBC), Fibrinogen con- centration (Fibrinogen), Platelet count (Platelets)
Socio-cconomic factors, such as access to healthcare. can influence both the like- lihood of sepsis and the observed clin- ical biomarkers. For example, patients from lower socio-economic backgrounds may have delayed hospitalizations or in- adequate care, which affects both Sepsis- Label and the progression of sepsis in- dicated by WBC, fibrinogen, and platelet levels.
Slospital-speciine protocols and treatment guidelines
SepsisLabel, Lactate, Glu- cose, Creatinine
Differences in hospital protocols for sepsis treatment, such as timing of interventions choice sepsis bundles, can affect both the SepsisLabel and biomarkers like lactate, glucose, and creatinine. These protocols lead variability in how sepsis is diagnosed and treated across different hospitals ICU units, confounding the relationship between biomarkers and sep- sis outcomes.
48

---

Published as a conference paper at ICLR 2026
Table E16: Dataset: Hospital readmission. Target: Predict whether a diabetic patient is readmitted to the hospital within 30 days of their initial release. Environments: Admission sources.
Hidden confounder
Affected variables
Reason for confounding
Socio-cconomic (SES)
status
Racc, Gender, Age, Payer code, Medical specialty, Number of outpatient vis- its. Number of emergency visits, Number of inpatient visits, Diahetes medication prescribed
SES influences access to hcaltheare, pa- tient demographics, and chronic disease rates. It can also determine the type of received based admission source (e.g., emergency department vs. outpa- tient settings). Differences in healthcare access, such as availability of medications, may impact readmission rates and associ- ated variables like outpatient visits and prescribed medication.
Severity of illness
Primary diagnosis. Sec- ondary diagnosis, Number of diagnoses, Discharge type, Medication changes (e.g., Insulin, Glipizide)
More severe illness increases the likeli- hood of readmission and influences the complexity of diagnoses and the treat- ments administered. The severity of ill- ness may differ based on the admission source (e.g.. emergency versus outpatient), impacting the number and type of diag- noses and treatments prescrihed dis- charge, affecting readmission likelihood.
Access sources
healthcare
Time in hospital, Discharge disposition, Number of pro- cedures, Number of medi- cations, Number of lab tests
Access healthcare resources (e.g.. time hospital, availability of procedures and medications) influences treatment deci- sions and outcomes. Different admission sources may have varying levels of avail- able resources, leading to different lengths of stay, the types of procedures performed, overall treatment quality, which can affect the likelihood readmission.
Paticnt's adherence med- ication
Change in medications, Diabetes medication prescribed, Number of outpatient visits, Number of emergency visits
Adherence prescribed medications of- ten influenced by vecio-cconomic status, which can vary depending on admission source. Non-adhcrena may lead to med- ication changes, affecting diabetes con- trol and subsequent readmission risk. The number outpatient emergency visits can also reflect how well a patient man- ages their diabetes and the likelihood of complications.
Hospital-speciff protocols
Time in hospital, Number of diagnoses, Discharge dis- position, Readmitted
Different hospitals and healthcare sys- tems implement various protocols for dis- charge planning and readmission preven- tion, which can affect readmission rates. These protocols may vary by admission source, where patients admitted via the emergency department may receive differ- ent follow-up instructions and care than those admitted through other channels.
49

---

Published as a conference paper at ICLR 2026
Table E17: Dataset: MEPS. Target: Measure of health care utilization. Environments: Insurance types.
Hidden confounder
Affected variables
Reason for confounding
Socioeconomic
status
Years of education, Em- ployment status, Hourly wage, Paid sick leave, Paid leave to visit doctor, Fam- ily size, Insurance cover- age, Healthcare utilization
SES can influence access healthcare vices, insurance coverage, and the ability use medical services. Insurance types often correlate with SES levels, where in- dividuals with lower SES may be more likely to be insured by government pro- grams (e.g.. Medicaid), which in im- pacts healtheare utilization patterns different SES groups.
(SES)
Healtheare
Paid sick leave. Insurance coverage, Employer offers health insurance, Health- care utilization
Limited access healtheare, such lack insurance or paid sick leave, directly impacts healthcare utilization. The type insurance person holds is often tied to access to various medical services. The level of coverage and accessibility differs across insurance types, influencing health- care hehaviors such as whether patient can afford and utilize healthcare services.
Health behaviors
Perceived health status, Asthma medications, Limitations in physical functioning, Healthcare utilization
Lifestyle factors smoking, alcohol consumption, and physical activity can directly affect health status and health- care needs. Health behaviors differ across groups with different insurance types, and these behaviors contribute to healthcare utilization. Insurance coverage can also be influenced by perceived health status, which varies across insured groups, affect- ing utilization of medical services.
Chronic hcalth conditions
Asthma medications, Per- ceived health status, Limi- tations in physical function- ing, Healtheare utilization
Chronic conditions like asthma or dia- betes increase the need for healthcare ser- vices, lcading to higher healthcare utiliza- tion. People with chronic conditions are more likely be covered by Medicare or Medicaid, which influences the type of insurance they hold and impacts their healthcare utilization and associated fea- tures (e.g.. medications and physical limi- tations).
Regional healthcare infras- tructure
Region. Family Hcalthcare utilization, Paid sick leave, Paid leave visit doctor, Employment status
size,
The quality and availability of healthcare infrastructure vary across regions, which can impact healthcare utilization. un- derserved regions, people may be more reliant on public insurance options like Medicaid, affecting healthcare access and behaviors. The regional differences in healthcare systems can lead disparities access to medical services, insurance coverage, and utilization of healthcare.
50

---

Published as a conference paper at ICLR 2026
Table E18: Dataset: Poverty. Target: Predict household income-to to-poverty ratio. Environments: Citizenship statuses.
Hidden confounder
Affected variables
Reason confounding
Social capital
Income, Unemployment compensation, Disability benefits, Family size, Hous- ing conditions, Household income-to-powee poverty ratio
Stronger social support networks can im- prove access income, benefits, and sources like unemployment or disability compensation. Social capital can vary based on citizenship status, which in turn influences income, housing conditions, and eligibility for benefits, confounding the relationship between household come and poverty ratios.
Workplace discrimination bias
Discrimination in the workplace Household income, Worker's compensation, Unemployment compen- sation, Pension, Family income, Disability benefits
limit opportunities for higher wages, pen- sions, or disability benefits, especially for marginalized groups (c.g.. racial minori- ties, non-citizens). This bias varies by citizenship status and leads biased sociations between income, benefits, and poverty levels.
Healtheare access
Household income, Health insurance premiums, Med- ical expenses, Disability benefits, Family size, Health conditions, Medi- caid/Medicare assistance
to healthcare, particularly non-citizens, can lead higher medi- cal expenses and greater reliance on public health assistance. Citizenship status di- rectly impacts eligibility for public health- care programs, confounding relationships between medical expenses and income, and affecting poverty ratios.
Limited
Access education and skills development
Educational attainment, Household income, Employment status, Unem- ployment compensation, Income from assistance, Savings
Disparities in educational opportunities, often linked to citizenship status, influence income potential and eligibility for govern- ment assistance. These disparities, in turn, affect the household income-to-poverty tio and employment outcomes, creating confounding relationships between educa- tion and income.
Cultural factors
Family size, Household in- come, Savings. Disabil- ity benefits, Healtheare uti- lization, Income assistance, Living arrangements
Cultural factors influence financial agement, family support. and use social programs. These factors can vary across citizenship statuses, leading to dif- ferences in how household income and assistance distributed, and thus founding relationship between income, benefits, financial behavior.
Housing market and conditions
Housing ownership, Hous- ing costs, Family size, Household income, Rent payments, Social security benefits, Medical aid
Housing market conditions, especially rent disparities, can create financial strain and impact the household income-to- poverty ratio. Citizenship status often in- fluences eligibility for housing assistance and rent subsidies. which confounds the relationship between housing costs and come, particularly in areas with significant immigrant populations.
51

---

Published as a conference paper at ICLR 2026
LLM USAGE
In addition to querying LLMs about the semantic meaning of hidden confounding variables ($ E). LLMs are used to aid or polish writing.
52


Reviews:
[{"review_id": "dT6ztfkbRR", "reviewer": "Reviewer_5zx4", "summary": "The established understanding of OOD generalization suggests that learning invariant and causal representations improves generalization. However, recent empirical work has shown two things: 1)ERM often generalizes better than methods designed for OOD generalization 2)Non-causal features can improve performance. This paper gives theoretical justification for these empirical findings. This work shows that under hidden-confounder shift, invariant causal representations could be suboptimal to ERM and using non-causal features. They also empirically verify that several real datasets do have hidden confounder shifts and their OOD performance using various approaches can be explained by this work’s theoretical framework.", "questions": "Why is it surprising/insightful that environment information should give better models? If not, are there any quantitative predictions we can make in practice when E is hidden?\n\nWhy extra insight does the information theory framework provide that the d-separation argument does not?\nOpen to raising the rating if questions are addressed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper provides a neat information theory based framework to understand the OOD performance of predictors under hidden confounding shift\nThe theory explains two empirical phenomenon: 1) Learning invariant representations is not optimal for hidden confounder shift 2)Non-causal features can improve performance\nThe theory does indeed explain the empirical OOD performance of different training algorithms and the different sets of features (causal, arguably causal, all) used. However, in practice E is often hidden and calculating the terms would not be possible.\nThe paper is generally well-written", "weaknesses": "Regarding invariant representations: It is well known that under hidden confounder shift, they would underperform compared to methods that use information about the environment or its proxies. This intuitively makes sense since having information about the environment should help over methods that do not consider that. I believe the primary strength of invariant methods is in cases where the environment in test is out of the support of the train environments. Therefore, I do not see explaining the underperformance of invariant representations in hidden confounder shift setting a substantial contribution.\nRegarding using non-causal features improves performance: The way I intuitively understand this is that if there is a hidden confounder, then we do not observe all causal variables. Therefore, conditioning on the observed causal variables does not d-separate the non-causal ones. Hence, using non-causal features may help. It's unclear what additional insight the information-theoretic framework provides. Moreover, the d-separation argument applies to other hidden variable scenarios as well (e.g., measurement bias).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jeTjxSiqvV", "reviewer": "Reviewer_cZis", "summary": "The paper provides a theoretical and empirical explanation for observed paradoxes in Out-of-Distribution (OOD) generalization, specifically why conventional Empirical Risk Minimization (ERM) models often match or surpass state-of-the-art OOD methods, and why including all available covariates, even non-causal ones, improves OOD accuracy. The core argument states that shifts in hidden confounding variables induce simultaneous distribution shifts that invalidate the assumptions of invariance-focused OOD methods. Under this causal structure, the authors prove that generalization success depends on learning environment-specific relationships. The benefit of non-causal but informative covariates is explained by their ability to act as proxies for the hidden confounders, thereby mitigating the negative effects of the shifts and increasing conditional informativeness. This conditional informativeness measure correlates strongly with both In-Distribution and OOD accuracy across eight real-world datasets, validating the hypothesis that under hidden confounding, exploiting environment-specific information is necessary for robustness.", "questions": "Could the authors elaborate on the architectural properties of XGBoost that make it inherently proficient at capturing the environment-specific relationships required for maximizing conditional informativeness compared to IRM, which attempts to enforce invariance by minimizing penalties associated with environment dependency?\n\nGiven the complexity of feature extraction in computer vision, how would the predictive information decomposition terms (especially variation and feature shift behave in a system dealing with spurious correlations in image data, and what theoretical adjustments might be needed to apply this framework to deep OOD models outside the tabular setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work theoretically justifies the importance of non-causal, informative covariates (XI) by showing they help maximize generalization performance.  Proposition 4.1 demonstrates that adding these variables reduces concept shift and increases conditional informativeness, thereby mitigating the negative impact of unobserved confounders.\n\n\n\nExperiments using synthetic data with known causal structure and extensive testing on eight real-world tabular datasets (TableShift benchmark) consistently confirm the core hypotheses: hidden confounding is prevalent (inducing simultaneous shifts), and conditional informativeness is highly correlated with both ID and OOD accuracy.", "weaknesses": "The core theoretical insights (Theorems 4.2 and related decompositions) are dependent on assuming a very specific, unobserved causal graph. This foundational assumption remains unverifiable in real-world data. This limits the ability of the derived guidance to be applied with certainty, as the true underlying causal structure is unknown.\n\nThe empirical validation is focused almost exclusively on tabular prediction tasks employing relatively simple models like XGBoost and MLP. The role of the feature extractor in complex domains is much more sophisticated. It is questionable whether the conclusion that invariance is insufficient and environment-specific information should be prioritized holds true when the extractor is a large, non-linear model and the data shifts are visual or semantic rather than structural/tabular.\n\nThe authors explicitly state that their primary goal is to explain existing phenomena rather than provide a novel solution to the problem of hidden confounding shift. While the theoretical explanation is robust, the paper primarily offers a lens through which to understand why simple methods already work well, rather than proposing a new method that reliably and optimally maximizes conditional informativeness across generalized shift settings.\n\nThe identification of \"informative covariates\" is critical to the proposed strategy. However, outside of carefully constructed synthetic settings, rigorously identifying these non-causal proxy variables remains a challenge. The reliance on LLM queries for semantic insight into potential confounders in real-world datasets underscores the difficulty of establishing these variables non-qualitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GRhbuXqt2C", "reviewer": "Reviewer_ZFUv", "summary": "This paper provides a comprehensive information-theoretic framework to understand various components of distribution shift in out-of-domain generalization. Based on existing puzzles on why simple ERM outperforms robust/causal methods, the authors point out -- through both careful theoretical analysis and empirical benchmark -- that confounding bias is the key issue that affects generalization performance, and adding non-causal yet confounding-informative variables can improve generalization. The proposed theory centers around a decomposition of the predictive information $I(Y;\\hat{Y})$ into several terms related to environment change and variable-related shift terms.  This result is well explained through several results in representative settings. The proposed theory is corroborated via numerical experiments and provides insights on generalization performance in popular distribution shift benchmark datasets.", "questions": "1. (same as weakness 1) How are the results in Section 5 obtained -- are they from a single dataset (synthetic or real) or merged everything?\n2. The sign consistency metric is interesting. It's natural to expect a high fraction when a decomposition term is related to the performance. However, I'm a bit confused as to how to interpret a metric as low as $0.2$ (e.g., FS). Even for random guess (sth totally unrelated to prediction performance) the metric should be like $0.5$. Does a low metric mean this term predicts the improvement accurately in an opposite direction? If so, does it mean the theory is not accurate?\n3. Given the important role of CI and CS in sign consistency metric, would you suggest optimizing them in training OOD generalization algorithms? \n4. A potential use of the results is to estimate these terms in the existing environments to guide model choice. Could the authors comment on such applications, e.g., could it work and how future method development may involve them?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Understanding out-of-domain generalization is a very important problem, and the paper targets at a key issue in this area, which is the performance gap between robustness-oriented methods and plain ERM. The results in the paper should benefit the community in both understanding and subsequent method development. \n2. The paper is well-written and well-structured, with rich discussion and inspiring insights. \n3. The theory is supported by solid experiments and empirical insights.", "weaknesses": "1. Maybe I missed something but it would be helpful to clarify what datasets the results in Section 5 are from (are they from synthetic data or the TableShift benchmark?). Seems the results are from one single dataset? Or did you merge all the data?\n2. While the authors clarify that this paper aims to provide empirical insights instead of solutions, which is totally fair, it might be useful to suggest some solutions based on the observed results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BRk6POXVWI", "reviewer": "Reviewer_ov4h", "summary": "The paper investigates data generating processes in which confounding of $X$ and $Y$ takes place through an unobserved variable $U$ (‘hidden confounding’, Figure 3). \n\nTheory:\nUsing mutual information (MI) of $Y$ and its prediction $\\hat{Y}$ as objective (O) to be maximised, the authors provide results that show that in cases where either $X \\to Y$ or $Y \\to X$ (but not both), O is equal to the sum of the environment specific MI between $Y$ and learned representations $\\phi(X)$, plus a residual term (will refer to this sum as S below), showing that in such scenarios learning environment specific information is expected to be superior to invariant approaches. \n\nExperiments:\nThe empirical part of the paper focusses on the TableShift benchmark, which contains a range of tabular datasets with distribution shift. According to the authors, most of these datasets follow a data generating process in which either fully or predominantly $X \\to Y$ (but not $Y \\to X$)\nUsing measurements of MI, the authors find changes in conditional MI terms that would be expected under the data generating process the paper considers (Table 3). The authors further evaluate 5 different predictive methods (XGBoost, MLP, GDRO, IRM, VRex) on TableShift and find a positive spearman correlation between average test accuracy of a method and the value of S it achieves in both ID and OOD (Table 2).", "questions": "- Line 76: Why is learning an invariant representation sufficient? The paper seems to suggest the opposite in Eq 1?\n- Line 209: Why is the sum over $\\mathcal{R}^{\\mathcal{D}^e}$ and not simply over $\\mathcal{R}^{e}$? For the other methods in the same paragraph the objective is always stated using $\\mathcal{R}^{e}$. \n- Table 2: why are the authors excluding the results from Figure C4 in the analysis? Adding these would make the estimate of the correlation more robust (and also their value lower I think)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The fundamental problem of distribution shift is relevant and recent findings by Nasal and Hardt (2024) on the effectiveness of ERM make the paper timely \n- The decomposition in Theorem 4.1 is innovative and a clever way to analyse different effects of hidden confounding shift.\n- The derived result in Theorem 4.2 is insightful and relevant\n- The experiments to corroborate the applicability of the assumptions in Propositions 4.1 and 4.2 are generally speaking well designed (but could be improved, see weaknesses)", "weaknesses": "- It would be good if the authors could show how their insights from Theorem 4.2 can be turned into practical algorithms. They mention ideas in the the paragraph “Remarks on conditional informativeness” on page 7, but do not pursue those to derive a method that maximises Eq (4) that is compared to existing methods in the experiment section. One idea could be for example to equip the standard MLP with environment variable(s) similar to Figure 2 (ii), e.g. the feature means or similar and see if this can improve results.\n- Along similar lines, the empirical results in Table that are meant to corroborate practical relevance of Theorem 4.2, are all only observational and based on correlations of measured quantities. Doing some intervention on conditional informativeness - residual values of a method (e.g. actively minimising or maximising the term) and observing changes in accuracy would make a stronger point, or alternatively add some more methods (e.g. logistic regression ERM to keep it simple, or other invariant or domain generalisation approaches) or the same methods with different hyper parameters (e.g as found in Figure C4) to the analysis in Table 2 would make the correlation estimate more reliable. \n- I think Eq (2) and (3) should be derived formally in a proposition, rather than justified in a few words of text as the derivation is not clear to me as such\n- Estimating MI of continuous, high-dimensional variables is a non-trivial problem. It would be good to summarise  in the main text how this was achieved and reference where to look to find details on it in the Appendix\n\n\nMinor:\n- Presentation of results in Table 2: as is, it does not clearly show the correlation between conditional informativeness - residual and accuracy as the main focus is on the test accuracies and the spearman correlations are just somewhat floating around. Why not add the values of conditional informativeness - residual for both ID and OOD in the table to make the picture more complete? Or otherwise you might as well just give the Spearman correlation coefficients in line in the text, and don’t need to use all the space for the table. \n- Proof of theorem 4.2 should be referred to in the main text. \n- An example graph similar to Figure 3 (or an extension thereof) would be helpful to understand definition 4.3 / the concept of informative covariates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)