Boosting for Predictive Sufficiency
Authors: Abbavaram Gowtham Reddy
Rajeev Verma
Celia Rubio-Madrigal
Krikamol Muandet
Rebekka Burkholz
Venue: ICLR 2026 Poster (2026)
Keywords: OOD Generalization
Boosting
Predictive sufficiency
Reference Class
Primary area: interpretability and explainable AI
TL;DR: We explain how boosting can improve OOD generalization under hidden confounding shifts by connecting boosting and predictive sufficiency

Abstract: Out-of-distribution (OOD) generalization is a defining hallmark of truly robust and reliable machine learning systems. Recently, it has been empirically observed that existing OOD generalization methods often underperform on real-world tabular data, where hidden confounding shifts drive distribution changes that boosting models handle more effectively. Part of boosting’s success is attributed to variance reduction, handling missing variables, feature selection, and connections to multicalibration. This paper uncovers a crucial reason behind its success in OOD generalization: boosting’s ability to infer stable environments robust to hidden confounding shifts and maximize predictive performance within those environments. This paper introduces an information-theoretic notion called $\alpha$-predictive sufficiency and formalizes its link to OOD generalization under hidden confounding. We show that boosting implicitly identifies suitable environments and produces an $\alpha$-predictive sufficient predictor. We validate our theoretical results through synthetic and real-world experiments and show that boosting achieves robust performance by identifying these environments and maximizing the association between predictions and true outcomes.

Published as a conference paper at ICLR 2026
BOOSTING FOR PREDICTIVE SUFFICIENCY
Abbavaram Gowtham Reddy', Rajeev Verma", Celia Rubio-Madrigal',
Krikamol Muandet Rebekka Burkholz CISPA Helmholtz Center for Information Security
FUniversity of Amsterdam
ABSTRACT
out-of-distribution (OOD) generalization is a defining hallmark of truly robust and reliable machine learning systems. Recently, it has been empirically observed that existing OOD generalization methods often underperform on real-world tab- ular data, where hidden confounding shifts drive distribution shifts that boosting models handle more effectively. Earlier work attributes a part of boosting's 's suc- cess to variance reduction, handling missing covariates, feature selection, and con- nections to multicalibration. Complementary to these explanations, we uncover a crucial reason behind boosting's ' success in OOD generalization: its ability to identify environments created by hidden confounding shifts and maximize predic- tive performance within those environments. To this end, this paper introduces information-theorrs notion called a-predictive sufficiency and formalizes its connection to OOD generalization under hidden confounding shift. We show that boosting implicitly identifies suitable environments and produces a-predictive sufficient predictor. We validate our theoretical results through synthetic and real- world experiments and show that boosting achieves robust performance by iden- tifying these environments and maximizing the mutual information between pre- dictions and true outcomes.
INTRODUCTION
Generalization beyond the training distribution is central to trustworthy machine learning. Numer- ous methods have been proposed to enhance out-of-distribtiio (OOD) performance on data that differs from the in-distribution (ID) training data (Muandet al., 2013; Arjovsky et al., 2019; Sagawa et al., 2019; Liu et al., 2021c; Zhou et al., 2022; Singh et al., 2024; Yang et al., 2024). These methods typically rely on assumptions such invariance to ensure generalization beyond training environments. In practice, however, factors such as differing data gataraenn processes, selection bias, measurement error, and shifts in unobserved confounding variables often undermine the validity of these assumptions (Fan et al., 2014; Alabdulmohsin et al., 2023; Tsai et al., 2024; Liu & Cui, 2025; Prashant et al., 2025). Consequently, sophisticated methods for OOD generalization often underperform more traditional methods such as boosting, mixture-of offexeerss and multi-layer perceptrons (Gulrajani & Lopez-Paz, 2021; Vedantam et al., 2021; Rosenfeld et al., 2022; Gardner et al., 2023; Liu et al., 2023; Nastl & Hardt, 2024). It is therefore crucial to understand the under- lying mechanisms that enable traditional methods to generalize better under real-world distribution shifts (Fan et al., 2014; Liu & Cui, 2025; Reddy et al., 2026).
The nature of underlying distribution shifts guides the development of generalizable methods. Tra- ditionally, it is assumed that distribution shifts are due to either label shift or covariate shift. Re- cent studies however reveal that hidden confounding shift (confounding variable is a variable that causally influences both label and covariates) also prevalent in real-world data (see $ 3). Based on these assumptions, existing approaches typically consider partitions of the data to capture the inherent heterogeneity of the underlying data distribution. Such partitioning whether specified a priori or defined by experts-senee serves as the basis for different notions of generalization such as in- variance (Arjovsky et al., 2019; Krueger et al., 2021; Creager et al., 2021), robustness (Sagawa et al., 2019), multicalibration (Kim et al., 2019; Wald et al., 2021; Gopalan et al., 2022; Wu et al., 2024a), and predictive information (Reddy et al., 2026).
'These authors share senior authorship.

---

Published as a conference paper at ICLR 2026
A popular approach of defining such partitioning is to assign environment labels using metadata from the data data-collection process. For instance, in housing price prediction, region identifiers such as zip codes or state/country names are commonly used environment labels (Gardner et al., 2023). Similarly, in medical diagnostics, hospital IDs-reflecting differences in equipment, protocols, and patient populations often serve environment labels when training models predict disease out- comes from lab-test data. Data categorization into different subpopulations or environments may lead to different performances (Liu et al., 2021a; Liu & Cui, 2025). When the underlying environ- ment labels are not available or the readily available environment labels do not accurately represent the underlying data heterogeneity, recent methods focus on identifying correct subpopulations so that the invariance relationships between covariates and labels can be learned effectively (Liu et al., 202 la;;; Lin et al., 2022; Liu et al., 2024).
Due to its reliance on the data partitioning, OOD generalization problem is an algorithmic manifes- tation of the reference class problem (Hájek, 2007: Hu, 2025): Given a single case (an individual, an event, a situation), which group or "reference class" should we use to assign the probability to that case? For example, to predict the price of a house in New York City, one may consider the reference class to be the set of all houses in the New York City and attribute their average price as a prediction for the current house. Another possible reference class is the set of houses within a radius of ten kilometers. Hence, the prediction depends heavily on the partition model partitions the data "wrong", e.g. grouping patients by hospital ID rather than disease mech- anism, we may get incorrect predictions for OOD data. This particularly challenging when the distribution shifts are due to shifts in hidden confounders, as we discuss in $ 3.
choose. If the
We hypothesize that ensemble methods, such as boosting, achieve strong OOD performance because they implicitly partition data into reference classes that align with hidden confounding shifts. Specif- ically, the clusters formed by leaf embeddings of boosted trees align with hidden confounding shifts, which contributes to improved robustness under distribution shifts induced by hidden confounding shifts. To formally investigate this, we define an rnformation-theoreee sufficiency. We first show the connection between a--redictive sufficiency and OOD generalization under hidden confounding shifts. We then show that boosting can be viewed as an algorithm that learns a a-predictive sufficient predictor. Since boosting returns a-predictive sufficient predictors, boosting is effective in solving the OOD generalization problem under hidden confounding shift. Unlike the traditional explanations for the success of boosting based methods, explanations fo- cus on the aspect of implicit identification of environments that lead to generalization under hidden confounding shift. As a special case, we extend our theoretical analysis gradient boosting. Our contributions are summarized as follows.
notion called a-predictive
We define an information-theerrtiic notion of a-predictive sufficiency. We then show its con- nection to OOD generalization under hidden confounding shift, expressed in terms of mutual information between ground truth labels and predictions ($ 4).
We show that the boosting algorithm returns a predictor that is a-predictive sufficient and, in doing so, boosting implicitly identifies environments corresponding to hidden confounding shifts ($ 5).
Our experiments on synthetic and real-world data validate our claims that boosting implicitly captures hidden confounding shifts for generalization ($ 6).
2 RELATED WORK
OOD Generalization Under Hidden Confounding Shift. In recent years, dutto--distributio (OOD) generalization under hidden confounding has attracted considerable attention due to its prevalence in real-world data (Landeiro & Culotta, 2018; Alabdulmohsin et al., 2023; Tsai et al., 2024; Prashant et al., 2025). Solutions to this problem often include either adjusting for hidden confounder value (Alabdulmohsin et al., 2023; Tsai et al., 2024) or inferring hidden confounder value (Prashant et al., 2025) under proxy variable assumptions. Because the true confounder (parent of the outcome) is latent, achieving full invariance under hidden confounding is challenging. A prac- tical alternative is to identify regions of the input distribution corresponding to hidden confounding shifts and deploy specialized predictors per region (Reddy et al., 2026).
When model architectures align with the underlying data- data-generating process, generalization may im- prove (Li et al., 2023; Wu et al., 2024b). Motivated by the conditional i f f-tteenee se structure of
2

---

Published as a conference paper at ICLR 2026
visual attributes, Li et al. (2023) propose sparse mixture-of-experts (MoE) models for learning gen- eralizable predictors. More generally, models of the form: P(Y X = PPU X)I(YY X, U), where U is a hidden confounder influencing both covariates X and label Y, can equivalently be interpreted as routing weights P(U X) with per-region invariant predictions TP(Y X, U). Con- sequently, several recent architectures for OOD robustness draw inspiration from MoE designs (Wu et al., 2024b; Prashant et al., 2025). In this work, we study how boosting provides structural in- ductive biases that enable generalization under hidden confounding shift. Although boosting empirically effective in such settings, the mechanisms behind this success remain underexplored.
Boosting and multicalibration. Multicalibration, originally introduced for faimess (Hébert- Johnson et al., 2018), has recently been adapted for invariance learning (Wu et al., 2024a). Prior work establishes strong connections between multicalibration and boosting for regression (Globus- Harris et al., 2023; Wu et al., 2024a) and classification (Kim et al., 2019; Gopalan et al., 2022). Bayestoptimality guarantees for these methods rely on structural assumptions about the grouping functions used for multicalibration. For example, Wu et al. (2024a) show that multicalibration across source environments by grouping data using density-ratio functions between source and target joint distributions also implies multicalibration across target environments. These density ratios act as reference classes and isolate subpopulations most affected by shift, enabling invariance without ex- plicit group annotations. However, this framework addresses covariate shifts in IP(X) and concept shifts in IP(Y X), but does not explicitly accommodate shifts in P(X Y), which often arise un- der hidden confounding shifts. Similar approaches for inferring environments appear in (Liu al., 2021a; Creager et al., 2021). While earlier work shows how level set boosting achieves invariance via multicalibration (Wu al., 2024a), we show how boosting linked maximizing conditional informativeness as well as achieving invariance via predictive sufficiency.
Reference Classes and Predictive Information. Density-ratee groupings discussed above can be viewed as surrogate for partitioning data by unobserved confounder values. While such groupings may not recover latent assignments exactly, they can highlight regions of equal confounder val- ues. Recently, Reddy et al. (2026) showed that, OOD generalization can be framed as maximizing predictive information between true labels and predicted labels across environments that encode hid- den confounding shifts. These environments should correspond to reference classes induced by the confounder values. We show that boosting maximizes predictive information within the reference classes induced by hidden confounding.
3 OOD GENERALIZATION UNDER HIDDEN CONFOUNDING SHIFT
In this section, we provide the necessary back- ground on hidden confounding shifts and ref- erence class to understand the rest of the paper. We use lowercase letters (e.g. .)) for scalars, up- percase letters (e.g. X) for random variables, bold lowercase (e.g. x) for scaler-valued vec- tors, bold uppercase (e.g. X) for random vec- tors, and script letters (e.g. X)) for sets. Fol- lowing previous work (Alabdulmohsin 2023; Tsai et al., 2024; Prashant et al., 2025; Reddy al., 2026), we model hidden con- founding shift across environments using the causal graph G shown in Figure 1, G contains as nodes the covariate vector X, the label Y, the environment variable E, a hidden confounder U, the feature extractor d, and the prediction Y. Directed arrows among X, Y, U, E denote the direct causal influences in the true data dettrgenn process. Directed arrows towards o(X) and Y encode the structural (computational) dependencies that generate representations o(X) from X and predictions Y from ((X). A model's prediction Y can be obtained as Y = (f o (X) where f is a function that makes predictions using the repre- sentations o (X). Let X, N2 2,, & denote the domains of X, Y, U, E respectively. Let E be discrete variable that encodes distribution shifts of the continuous hidden confounder across environments. That is, for any two different environments e, e' E E, we have Ie (U) # F (U). We model E as an unobserved variable because boosting methods do not require access to and work on the pooled data from all environments.
E
U
Y
Φ(X
X
Y
al.,
Figure 1: Causal graph for modeling hidden
founding shifts across environments.
3

---

Published as a conference paper at ICLR 2026
We denote the entropy of a random variable X by (X) = - -Ex (Po(((((())) and the mutual in- formation between two random variables X,Y Y by Y();Y) = Ex,y Dlg F(X,Y) ]- Conditional P(X))()))-
entropy and conditional mutual information are defined similarly. For the theoretical analysis, we consider the mutual information T(Y;Y) between the true label Y and the predictive counterpart Y as the performance measure of a model. Following Federici et al. (2021) and Reddy et al. (2026), we measure the concept shift across environments with /(Y; E Φ (()),, which can also be viewed as a measure of invariance. Concretely, from the causal graph in Figure 1, we assume the causal mecha- nism I X, U)) is invariant across environments: IP (Y X, U) pd (Y X, U) Ve, E. If the representation <(X) encodes the hidden confounder in the sense that there exists a measurable function g with U- = (((())) almost surely, then the causal mechanism of Y given (xX) is invari- ant as well: PT ((X)) = F (Y o(X)). Equivalently, the environment is independent of Y given d(X), the conditional mutual information vanishes: I(Y; E ((X)) O.
We now briefly review key definitions from causal graphical models (Pearl, 2009). A causal directed acyclic graph is a directed graph whose vertices correspond to random variables and directed arrows represent direct causal influences. A path in a graph is a sequence of distinct nodes connected by edges. A path is said to be directed if every edge aligns with the direction of the path. Along such path, one may refer to parents, children, ancestors, and descendants in the usual sense. For instance, in the directed path: X + Y + Z, X is the parent of Y. Y is the child of X, Z is the descendant of X, and X is the ancestor of Z. Any three variables X, Y. Z in causal graph can form one of the three basic substructures: (i) chain (X + Y + 2), (ii) a fork (X +
Y + Z), and (iii) a collider (X - Y < Z). In both chains and forks, X and Z are marginally dependent but become independent conditional on Y. In a collider, X and are marginally independent, yet conditioning on the collider Y (or its descendants) renders them dependent. A path p is said to be blocked by conditioning set B if either (a) p contains a chain or fork with its middle node in B, or (b) p contains a collider such that neither the collider nor its descendants belongs to B. Two nodes are conditionally independent given B only when every path between them is blocked by B.
Hidden Confounding Shift. Distribution shifts are usually modeled through shift in the distri- bution of a hidden variable U and how U influences other observed variables X, Y. For instance, when U -> Y and U A X, we observe lable shift (Tachet des Combes et al., 2020; Garg et al., 2020; Alexandari et al., 2020; Wu et al., 2021). When U A Y and U + X, we observe covariate shift (Gretton et al., 2009; Sugiyama & Kawanabe, 2012: Schneider et al., 2020). In this work, we consider the case where U - Y and U - X, which is more prevalent in real-world data and is referred to as a hidden confounding shift (Alabdulmohsin et al., 2023; Liu al., 2023; Tsai et al., 2024: Prashant et al., 2025; Reddy et al., 2026).
Reference classes. As discussed in $ 1 and $ 2. the notion of reference class is crucial for OOD generalization if the generalization is achieved via multicalibration or predictive information. Here, we first formally state the definition of reference class, and environments induced by those reference classes (Definition 3.1), and then describe the crucial assumption of common confounder support.
Definition 3.1 (Reference Classes and Environments). Let X C TRd be the covariate space and S S (1.....d) d) be an index set. For any x E X, we write xs for the projection of onto the coordinates in S. The reference class of x with respect to S, denoted by [x]s, is defined as [x]s := (x' E xxxs xs). On a finite dataset D = (x!1), (...,.())), }. we define equivalence relation x ~S x - X5 = X's, and the equivalence classes under the relation ris are precisely the reference classes. In D, the collection of all reference classes Es := {[x]s : x DD} forms partition of D, which we refer to as the environment partition of
Assumption 3.1 (Common confounder support (Prashant et al., 2025)). We assume that every con- founder value that occurs in a test environment Ete also occurs in the train environment Ctr i.e., (upp((P(U E = ett)) S supp(P(U)E = exr)). In In finite-sample terms, the set of confounder values possible in the test data is contained in the set of confounder values possible in the train data.
Assumption 3.1 rules out target-only confounder values and ensures that the training data provide examples from all confounder regions that may be encountered at test time. Common confounder support is necessary for generalization because, under the standard invariance assumption that PY x, U) is shared across environments, we have the decomposition: Prte (Y x) = Ev Peee (U x)T (Y U, X). From this decomposition, if Peee (UIX) >> 0 and pptr (U X 0. then pear (Y
4

---

Published as a conference paper at ICLR 2026
U, X) cannot be identified from the training data for those values of U, leading to fundamental limits on generalization when the model encounters unseen confounding patterns at test time.
PREDICTIVE SUFFICIENCY FOR OOD GENERALIZATION
In this section, we use the notion of maximizing predictive information as the target metric for OOD generalization under hidden confounding shift. We then define the notion of a-predictive sufficiency (Definition 4.3) as proof concept that relates to this target metric, and using the decomposition results recently proposed by Reddy et al. (2026), we prove that achieving a-predictive sufficiency linked to achieving generalization under hidden confounding shift (Proposition 4.1). We will leverage these insights to explain the success of boosting methods for OOD generalization under the hidden confounding shift. We begin by formally defining predictive information.
Definition 4.1 (Predictive Information). The predictive information between true outputs Y and model predictions Y is defined as the mutual information I(;;Y).
In a recent work, Reddy et al. (2026) show that predictive information admits an information- theoretic decomposition into interpretable components, namely variation I(o(X); E Y. shift I()(X);E) label shift I(Y; E), concept shift I(Y; E ((X)), conditional informativeness [(o(X); E), and residual I (Ø(();; Y Y). Moreover, under hidden confounding shift, this decomposition simplifies as follows:
I(Y;)) ((;; B) -
IY; ((X;o(X))
Minimizing the residual term implies that the information contained in ((X) utilized by the function f so that residual information is left in ((X). Conditional informativeness motivates maximizing the mutual information between representations and labels within each environment corresponding to the reference class induced by hidden confounder values. Model architectures such as boosting trees and MoE are especially good at modeling such cetrroommet-sppecci require- ments (Wu et al., 2024b; Prashant et al., 2025; Li et al., 2023). Our goal in this work is to explain this puzzling empirical phenomenon and provide a better theoretical understanding of the mechanisms behind the success of boosting methods.
Definition 4.2 (Prediction Error). We define the prediction error as the difference between the true label Y and the model prediction Y: Y Y.
In regression tasks, the quantity Y Y has a natural interpretation as the prediction error between two scalar values. In classification tasks, we treat Y as the predicted probability instead of the predicted label so that Y - y corresponds to the probability residual. To understand how the prediction error manifests in boosting, consider the example of gradient boosting. In gradient boosting, weak learners are trained by fitting pseudo-residuals, defined as the negative gradient of the loss with respect to the model output. These iseudo-residuall coincide with the raw prediction error Y y-1 Ý in common settings such as regression with mean squared loss and binary classification with cross- entropy loss. In the latter case, when the model outputs logits, the resulting pseudo-residuals take the form Y. where Y denotes the predicted probability.
Definition 4.3 (a-Predictive Sufficiency). For 2 O. a prediction Y is a-predictive sufficient for Y across environments E if T(Y Y; Y) S a.
Intuitively, O-predictive sufficiency implies that the prediction error Y is independent of for each outcome value Y. Since E encodes the information about U (a direct parent of Y), achiev- ing predictive sufficiency implies that the predictor Ý relies on X and Y through model training to implicitly account for the impact of U. This helps in learning predictor capable of OOD gener- alization. As a first technical result, we prove that a- a-predictive sufficiency naturally relates to the predictive information in the following proposition.
Proposition 4.1. For covariate vector X, label Y, with causal structure X -+> Y, an environment variable E, feature extractor o, and prediction Y, a-predictive sufficiency of Y for Y across environments E can be decomposed as follows:
T(Y - v) T(Y: E. v) T(Y: (X)) 88 I(Y: (())..
(2)
5

---

Published as a conference paper at ICLR 2026
Proofs are presented in Appendix $ A. Based on the causal graph in Figure 1, I(Y; ((X) E, Y) is
a lower bound on the conditional informativeness [(Y; E) (see Appendix A. for a formal proof). Intuitively, conditioning on does not increase mutual information between Y and ((X) because Y is neither a collider nor belongs to set of descendants of any colliders in the path between Y and ((X). Residual term from Equation appears as in Equation 2, and in addition have the concept shift term, whose minimization is crucial for achieving invariance (Liu et al., 2023; Reddy et al., 2026). From Equations and 2, we note the crucial observation: achieving a-predictive sufficiency for small a is closely linked to maximizing predictive information under hidden confounding shift. Building on this, we now show that gradient boosting returns an a- predictive sufficient predictor.
BOOSTING FOR PREDICTIVE SUFFICIENCY
In this section, we leverage the insights from the previous section to argue that boosting returns a-predictive sufficient predictor. On mechanistic level, boosting iteratively combines several weak learners to form strong learner. Weak learners usually take the form h X - R; E H where H a hypothesis space of weak learners. For example, gradient boosting starts with base predictor (X) = hx(X) (commonly constant), and at each iteration finds a new weak learner hy (X) that best fits the current pseudo-residuals R2 (X, Y) (negative gradients of the loss with respect to the output) and adds (X,(() to the previous predictor 1 to get the current predictor (X)
Y--1 (X), where 7) is the learning rate. To avoid notational clutter, we use R, and hg to denote the random variables (X, Y) and h, (X).
There are several notions of weak learning (Natekin Knoll, 2013; Schapire & Freund, 2013; Mayr et al., 2014; Bentéjac et al., 2021; Globus-Harn et al., 2023; Wu et al., 2024a), but on an intuitive level, a weak learner is a predictor that performs slightly better than random guessing or a constant predictor. In the same spirit, we begin by defining an information-theored weak learner below.
Definition 5.1 (1-approximate weak learner). For any environment E E, we say that the hypothesis class H satisfies the "-approximate weak learning condition the sense that whenever the Bayes predictor Y*. defined as Y*(X) P(Y | X), yields strictly more predictive information than baseline predictor be by a margin y, i.e.
I(Y;Y* (XEE) x(b)) +
where k(be) == I(Y e). Then, there exists an h E H that yields strictly more predictive information than a baseline predictor be by the same margin 7, ie..
1(Y: h(X) X 2 x(b.) +7.
In Definition 5.1, x((b)) = for a constant predictor be. In what follows, we consider the constant predictor b. baseline predictor. In the context of gradient boosting, at each iteration f, the weak learning condition takes the form: T(Y; h4) 2 7, because the entire data is used at each iteration instead of any environment-speeific data. Even though gradient boosting fits pseudo residuals R2 at each iteration, leading to / (Re; he) 2 7, under mild assumptions, we prove that / ((;ht)) 2 holds in gradient boosting (see Appendix B for formal proof). We next state some standard assumptions before stating our main result.
Assumption 5.1 (Existence of reweighted distributions), At each round t, the exists a E (0. such that the boosting algorithm chooses reweighted distributions D.(v) (e.g., based on model out- puts/level sets/residuals) such that IP: = v) 2 p. for all t.
Assumption 5.1 avoids degenerated reweighting at each step and ensures non-trivial information is gained at each round i when combined with Assumption 5.2 described below. In the following, we make a conditional weak-learning assumption to ensure that each weak learner h contributes a nontrivial amount of new predictive information beyond what is already captured by the previous weak learners
Assumption 5.2 (Conditional appproxieatee weak learning). We assume that there exists a constant y such that for all iterations& i, (Y;;,, ho...., hh-11) 2 7.

---

Published as a conference paper at ICLR 2026
Algorithm Standard boosting algorithm
Require: Step size 77, base predictor Yo, hypothesis class H. reweighting rule to obtain Dr
1: Initialize Y, < has & <-
hnn is often constant predictor
2: while training error decreases do
3: Find weak learner h4+1 E H. that achieves I(Y; h (X)) 2 7 under distribution D;
4: Update predictor: V++1 +- Y, + 1
5: Update distribution D,+1 using the reweighting rule and increment by 1.
6: end while
Assumption 5.3 (Strong learner is deterministic function of weak learners), At any round Y, is deterministic function of (hoo-....l.
Assumption 5.3 required to decompose the predictive information I(Y; Ý) into contributions from each individual weak learner. In practice, this assumption holds as the models are usually deterministic once they are trained. For our analysis, we consider the standard boosting algorithm described in Algorithm We now show that this boosting algorithm can return an a-predictive sufficient predictor after training for certain number time steps.
Theorem 5.1. Under Assumptions 5.1-5.3. there exists finite T < such that the predictor Y. learned by Algorithm after t 2 T rounds is a-predictive sufficient. The lower bound is given by
T H(Y) H(Y X, a
p-7
When the environment E is unknown, the same result holds by setting H(Y X, E) 0.
In particular, Theorem 5.1 guarantees that the iterative boosting algorithm converges in finite number of iterations, and lead the a-sufficient predictor.
We now prove the correspondence between a boosting model's 's leaf embeddings and the hidden confounder variable U. Since leaf embeddings in boosting model correspond to both outcome and input representations, we show how boosting achieves H(U Yr) | for some 0 such that after the time step T', the uncertainty in U given the predictions Yr less than or equal to 3. We start with the assumption below.
Assumption 5.4 (Structural preservation of U-information). Let Y, denote the predictor produced by gradient boosting after L iterations. We assume that there exists constant >> 0 for all finite I such that I (U: Y:) 2 c. YY)
Motivated by the causal graph in Figure 1, Assumption 5.4 requires that ((X) and the learning procedure preserve the components of X that carry U-signal, and the predictor actually uses those components rather than discarding them.
Corollary 5.1. Under the Assumptions 5.1-5.4, there exists finite T < such that the predictor Y, learned by Algorithm I after 2 T rounds satisfies HUU (%) < B for small B. The lower bound on given by
T H(U) - 3 3- - I(Y; 8o)
p.
In Appendix B, we show similar theoretical bounds for the gradient boosting algorithm.
EXPERIMENTAL RESULTS
We perform experiments on both synthetic and real-world datasets to empirically explain how boost- ing excels at OOD generalization under hidden confounding shifts. Specifically, we compare the performance of boosting methods in terms of performance, predictive information, and predictive sufficiency. Additional results on both synthetic and Tableshift (Gardner et al., 2023) benchmark datasets are presented Appendix $ C.

---

Published as a conference paper at ICLR 2026
Methods: We experiment two standard boosting algorithms: CatBoost (Dorogush et al., 2018) and XGBoost (Chen & Guestrin, 2016). We use t-SNE (Maaten & Hinton, 2008) and PCA (Abdi & Williams, 2010) dimen- sionality reduction methods for visualizations. We perform hyperparameter tuning to choose the best model when comparing the perfor- mance of models.
CatBoost (e destimators-100 100, max dapth--10) ANN00,10)--0.700 NNB(U,)U))-000e0 942
U valuue used training)
-20 8
Evaluation Metries: We consider the test ac- curacy to evaluate the performance of models in a classification setting, and test mean squared error (MSE) in a regression setting. We use the Adjusted Rand Index (ARI) and the Nor- malized Mutual Information (NMI) to evaluate the goodness of clustering (usually with respect to the hidden confounding variables in synthetic data). We evaluate predictive sufficiency and predictive information and compare the performance of models related to these measures. Mutual information is computed using the Kraskov-Stögbauer Grassberger (KSG) estimator (Kraskov et al., 2004) from the nonparametric entropy estimation tool- box (Steeg & Galstyan, 2011; 2013).
Figure 2: CatBoost representations with respect to hidden confounder value.
Synthetic Experiment 1: We use linear structural equations generate synthetic data following the causal graph: U + X,U + Y,X + Y. U ~ {1, ..... 10), pe is sampled randomly between - 50 and +50, and 0r is set to 0.5. Each environment has 500 samples. We shift 4e by a small fraction to induce a distribution shift where the test data lives. We train CatBoost and XGBoost on this dataset. To get the representations from these boosting methods, we obtain leaf embeddings for each data point as vector of length d, where is the number of trees in the model. We observe that models that achieve low MSE are those whose rep- resentations are aligned with hidden confounder values. As shown in Figure 2, the clusters of representations of a trained CatBoost model align with the hidden confounder value. For similar vi- sualizations for various combinations of dimensionality reduction techniques, values of the number of estimators (trees), and the maximum depth hyperparameters of CatBoost and XGBoost models, see the results in Figures C1-C4 in Appendix C.
tSNE
Usiy X Liing X,S
N N(ee σe) where
600
200
0.60
0.65
0.70
0.75
0.50
0.55
3.90
0.55
1.30
ARI
Figure 3: XGBoost ARI vs. MSE. Markers indicate results for different random seeds.
Synthetic Experiment 2: Next, we consider causal graph of U1 -
S,U, + Y, Ui ~ Miuee oe),, U2 ~ N(u,, oy)) We generate 10 environments, each containing 50 samples. Note that does not causally influence the outcome Y. When XGBoost is only trained with X as input, it fails to capture the underlying shifts in hidden confounder values because neither U2 nor S are obscrved during training; see Figure 3 where ARI is low and MSE is high. How- ever, when XGBoost is trained using both X and S as inputs, we observe low MSE and high ARI. This result explains the observed phenomena that adding additional covariates helps in generaliza- tion performance observed in (Nastl & Hardt, 2024). Any covariate that acts proxy for an unobserved confounder helps improve generalization performance.
Real-world Data California Housing Dataset: In this dataset, the goal is to predict the me- dian house price based on the features median income, house age, rooms, bedrooms, population, occupancy, latitude, longitude. We simulate an artificial hidden confounding shift using observed covariates and the outcome. Specifically, we use a combination of the values of median income, house age, and median house price to induce a distribution shift. Notably, we ensure that the values of hidden confounders at test time belong to the set of hidden confounder values at train time (As- sumption 3.1). The results in Figure 4 clearly show clustering of PCA representations of XGBoost leaf embeddings with respect to hidden confounder values.
8

---

Published as a conference paper at ICLR 2026
Brieliaaa Dan
Figure 4: California housing dataset. XGBoost model representations are clustered according to hidden confounder values. There is common confounder support between the train and test data.
Tean PLA
on Burtinsneer
Figure 5: 20 Newsgroups dataset. XGBoost model representations are clustered according to the hidden confounder values. There is common confounder support between the train and test data.
The goal is to predict the news category of a document from its raw text. Documents are vectorized using TF-IDF and subsequently reduced with Truncated SVD to have 200 features. We simulate an artificial hidden confounding shift between train and test data using a combination of features such as document length, keyword indicator, and the outcome (class index). Similar to the previous experiment, we ensure that the common confounding support (As- sumption 3.1) holds. Figure 5 shows clear association between clustering of PCA representations of XGBoost leaf embeddings and the clusters associated with hidden confounder values.
Real-world Data 20 Newsgroups:
Table 1: Comparison of XGBoost and CatBoost real-world datasets.
California Housing
20 Newsgroups
Method
MSE
Pred. Info.
Pred. Suffi.
Accuracy
Pred. Info.
Pred. Suffi.
XGBoost
0.31 0.00
0.47 0.03
0.00 0.00
62.35 + 0.40
0.27 0.10
0.03 + 0.00
CatBoost
0.00
0.56 0.10
0.00
62.61 0.00
0.68
0.00 0.00
Finally, compare the performance of XGBoost and CatBoost and observe the underlying predictive information and predictive suffi- ciency values. is evident from the results, shown in Table 1, that higher predictive information im- plies better performance, and lower predictive sufficiency value means better performance. These results corroborate our theoretical claims.
Comparison of Performance and Predietive Information:
LIMITATIONS, CONCLUSIONS, AND FUTURE WORK
We reframed OOD generalization under hidden confounding as reference-class inference prob- lem and introduced a-predictive sufficiency as an information-ttherret target that characterizes when predictors transfer across environments. We prove that standard boosting algorithms return a a-predictive sufficient predictors in finitely many rounds, thereby implicitly inferring environments and maximizing predictive information, explaining their strong OOD behavior beyond variance re- duction or feature selection alone. Empirically, across synthetic and real-world tabular tasks, boost- ing's 's learned representations cluster by hidden confounders and achieve high predictive information with low predictive-sufficies mance. These results provide a principled account of why boosting often outperforms specialized OOD methods. We see this work foundation for new OOD algorithms that estimate regu- larize a, relax common-suppon assumptions, and extend mrddicive-sufficcien guarantces beyond tabular settings.
residuals, aligning with the theory and yielding robust OOD perfor-

---

Published as a conference paper at ICLR 2026
ETHICS AND REPRODUCIBILITY STATEMENT
All authors have read and agree to adhere to the ICLR Code of Ethics. This work complies with all ethical guidelines outlined therein. Proofs of the theoretical results presented in the ap- pendix. The code and instructions to reproduce the results are provided in https: //githu.. stw.frcccceteeeefieenieeeeee
ACKNOWLEDGMENTS
The authors gratefully acknowledge the Gauss Centre for Supercomputing e.V. for funding this project by providing computing time on the GCS Supercomputer JUWELS at Jülich Supercom- puting Centre (JSC). We also gratefully acknowledge funding from the European Research Coun- cil (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101116395 SPARSE-ML.
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2010.
Ibrahim Alabdulmohsin, Nicole Chiou, Alexander D' Amour, Arthur Gretton, Sanmi Koyejo, Matt J. Kusner, Stephen R. Pfohl, Olawale Salaudeen, Jessica Schrouff, and Katherine Tsai. Adapting to latent subgroup shifts via concepts and proxies. In International Conference on Artificial Intelli- gence and Statistics, 2023.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat at label shift adaptation. In International Conference on Machine Learning. PMLR, 2020.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: 1907.02990000933, 2019.
Candice Bentéjac, Anna Csörgö, and Gonzalo Martinez-Muñoz. A comparative analysis of gradient boosting algorithms. Artificial Intelligence Review, 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In ACM SIGKDD international conference knowledge discovery and data mining, 2016.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard Zemel. Environment inference for invariant learning. In International Conference on Machine Learning, 2021.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical features support. arXiv preprint 1810. 1830,,1363,,, 2018.
Jianqing Fan, Fang Han, and Han Liu. Challenges of big data analysis. National Science Review, 2014.
Marco Federici, Ryota Tomioka, and Patrick Forré. An information- approach to distribu- tion shifts. In Advances in Neural Information Processing Systems, 2021.
Joshua P Gardner. Zoran Popovi, and Ludwig Schmidt. Benchmarking distribution shift tabu- lar data with tableshift. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A unified view of label shift estimation. Advances in Neural Information Processing Systems, 2020.
Ira Globus- Declan Harrison, Michael Kearns, Aaron Roth, and Jessica Somrell Multicalibra- tion as boosting for regression. In International Conference on Machine Learning, 2023.
Parikshit Gopalan, Adam Tauman Kalai, Omer Reingold, Vatsal Sharan, and Udi Wieder. Om- nipredictors. In nnovations in Theoretical Computer Science Conference, 2022.
10

---

Published as a conference paper at ICLR 2026
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, Bernhard Schölkopf, et al. Covariate shift by kernel mean matching. Dataset shift in machine learning, 2009.
Ishaan Gulrajani and David Lopez-Paz, In search of lost domain generalization. In International Conference on Learning Representations, 2021.
Alan Hajek. The reference class problem is your problem too. Synthese, 2007.
Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Cal- ibration for the commuutioonaly-ddentiiabb masses. In International Conference on Machine Learning, 2018.
Lily Hu. Does calibration mean what they say it means; or, the reference class problem rises again. Philosophical Studies, 2025.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In AAAI/ACM Conference on AI, Ethics, and Society, 2019.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical Review E, 2004.
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribttion generalization via risk extrapo- lation (rex). In International conference on machine learning, 2021.
Virgile Landeiro and Aron Culotta. Robust text classification under confounding shift. Journal of Artificial Intelligence Research, 2018.
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-experts are domain generalizable learners. In International Conference on Learning Representations, 2023.
Yong Lin, Shengyu Zhu, Lu Tan, and Peng Cui. Zin: When and how to learn invariance without environment partition? Advances in Neural Information Processing Systems, 2022.
Jiashuo Liu and Peng Cui. Data heterogeneity modeling for trustworthy machine learning. arXiv preprint arXiv:2506. 00969, 2025.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In International Conference Machine Learning. 2021a.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Kerneiized heterogeneous risk mini- mization. In Advances in Neural Information Processing Systems, 2021b.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: survey, arXiv preprint arXiv:2108. 10008.13624 13624, 2021c.
Jiashuo Liu, Tianyu Wang, Peng Cui, and Hongseok Namkoong. On the need for a language de- scribing distribution shifts: Illustrations tabular datasets. In Advances in Neural Information Processing Systems, 2023.
Jiashuo Liu, Jiayun Wu, Jie Peng, Xiaoyu Wu, Yang Zheng, Bo butional stability among sub-populations. International Conference on Artificial Intelligence and Statistics, 2024.
Bo Li, and Peng Cui. Enhancing distri-
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Llew Mason, Jonathan Baxter, Peter Bartlett, and Marcus Frean. Boosting algorithms gradient descent. Advances neural information processing systems, 12, 1999.
Andreas Mayr, Harald Binder, Olaf Gefeller, and Matthias Schmid. The evolution of boosting algorithms. Methods of information in medicine, 2014.

---

Published as a conference paper at ICLR 2026
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning. 2013.
Vivian Yvonne Nastl and Moritz Hardt. Do causal predictors generalize better to new domains? In Advances in Neural Information Processing Systems, 2024.
Alexey Natekin and Alois Knoll. Gradient boosting machines, tutorial. Frontiers in neurorobotics, 2013.
Judea Pearl. Causality. Cambridge university press, 2009.
Parjanya Prajakta Prashant, Seyedeh Baharan Khatami, Bruno Ribeiro, and Babak Salimi. Scal- able out-o--distribution robustness in the presence of unobserved confounders. In International Conference on Artificial Intelligence and Statistics, 2025.
Abbavaram Gowtham Reddy, Celia Rubio-Madrigal, Rebekka Burkholz, and Krikamol Muandet. When shift happens confounding is blame. In The Fourteenth International Conference on Learning Representations, 2026.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distribution generalization. arXiv preprint 2.Xiv:2202.06856., 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv:1911.0877 2019.
Robert E Schapire and Yoav Freund. Boosting: Foundations and algorithms. Kybernetes, 2013.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in Neural Information Processing Systems, 2020.
Anurag Singh, Siu Lun Chau, Shahine Bouabid, and Krikamol Muandet. Domain generalisation via imprecise learning. In International Conference on Machine Learning, 2024.
Greg Ver Steeg and Aram Galstyan. Information transfer in social media, 2011.
Greg Ver Steeg and Aram Galstyan. Information--heorrtic measures of influence based on content dynamics, 2013.
Masashi Sugiyama and Motoaki Kawanabe. Machine learning non-stationary environments:
Introduction to covariate shift adaptation. MIT press, 2012.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain adaptation with conditional distribution matching and generalized label shift. Advances in Neural Informa- tion Processing Systems, 2020.
Katherine Tsai, Stephen R Pfohl, Olawale Salaudeen, Nicole Chiou, Matt Kusner, Alexander D' D'mourr, Sanmi Koyejo, and Arthur Gretton. Proxy methods for domain adaptation. In In- ternational Conference on Artificial Intelligence and Statistics, 2024.
Shanmukha Ramakrishna Vedantam, David Lopez-Paz, and David J. Schwab. An empirical investi- gation of domain generalization with empirical risk minimizers. Advances in Neural Informa- lion Processing Systems, 2021.
Yoav Wald, Amir Feder, Daniel Greenfeld, and Uri Shalit. On calibration and out-of-domain gener- alization. Advances in Neural Information Processing Systems, 2021.
Jiayun WU, Jiashuo Liu, Peng Cui, and Steven Wu. Bridging multicalibration and out-of- distribution generalization beyond covariate shift. Advances in Neural Information Processing Systems, 2024a.
Qitian Wu, Fan Nie, Chenxiao Yang, Tianyi Bao, and Junchi Yan. GraphSHINE: Training shift- robust graph neural networks with environment inference. In The Web Conference 2024, 2024b.
12

---

Published as a conference paper at ICLR 2026
Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. Advances in Neural Information Processing Systems, 2021.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-diitributioo detection: A survey, International Journal of Computer Vision, 2024.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. JEEE' Transactions on Pattern Analysis and Machine Intelligence, 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A PROOFS OF THEORETICAL RESULTS
Proposition 4.1. For covariate vector X. label Y, with causal structure X Y, an environment variable E, a feature extractor o. and prediction Y, a-predictive sufficiency of Y for across environments E can be decomposed as follows:
I(Y - YY:YEEIN Y I E,Y) I( +)+ I(Y: o(X)).
(2)
Proof. Given Y. the mapping + Y is bijective. Hence I(Y - 8; E Y) = I(Y; E 18).
Hence, by expanding T(Y; E Y), we have the following.
I(Y:El = I(Y (X) Y) I(Y; : ((X) E)
(Introducing 6(X) using the chain rule of mutual information)
= =((Y;((X) Y) T(Y; E Ý, ((X)) T(;; ((X) E, Y)
(Expanding the first term using the chain rule of mutual information)
= =((;o(X)) 1Y) I(Y, E e(X)) I(;; E (x)) ((x)( E.Y)
(Introducing Y using the chain rule of mutual information)
= +(X;;((X) 8) + I(V, E ((X)) - ((X) E,
(I(Y; E ((X)) is zero because Y T E ((X) from the causal graph in Figure
+XX) P) + I(;; E ((X)) + /(; ((X), Y) - ((X)) E, Y)
(Expanding the middle term using the chain rule of mutual information)
=(Y:((X) 8) I (xX)) (( E,Ý)
(I(Y; E 6( is zero because Ý 1 E 6(X), Y from the causal graph in Figure 1)
Theorem 5.1. Under Assumptions 5.1-5.3, there exists a finite < do such that the predictor Y learned by Algorithm after t 2 T rounds is a-predictive sufficient. The lower bound is given by
T - HY X, - I(Y;
p.7
When the environment E is unknown, the same result holds by setting H(Y X, E) 0.
Proof. From Assumption 5.3, Yr is a deterministic function of (ho, 12,,,..... .tr), and T(Y; Yo) = T(Y; because Yo initialized to ho before boosting training loop. Now we have the following:
I I(Y: bo ........ I(Y: Po) + Σ IY: he h--.)
(3)
Consider any round 2 1, and S (hi...,,,,-) 11-1). Then, we have
I(Y: h4 () [T(;; h SS = =)] E,(T(Y: | == s),, 1p.s1pp.. p.y.
(4)
From Equations (3) and (4) we have the following.
7(;; T(Y: Yo) +T
(5)
Given Y.Y - and have one-to-one correspondence. Hence. follows that: I(Y -( I(Y; E Y). We next use the following simple upper bound on I(Y; E Ýr):
()
T(Y; E Yr) H(Y Yr) H(Y Yr. F.) s H(Y (T),
14

---

Published as a conference paper at ICLR 2026
and hence E Y7)= H(Y Yr). Furthermore, since HY 17) H Pr) we have: T( F Yr) & H(Y) T() Now to get T(;; Yr) a, it is sufficient to have H(Y) - _ IT); and following from Equation 5, it follows that
(Y;;))+T+ P 2 H(Y) - a T H(Y) Xo)
P-7
We argue that the our claim will hold if we can show that T is some finite integer. However, it is easy as the right hand side in the above expression is bounded from above by log K-a E pa can always find T N
R+, and hence from the Archimedean property of the real numbers, for which the property holds, thereby arguing the existence of the iteration step T for which the boosting algorithm achieves the argued a-predictive sufficiency. Furthermore,
recall from 6, I(Y; E Yr) H(Y Yr)= H(Y Yr, E) Without removing the term with E, we have F( Vr) = H(Y Fr) H(Y Pr, F) H(V T(Y; Vr) H( Yr, F).
Now to get T(Y; E 18r== S ox, sufficient to have H(Y) - H(Y Fr, E) - a S P(::8p)
I(Y; fo) + Tpy 2 H(Y) H(Y E) E) - a
T H(Y) H(Y Yr, E) - -a- ((Y:%)
P.Y
To avoid the dependence on Ýr, we can replace H(Y Yr, E) with its lower bound H(Y X, E) and still have a valid bound as below.
T2 H(Y) H(Y
aaE))-a-((;;)
P.Y
Corollary 5.1. Under the Assumptions 5.1-5.4, there exists a finite T < cc such that the predictor
Yr learned by Algorithm/ I after 2 T rounds satisfies HU Y%) S B for small B. The lower bound on given by
T= H(U)- c- -o:
c-p-7
Proof. From the proof of the Theorem 5.1, we have the following:
T(Y; 2 I(Y: Vo) + + P
(7)
From Assumption 5.4, we have the following:
I(U)r) 2 (I(Y;Y) +T.p.
(8)
H(U 81) = - < H(U) - c. (I(Y;o)+T.pp. +T.p.7)
(9)
(10)
To ensure H(U 17) < 3, enough to ensure H(U) - c. P(YY;)) +T -..p))) S 8. Solving
this for T implies the desired inequality below:
T 3 3 c. -
(II)
c-p-7
15

---

Published as a conference paper at ICLR 2026
A.1 LOWER BOUND ON CONDITIONAL INFORMATIVENESS
To prove that r(;; x(X) ) E, Y) is a lower bound on I(;; ((X) E), we use the key property from the causal graph shown in Figure 1. Since Y and Y become independent conditional on ((X), we have [(;; Y (X),, , B) = 0 (conditioning on E does not change the independence property). Now Express the mutual information between Y and the pair (e(X), Y) given E in two ways:
| 1(Y; Y B) ( E) + I(; E (X); Y E) = I E) + (Y; E,
(12)
Equating the right hand side terms of the two equations above and using the fact: I(Y; E, (XX) = O, we have the following.
I(Y: E) 0= I0; E)+ E.
(13)
1Y;60X E) - I(Y; (X) E, Y) 1(;; E)
Since T(Y:Y E) 0, we have the desired inequality:
o(Y:o(X) E) 2 ( | E, E)
(14)
B A SPECIAL CASE: GRADIENT BOOSTING
In this section, we apply our analysis in the main paper to a special type of boosting method called gradient boosting. For case of understanding, we repeat some of the text from main paper here. Gradient boosting initializes with base predictor (X(
ho (commonly constant), and at each iteration I, it finds a new base learner h₂ (xx that best fits the current pseudo-residuals (negative gradients of the loss with respect to the output) and adds (X) to the previous predictor
(X) to get the current predictor (X) Algorithm 2). We use the notation R. ((X),, Y) to denote the random variable corresponding to the pseudo-residuals at iteration f the gradient boosting algorithm. To avoid notational clutter, we use RG and h. to denote the random variables RG (X, Y) and h (X).
8,- + (X), where 1) the learning rate (see
Definition B.1 ap-approximate weak learner for gradient-boosting). Let (X) denote the grudient boosting predictor at iteration I, and R2 be the random variable corresponding to the pseudo- residual at iteration with R₁ == 2 (X))
8ービ.. .where is a loss function. Let H. be a class of hypothesis functions h X - TR. and let b(.) be a fixed baseline predictor. We say H satisfies the 7-MI weak learning condition at iteration t there exists by E H. such that the following holds for some 7 >> 0:
aY (X)
I(Rg; hτ) 2 I(Rs; be bx(X)) +
For a constant baseline be(.), (Re by (X)) = 0 and the condition reduces to [RRe h4) 2
a-appoxxmmaee weak learners in gradient boosting: At each iteration é, a weak learner trained to fit the pscudo-residuals RG using the covariates X as input. Assumption 1.1 is stated in information- theoretic terms i.e., mutual information between the weak-learner outputs h1 and the residuals R2 It is therefore important to argue that this information-theonn gradient boosting. Intuitively, if R, were independent of ht then h2 would carry no information about RG and could not provide a weak-learning edge. In practice, however, the weak learner hy obtained by fitting pseudo-residuals and thus typically has nonzero correlation with R. Nonzero correlation implies statistical dependence and hence strictly positive mutual information I [R,; he) > Indeed, because gradient boosting performs functional gradient descent in prediction space at each iteration (Mason et al., 1999), the fitted weak learner h is expected be positively correlated with the pseudo-residuals, i.e., Cov(Re, h4) > 0. Nonzero covariance implies statistical dependence, and hence I [RR;h,) > 0.
weak weak-learming condition holds in
Algorithm presents an information theioottttteertiiiccc interpretation of the standard gradient boosting al- gorithm. For ease of exposition, omit the usual optimal step-size / line-search computation
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Standard gradient boosting algorithm. Comments in orange describe the information- theoretic interpretation of the standard gradient boosting algorithm.
Require:
Training set {(x(1) ( loss function L(y: 9). hypothesis class H. learning rate 7),
number of boosting iterations T.
I: Initialize (X) arg min, (
1 Run for number of iterations where T can be chosen based on Equation 17 to ensure leaf embeddings clustered according hidden confounders. */ ,
2: for iteration / = 1, , do
Compute pseudo-residuals {r 1n where ri = al (L(("),((x()))
3:
OY (x
8 4: Find weak learner h; E H by fitting the weak learner to the pairs {(x satisfies Cov (R,, ht) > 0.
(6) ) that
1 Rg is random variable corresponding to pseudo-residuals. Lemma B.1 shows how weak leamner
that fits pseudo-residuals achieves T T(R;; he) 2 */
5:
Update predictor (x) Y + h
6: end for
7: return (X)
performed after Step 4 in Algorithm 2. Under the common assumption that, if the learning rate is sufficiently small, the effect of the omitted step value can be absorbed into 1), that our theoretical guarantees remain unchanged. We now present lower bounds on 7 TRR; h2) and T(Y; h2). The lower bound [R2;; he) provides I(Y; ht) is needed to prove that gradient boosting returns a a-predictive sufficient predictors.
explicit ? for which Assumption 5.1 holds, while the bound
Lemma B.1. Let Rs and by be the random variables corresponding to residuals and weak learner predictions at iteration of gradient boosting. Let g(R,) supp( R.). + [0, 1] be measurable function. Assuming O Var(h.) Vinax < 0o and (oov(g(R)) (Re), h4) 2
then we have
[[Rg h4) 2 Vinax
2²²
Proof. Since R(R,) is measurable function of R., we have I(R;h,,) 2 [[g(R,); h2). Hence, focus on lower bounding [g(( (R)); he) instead. From the properties of mutual information, we have the following:
1 (g RR):: h.)) = Es, [DLL (R)))) Pg(n))]
(15)
Where DEL denotes the KL divergence and IP, (PP(R)),, ) and IP PP)()) denote the corresponding proba- bility measures. From Pinsker's 's inequality, which relates the KL divergence and total variation (TV) distance, have the following:
(g((R));he) = Ex,, [D (Pg()))),, PI (g)n))
2 2ER, [TV (IP (.Pocc))), (P
(Applying Pinsker's inequality)
Since g(R.) is bounded between 0 and 1, from the properties of total variation distance for bounded functions, we can write as follows:
[gg(Re); h.) 2R,, [(E]g( B[)(()))) (ar((((((RR) he])
(16)
Now, from Cauchy-Sehwarz inequality involving covariance and variances, we have the following: Cov(B[((R.) he], h.)² Var(B RR) he]) Var(he
Var(E(g(R.)) h.])) Z Cov(B(o(R) R.) he], h2)2
Vmax
(oov)((R)),a) h2
Vmax
V
Finally, from Equation equation 16 and the first line of the proof we have (RR; N
((g(R)); hx) 2 V
2n²
17

---

Published as a conference paper at ICLR 2026
Lemma B.1 shows that the weak learner h2 provides information about the pseudo-residual Rg. We next show that h4 also carries nontrivial information about the label Y.
Theorem B.2. Let Re, hy denote the random variables corresponding to residuals and weak learner
predictions at iteration t of gradient boosting. Let g(Y) y - [0. 1 be measurable function.
Let s(X)
R. X] and xx() = [Ygg(Y) X). Assume he satisfies the standard weak learning
condition EE (R 2 p O, and assume local linearity of the expected gradient, i.e., s(X)
a" (x) T7) + ((X) some a E ;",l;aa = À > 0 and for some k. Let fj = E (Y) X]]
and either k(X) = 0 or [[
0. Additionally, assume Var(he) S Vimax DO. Then,
have: :(;; Z mA=Vmar
20²
Proof. From the weak learning assumption about h4 i.e., (R 2 p O, we have the following
E[R = E X [v] = [[)( 2
From the assumption of the linearity of the expected gradient i.e., s(X) a" (x(( 77) k(X). we have the following:
E (1) = a" (X) 7) B = a E h)) 2
If we set [ -e], from Cauchy-Schwarz inequality, we have the following:
a s |||||||||||2 = ||1112 2 e X
Since ||e||2 2 L. then there is atleast one coordinate fig of Ihe that satisfies the following:
||e||2
P
|0g| 2
vm
A/m
Since 1/2 [(7gg (X) Fig 3 h. using 7/ = B[g (Y)
X and fgg Elgg (Y),, we have the following:
14g = E [E[ ( X hh2] [E (V) = L (Y) )b,]
Elgy [ = Cov(g, (Y), h4)
From the Lemma 1. the term c. Following the similar arguments from the previous proof with
2 we have the desired inequality: T(Y; h2) 2 T(g, (Y); hT) 2 2p/AA//)
20º
Vinae
While the exact bound in Theorem B.2 is not crucial for our analysis, the theorem establishes that each weak learner is individually predictive of the label Y, i.e. hy) > O. In the following, make a conditional weak- learning assumption (Assumption 5.2) ensuring that each weak learner h4 contributes nontrivial amount of new predictive information beyond what is already captured by the previous weak learners ho...., h
Assumption 1.1 (Conditional a-approximate weak learning). We assume that there exists a constant 0 such that for all tterations I, I(Y; ht, (ho....,ht..) 2 7.
Assumption B.2 (Strong learner is a deterministic function of weak learners). At any iteration I, we assume that Y, is a deterministic function of he).
Assumption B.2 is required decompose the predictive information I(Y; Y) into contributions from each weak individual learner. In practice, this assumption holds as the models are usually deterministic once they are trained. To ensure that the information present in each new weak learner is retained by the additive aggregation (i.e., the final prediction formed as a weighted sum of weak learners), we make the following assumption.
Assumption B.3 (Monotone information gain). At any iteration t of the gradient boosting algo- rithm, if be satisfies the conditional 7-approximate weak learning condition, we assume that there exists a de E [0, 7), such that I(Y; Y.) 2 I(Y; Y [Y; ht .4-1) Sg. Additionally, we assume that for all T, T 8, < A
We now show that Algorithm returns an a-predictive sufficient predictor after a finite number of boosting iterations.
18

---

Published as a conference paper at ICLR 2026
Theorem B.3. Under Assumptions B.I, 5.3, and B.3, there exists finite T such that the predictor Y, learned by Algorithm 2 after 2 T iterations is a-predictive sufficient. The lower bound is given by
T= H(Y) - H(Y - 1X)
(17)
?
When E is unknown, the same result holds with conservative upper bound obtained by setting H(Y F) =0
Proof. From Assumption 5.3, Yr is a deterministic function of (ho, h1, hey.... ... hty), and I(Y; Y)) ((y:))) because Yo initialized ho before boosting training loop. From Assump- tion B.3, we have the following:
I(Y; 2 1(Y;Yr-1)) I(Y; hr hoo...., hr--)) år
(18)
Consider any iteration 2 Then, there exists gamma such that the following holds.
I(Y; hr loo...., .TT 27
(19)
From Equations (18) and (19) we have the following.
I(: Fr) 2 I(Y; (T-;;--1)+77 dT
(20)
Applying the Assumption B.3 repeatedly for all obtain the following.
Fry; 2 T(: 1)) TT Σ 85
(21)
2 -YY: Yo + +
Given Y, Y- -- and Y have one-to-one correspondence. Hence, follows that: T(Y -Y- () = I(Y; E Y). We next use the following simple upper bound on EY;E Yr):
IY; Pr) H(Y Yr)) H(Y Yr, E) S H(Y 18r)
(22)
and hence I(Y: 17) < H(Y Yr). Furthermore, since H(Y 187) - I(Y; Pr), we
have: T(Y; E Y7)= < H(Y) - T(Y; Yr). Now to get T(Y; F Y7) < a, from Equation equa-
tion 21, it is sufficient to enforce HYY) - a a (o) + +Ty - A as follows.
I(Y: TT A 2 HY)
H(Y) I(Y; vo) +
(23)
T>
Now, from 22, I(Y; E Yr) = H(Y Yr - H(Y Yr, E) Without removing the term with E,
have E(;; Yr) HY Yy) - |Yr, E) 11 (Y) ((;; H1 E).
Now to get T(;; F. Yr)= S a, sufficient to have H(Y) - H(Y Pr, F) a S PY:
T(Y: 9)) + T- A> H(Y) - H(Y 8r, F)
T H(Y) H(Y Ýr. E) a I(Y; Yo) +A
To avoid the dependence on Ýr, we can replace H(Y Ýr, E) with its lower bound H(Y X, E) and still have a valid bound as below.
T> H(Y) H(Y X, E) - a 1(Y +A
(24)
19

---

Published as a conference paper at ICLR 2026
Theorem B.3 guarantees that the gradient boosting algorithm converges in finite number of itera- tions, and lead to the a-sufficient predictor. We now prove the correspondence between a gradient boosting model's leaf embeddings and the hidden confounder variable U. Since leaf embeddings in gradient boosting correspond to both outcome and input representations, we show how boosting achieves (U Yt) S 3 for some 3 2 0 such that after the time step i, the uncertainty in U given the predictions y, is less than or equal to 3. We start with the assumption below.
Motivated by the causal graph in Figure 1, Assumption 5.4 requires that Φ(X) and the learning procedure preserve the components of X that carry U-signal, and the predictor actually uses those components rather than discarding them.
Corollary B.1. Under Assumptions 5.2, 5.3, B.3, and 5.4, there exists a finite T 00 such that
the predictor Ye learned by Algorithm 2 after 2 T iterations satisfies H (Y) & B. The lower bound T is given by
T H(U) 3 cI(Y; Po) +
(25)
e7
Proof. From the proof of the Theorem 5.1, we have the following:
I(Y; Yr) 2 I(Y; (o) + Ty - A
(26)
From Assumption 5.4, we have the following:
T(U Vr) 2 c. (T(Y; + TT-- -
(27)
H(U Yr) H(U) I(;Y)) < H(U) c. (T(Y;Y) + Ty A)
(28)
(29)
To ensure H(U 17) < it is enough to ensure H(U) c. (I(Y; + Tr S Solving
this for I' implies the desired inequality below:
T H(U) 3 - cI(Y;; Yo) + c
(30)
We now discuss how choose a finite T based on the thcorem above. As- suming O, the right-hand sides of Equation equation 23 and Equation equa- tion 24 are bounded above by log K (for discrete Y with K possible values), and by
I 2 ==o
(for continuous Y with variance bounded by o²), where the continuous bound follows from the fact that a Gaussian maximizes differential entropy at fixed variance. Let R de- note the appropriate upper bound (one of the two expressions above). If R < 0 then any T N suffices; otherwise choose T [R] E N By construction T' is finite and satisfies the inequality in equation 17, which establishes the existence of an iteration T after which the boosting algorithm achieves the claimed a-predictive sufficiency.
C ADDITIONAL EXPERIMENTAL RESULTS
Figures C1- C4 show the results with respect to various choices of hyperparameters, dimensionality reduction methods, and metrics. A key takeaway from these results is that better clustering with respect to hidden confounders is consistently associated with better performance. This explains the reason behind the success of boosting methods. Figure C5 shows ID and OOD MSE for different values of distribution shifts. Since high distribution shifts cannot satisfy the common confounder support assumption, the generalization performance drops significantly due to large shifts data distribution. Figure C5 shows performance comparison of XGBoost and CatBoost with invariant risk minimization (IRM) (Arjovsky et al., 2019) and group DRO (Sagawa et al., 2019). We consider the same setup the synthetic experiment 2 presented in the main paper, where the causal graph is: U, + X, U, + Y, X + Y, U2 + S, U2 + Y, Un ~ N(p.. a.), ), U, ~ N(uy of). When only X is used input, we observe that XGBoost and CatBoost perform better than IRM and GroupDRO,
20

---

Published as a conference paper at ICLR 2026
POTE IE CO
ARWLOO MAE
IYSTED 103
LINE TONTOO,
Figure Cl: Results on CatBoost method. Dimensionality reduction done using t-SNE. Results show the goodness of clustering with respect to the hidden confounder value. Above each subplot, clustering metries: ARI, NMI are reported along with hyperparameter choices, ID, and OOD MSE values.
indicating that boosting methods are effective under a hidden confounding shift. When we use both X, S inputs, only GroupDRO performs par with boosting, while IRM still performs worse. This also shows that invariance learning is insufficient for generalization under a hidden confounding shift.
Real-world Experiments: We conduct exper- iments five large-scale tabular OOD bench- mark datasets for binary classification: Read- mission, Foodstamps, Diabetes, Income, and Unemployment. The datasets and their do- main splits are taken from the TableShift bench- mark (Gardner et al., 2023). Prior work shows these datasets exhibit hidden confound- ing when evaluated via observed distribution shifts involving X, and ((X) (Gardner al., 2023; Reddy et al., 2026); therefore, the en- vironment variables provided with these datasets appropriate for assessing
Foodstampa
Calhoest
W0leeet
Reration
Beration
Figure C6: Comparison of accuracy, predictive in- formation, and predictive sufficiency on the Food- stamps dataset.
pro- posed a arerediciive sufficiency. Note that we E. for model training. We show average results computed over 5 random seeds.
21

---

Published as a conference paper at ICLR 2026
POA
ACATE-IA
18323 4-1 20 s00 THE
toat tie o0 1200 dise
ASwil Hiscclt4
Mixt
ARWAE,
Figure C2: Results on CatBoost method. Dimensionality reduction is done using PCA. Results show the goodness of clustering with respect to the hidden confounder value. Above each subplot, clustering metries: ARI, NMI are reported along with hyperparameter choices, ID, and OOD MSE values.
22

---

Published as a conference paper at ICLR 2026
SOKE THIE BUD
1RB-1-16
TORE I9730 AVD
DOME 10-33 4-120
SOME ENTAO EYST)
SES OOD
ARI-5-705, SPE-0.555
U value
NSE-3-L7E 3.50
MSE MSL-2.723
105 Butdd
THULES 10-20 NSE-2 PSE-22244
Figure C3: Results on XGBoost method. Dimensionality reduction done using 1-SNE. Results show the goodness of clustering with respect to the hidden confounder value. Above each subplot, clustering metries: ARI, NMI are reported along with hyperparameter choices, ID, and OOD MSE values.
23

---

Published as a conference paper at ICLR 2026
APEI 0.77, API--SIS
PER 1o BhT
NSC-E990
POR TOLT 9-3
950 MSD-2.05
FOR Te ded
PGC-272
NSE HSC-2.737
FCR T0 dnd
U value
NEE 146 0:0 4.0-4.44
M6-6.746 NMI-1.335
MEE 2233 000 MO0-1.712
ARH-0.700 WHI-0.070
10 000 MODE-O
ARI-9.764 NMIH0.853
ID HEE GOD MSEKS 242
FCR
Mi-6713, 6M1--1.03
All- WMivo 013
ARH- 4M8-6-941
Figure C4: Results on XGBoost method. Dimensionality reduction is done using PCA. Results show the goodness of clustering with respect the hidden confounder value. Above each subplot, clustering metrics: ARI, NMI are reported along with hyperparameter choices, ID, and OOD MSE values.
and 000 MSE vs Shift factor
Shift factor
Figure C5: left: Performance of XGBoost for different shift factor values. Colors indicate different combinations of number of trees, number of samples each domain, and depths of trees. right: Comparison with OOD generalization methods.
24

---

Published as a conference paper at ICLR 2026
Rea desission
Unemployment
AI
Figure C7: Comparison of accuracy, predictive information, and predictive sufficiency Diabetes and Income datasets.
Dlabetes
Inco me
Culffcoot
LnTo and VouR Seft
Cathovd
XOBeest
Becurery
19
050
Figure C8: Comparison of accuracy, predictive information, and predictive sufficiency Diabetes and Income datasets.
From Figure C7, we observe that the predictive information I(Y;Y, for both ID and OOD test data increases as the number of trees/iterations increases. Crucially, I(Y Y;: E Y)) decreases toward zero as increases. Thus, even though E is not used explicitly during training, gradient- boosting methods empirically achieve predictive sufficiency: the prediction errors Y Y, become independent of the environment variables induced by hidden confounders as the model performance converges. From the results, we observe that I(Y - Y;; E Yi) remains zero for some datasets because the available data contain only a single environment. Since E is constant in this setting, the conditional mutual information trivially zero.
Figures C6 and C8 extend the real-world experiments from the main paper. Here we show results on Foodstamps, Diabetes, and Income datasets. Similar to the main paper results, we observe that the predictive information / (Y; Y.) for both ID and OOD test data increases as the ensemble grows (i.e., as the number of trees / iterations / increases). Crucially, T(Y Y;; F Y:) decreases toward zero as t increases. Thus, even though E is not used explicitly during training, gradient-boostin methods empirically achieve predictive sufficiency: the prediction errors Y, become (approximately) independent of the environment variables induced by hidden confounders the model converges. From the results, we observe that T(Y - Y;; F. Yx) remains for some datasets because the available data contain only single environment. Since E is constant in this setting, the conditional mutual information is trivially zero.
25


Reviews:
[{"review_id": "IBEBN1RfYk", "reviewer": "Reviewer_91R9", "summary": "The paper introduces an information-theoretic framework linking boosting-type algorithms to out-of-distribution generalization. It defines the notion of $\\alpha$-predictive sufficiency, based on conditional mutual information, and presents an iterative “information-boosting” procedure that aims to build predictors satisfying this condition. Theoretical guarantees and empirical results on synthetic and real datasets illustrate the proposed connection between boosting and predictive sufficiency.", "questions": "Refer to the weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper proposes an original theoretical connection between boosting and out-of-distribution generalization through an information-theoretic framework. In particular, the introduction of the α-predictive sufficiency concept provides a novel way to formalize robustness to hidden confounding shifts.\n\n- Empirical studies on both synthetic and real datasets offer some qualitative support to the proposed theory.\n\n- The paper is clearly written, with good motivation and intuitive explanations connecting information theory, causality, and ensemble learning. Sometimes the mathematical notation is ambiguous tough.", "weaknesses": "- The use of the term *boosting* is largely metaphorical: Algorithm 1 assumes access to an oracle that exactly maximizes a mutual-information objective, rather than invoking a weak learner trained under a reweighted data distribution. As a result, the procedure cannot be interpreted as a genuine boosting algorithm or as a weak-to-strong reduction.\n\n- The main theoretical result (Theorem~5.1) is deterministic and depends on the assumption that the learner performs an exact maximization step. This removes the probabilistic weak-learning premise that underlies classical boosting theory.\n\n- The notation and treatment of the environment variable $E$ are inconsistent: sometimes $E$ is treated as a random variable, elsewhere as a subset or conditioning event, making several definitions (e.g., Def. 5.1) mathematically ambiguous.\n\n- The quantity $I(Y-\\hat{Y};E\\mid\\hat{Y})$ is introduced without a rigorous definition of the residual term $Y-\\hat{Y}$, which limits the formal interpretability of the proposed $\\alpha$-predictive sufficiency condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pCsR8gl6yz", "reviewer": "Reviewer_ZKii", "summary": "The paper asks why boosting methods often outperform specialized OOD techniques on real‑world tabular data, especially under hidden confounding shift. It reframes OOD as a reference‑class inference problem and argues that boosting succeeds because it implicitly infers “environments” aligned with latent confounder values. It introduces an information‑theoretic target, α‑predictive sufficiency, measuring whether the residual depends on the (latent) environment.", "questions": "1. Can you formalize conditions under which gradient‑boosted tree splitting criteria (e.g., squared‑error or logloss gain) are monotone surrogates for the MI objective in Alg. 1 step 3? Even a lemma relating gain to a lower bound on (I(Y;h(X))) would narrow the theory–practice gap. \n\n2. How exactly is $Y-\\hat{Y}$ instantiated when (Y) is categorical (e.g., indicator of misclassification, one‑hot minus probability vector, or log‑odds residual)? Please specify and justify for the MI/KSG estimators.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Authors recast OOD under hidden confounding as reference‑class inference and formalizes when a predictor “uses the right class” via α‑predictive sufficiency (Def. 4.2). This helps bridge invariance, multicalibration, and MI‑based viewpoints. \nThe analysis offers a compelling narrative: boosting’s reweighting plus diverse weak learners implicitly induce environment‑like partitions aligned with confounder strata—consistent with clustering observed in leaf embeddings. \nAcross synthetic setups and two datasets, increased predictive information and lower sufficiency residuals correlate with better OOD metrics (Table 1), strengthening the paper’s thesis even if not conclusive.", "weaknesses": "1. Alg. 1 chooses weak learners by maximizing (I(Y;h(X))) under reweighting and assumes an oracle. CatBoost/XGBoost instead optimize gradient‑based surrogates on losses; no result shows that those procedures approximate the MI‑oracle selection or that their reweighting schedules satisfy Assumption 5.1. The main theorem therefore does not directly cover the widely used algorithms studied empirically. \n\n2. The definition of predictive sufficiency uses a numeric residual; for categorical (Y), this is ill‑posed without an embedding of labels. The paper empirically studies both regression and classification tasks but does not specify how the residual is defined for the latter (e.g., one‑hot, logits, or 0/1 error event).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DaCBfygYe2", "reviewer": "Reviewer_Pav5", "summary": "This paper introduce  $\\alpha$-predictive sufficiency and argues that boosting algorithms implicitly identify stable “reference classes” lead to predictors with better OOD behavior under hidden confounding shifts. The authors provide theoretical connections between boosting and predictive sufficiency and validate these claims with synthetic and real-world tabular experiments showing boosting’s robustness in such settings.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of $\\alpha$-predictive sufficiency is interesting\n\n2. Robust OOD performance under hidden confounding is a pressing problem for real-world tabular data, where many popular OOD methods underperform; the paper directly targets this gap.", "weaknesses": "1. The authors focus on tabular datasets. It would be helpful to briefly comment on whether the predictive-sufficiency perspective extends to other modalities like Image\n\n2. Comparisons to other OOD approaches.\n\n3. The author can summarize the contribution of this article in detail and the difference from existing methods to make it easier for readers to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cgogBrx48L", "reviewer": "Reviewer_75bL", "summary": "The paper introduces an information theoretic notion called $\\alpha$-predictive sufficiency to measure the generalizatiability of predictive models under hiding confounding shifts. It shows that boosting algorithms gives a predictor that is $\\alpha$-predictive sufficient and they implicitly identifies environments corresponding to hidden confounding shifts.", "questions": "- Why does the mutual information of the residual and the environment variable given the prediction need to be upper bounded by 1 for $\\alpha$-predictive sufficiency in Definition 4.2?\n- What is $c_{E}$ in definition 5.1? Is $c_{G}$ supposed to be written as $c_{E}$?\n- What the set of covariates $\\mathbf{X}$ is defined as the elements of environment $E$ in definition 5.1? How should one interpret it as they are separate nodes in the causal graph in Figure 1?\n- Is $p$ positive in Assumption 5.1? \n- Can the authors explain why assumption 5.1 is realistic?\n- For the derivation of equation 1, it relies on the two equations (eq 2 and eq 3) in Gowtham Reddy et al. 2025. The derivation of those inequalities claims that by conditioning on Y, the mutual information between $\\phi(X)$ and $E$ will increase, and it will be greater than or equal to $I(\\phi(X); E)$, but there is no theoretical support for this claim. A similar issue occurs for other inequalities in eq 2 and eq 3. Can the authors justify these claims to demonstrate the correctness of eq 1 in the submitted manuscript?\n- How is line 747 implied by Equation 4 in the proof in Theorem 5.1?\n- Why did the experiment choose to shift the mean by 0.2 only in the synthetic experiment 1 as indicated by the code? Can the experiment show the effect of the $\\mu_{e}$ for a wider range of values to see how robust the boosting algorithms are?\n- How is the shift implemented in the synthetic experiment 2? What is the role of $S$ in this experiment? Why would XGboost fail even when it is only trained by $X$? Isn’t that the opposite of what the paper claims when the hidden confounder shifts?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is mostly well-written. The introduction gives a very clear picture what problems it is trying to solve and sufficient background information, including the relevant work.\n\n- The observation on achieving $\\alpha$-predictive sufficiency by maximizing the predictive information is insightful. Using $\\alpha$-predictive sufficiency to evaluate the invariance of the predictors is logically sound given that equation 1 holds.  \n\n- Proposition 4.1 seems correct. \n\n- The paper provides codes to reproduce the experimental results.", "weaknesses": "- The paper acknowledges that the idea of $\\alpha$-predictive sufficiency is inspired by by $\\alpha$-approximate multi-calibration. It would be better to include some high-level descriptions about $\\alpha$-approximate multi-calibration in the background. Otherwise, the flow of the paper in lines 282-285 becomes a bit disconnected for someone who does not know anything about $\\alpha$-approximate multi-calibration. \n\n- Definition 5.1 is not quite clear.  For example, it is not clear why definition 5.1 says ‘then, there exists…’ and it also seems to use $c_{E}$ and $c_{G}$ interchangeably. The subscript $G$ does not seem to carry any meaning.  \n\n-  The main claim in Theorem 5.1 is built on the assumptions 5.1-5.3, but the description of the assumptions 5.1 is not clear enough for readers to assess the significance of Theorem 5.1.\n\n- The derivation of the predictive information (Eq. 1) based on a non-peer-reviewed paper seems very questionable. It is built on assumptions that are not described in the submitted manuscript about the information flow between variables. \n\n- The main theorem 5.1 is heavily built on equation 1 in the paper, but equation 1 is derived by Gowtham Reddy et al. 2025. The contribution of the paper seems limited by simply adding assumptions 5.1-5.3 for showing boosting algorithms to be $\\alpha$-predictive sufficient. \n\n- Some steps in the proof of Theorem 5.1 are not clear.\n\n- The instructions in the code says to reproduce Figure 2 by running ‘CatBoost_Cluster_wrt_U.py’, but it doesn’t use 100 estimators as shown in the paper.  It only shows up to 40 estimators with max_depth =5. \n\n- In the synthetic experiment 2, the paper does not explain why the boosting algorithm will need $S$ to improve its performance. \n\n- The real-world data seems synthetic by simulating an artificial hidden confounding shift.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)