Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Authors: Stephen R Pfohl
Natalie Harris
Chirag Nagpal
David Madras
Vishwali Mhasawade
Olawale Elijah Salaudeen
Awa Dieng
Shannon Sequeira
Santiago Eduardo Arciniegas
Lillian Sung
Nnamdi Peter Okechukwu Ezeanochie
Heather Cole-Lewis
Katherine A Heller
Sanmi Koyejo
Alexander Nicholas D'Amour
Venue: NeurIPS 2025 poster ()
Keywords: fairness
distribution shift
causality
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Disaggregated evaluation across subgroups is critical for assessing the fairness of machine learning models, but its uncritical use can mislead practitioners. We show that equal performance across subgroups is an unreliable measure of fairness when data are representative of the relevant populations but reflective of real-world disparities. Furthermore, when data are not representative due to selection bias, both disaggregated evaluation and alternative approaches based on conditional independence testing may be invalid without explicit assumptions regarding the bias mechanism. We use causal graphical models to characterize fairness properties and metric stability across subgroups under different data generating processes. Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis to control for confounding and distribution shift, including conditional independence testing and weighted performance estimation. These findings have broad implications for how practitioners design and interpret model assessments given the ubiquity of disaggregated evaluation.

Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Stephen R. Pfohl
Natalie Harris
Chirag Nagpal Google Research
Google Research spfohl@google.cmm com
Google Research
David Madras Google DeepMind
Vishwali Mhasawade New York University
Olawale Salaudeen
Massachusetts Institute of Technology
Awa Dieng Google DeepMind
Shannon Sequeira Google Research
Santiago Arciniegas The Hospital for Sick Children
Lillian Sung The Hospital for Sick Children
Nnamdi Ezeanochie
Heather Cole-Lewis Google
Google
Katherine Heller Google Research
Sanmi Koyejo Stanford University
Alexander D' D'mouur Google DeepMind
Abstract
Disaggregated evaluation across subgroups is critical for assessing the fairness of machine learning models, but its uncritical use can mislead practitioners. We show that equal performance when data are representative of the relevant populations but reflective of real-world disparities. Furthermore, when data not representative due selection bias, both disaggregated evaluation and alternative approaches based on conditional independence testing may be invalid without explicit assumptions regarding the bias mechanism. We use causal graphical models to characterize fairness properties and metric stability across subgroups under different data generating processes. Our framework suggests complementing disaggregated evaluations with explicit causal assumptions and analysis control for confounding and distribution shift, including conditional independence testing and weighted performance estimation. These findings have broad implications for how practitioners design and interpret model assessments given the ubiquity of disaggregated evaluation.
subgroups is unreliable measure of fairesss
Introduction
significant body of work uses disaggregated evaluation of machine learning models across sub- groups (e.g., by race, ethnicity, or gender) to assess algorithmic faimess properties [1]. In this paradigm, differences in a performance metric (e.g., accuracy, sensitivity, specificity, positive predic- tive value) or other statistical property (e.g. calibration or the distribution of predictions or covariates) across subgroups are taken evidence of fairness violation. healthcare contexts, for example. this approach has been applied to variety of settings to evaluate machine learning models across patient subgroups [2-4], including prognostic models of cardiovascular disease risk [5-7] as well as diagnostic classifiers for medical images [8, 9].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We assume a setting where the policy and faimess goal is consistent with accurate prediction for all subgroups of interest based on historical data, which can be justified in some cireumstances [6, 7, 10-12]. However, we emphasize that fairness is not an inherent property of a model, but rather related to the effect on outcomes that a policy leveraging the model has in deployment context [13], and even accurate models can introduce harm and exacerbate disparities [14, 15].
In this work, we investigate ways that disaggregated evaluation of the performance of predictive models across subgroups can be misleading. A central issue is that models that maximize predictive performance for each subgroup do not generally attain equal performance across subgroups. This follows because the optimal value of a performance metric typically changes under distribution shift [16] and data distributions tend to differ across subgroups contexts where disparities are present (e.g., under differential exposure to social and structural determinants of health (SDOH) [17, 18). We argue that disaggregated evaluation can thus mislead because differences in model performance across subgroups do not necessarily imply misestimation for any subgroup, nor an unfair or inequitable intervention or policy [6, 7, 12, 19]. As a consequence, disaggregated evaluation does not necessarily allow for identification of subgroups for which further data collection or model scaling would be beneficial. Furthermore, algorithmic strategies to develop models that attain equal performance (e.g. constrained optimization or post-processing [2. 20, 21 can directly introduce harm without inducing a contextually meaninngfu notion of faimess [6, 7, 12, 19].
We emphasize that the above considerations apply in settings that are well-speciffed,, that is, when the prediction task is ool-foomulated on the basis of specific use case and target population, and a large and high-quality dataset representative of that target population is available for model development and evaluation [22-24]. If this is not the case, e.g., if the observed data are misrepresentative of the intended target population, model that predicts outcomes well in the observed data may not generalize to the intended target population, potentially directly introducing fairness-related harms when the structure of the misgeneralization is systematic across subgroups. For example, under various forms of bias affecting problem formulation, data collection, and measurement, accurate prediction in the observed data implies structured forms of misestimation in the target population that cannot be detected in the observed data without knowledge of the structure of the bias [22, 23, 25-29) In this work, while we primarily operate in the ideal, well-specified setting detailed above, do extend our analysis to the non-ideal setting to include structured forms of selection bias [25].
We argue that building understanding of the issues discussed above is critical to the design and interpretation of disaggregated evaluations. To that end, we make several contributions, which can be summarized as follows:
We characterize the properties of models learned and evaluated under a variety of data gen- erating processes. We use causal graphical models of distribution shift to encode structured forms of heterogeneity across subgroups and to describe explicit forms of distribution shift through selection bias. This characterizes the fairness properties of models under various assumptions on the data generating process. This approach builds on prior works that use causal directed acyclic graphs to study algorithmic fairness and distribution shift [29-36]. as well as those that study incompatibilities between different notions of fairness [37-40].
We present theoretical results that show that in simple, prototypical cases, average perfor- mance of the optimal predictor is not expected to be the same across subgroups, but that these differences be directly anticipated based on the causal structure of the data generating process. In some cases, performance differences can be directly explained by differences in the marginal distribution of a confounder across subgroups, potentially motivating the use of evaluation procedures that control for such confounding.
We investigate the use of weighted evaluation procedures as means of constructing eval- uations that control for confounding due to distributional differences across subgroups, building off of Cai et al. [16]. We show how such procedures can be interpreted class of configurable conditional independence tests and provide guidance for the interpretation of such techniques in concert with standard disaggregated evaluations.
We conduct experiments with synthetic and real-world data to empirically verify the proper- ties suggested by our theoretical analysis*.
*Code to reproduce the experiments is available at https://github. httpss//ggthbbb.omgooggoerereeee researchh onaluatioo..........

---

2
Preliminaries
We consider data with covariates X E IR" binary label Y E {0, 1}, and categorical subgroup indicator A E A. We reason about models f that take as input Z E (X, {X, A}} } to produce scores
R = =()) that can be compared to a threshold r to yield binary predictions Y = 1 2 r|-
Several of our findings relate to the properties of oracle models f that can be considered to return B|y Z1, such that (Z) = EE|Y | 2). Following Mhasawade et al. [29], we define (X) the population Bayes-optimal model that returns the conditional expectation of Y given covariates X, such that R = (X) = E[y X] and the subgroup Bayes-optimal model as mode Bayes-optimal model can also be represented as a set of subgroup-specifie Bayes-optimal models (ffa JoE for (X) = EY | X, = a)). For arbitrary, potentially non-optimal models, refer to models that only have access X as subgroup-agnostic and those that have access to A as Sabgroup-awarr We refer to fitting separate models for each subgroup as type of subgroup-awan prediction called stratified prediction.
Ra f (X,
B|Y A] The subgroup
Because our scope is limited to modeling binary outcomes, it follows that E[Y | X P(Y = 1 1-11x, X).
This implies that the Bayes-optimal predictor (Z) captures all of the information that Z has about Y. and thus optimal predictors
Z (Z) and IE[Y [Z]] = EE[Y (Z), Z]. Furthermore, Bayes- calibrated, meaning that c(r)
for all [0. 1] for calibration curve c(r) = E R = r]. We note that calibration is necessary but not sufficient for Bayes- -attia-ptimmality: miscalibration implies lack of Bayesooptimality, but calibration does not imply Bayes-optimality.
We assume that the model of interest fit and evaluated based on data drawn from a source distribution over {X, Y, A} given by P(X, Y, A) and evaluated on a targer distribution indicative of the target population for which it of interest to deploy the model. Unless stated otherwise, we assume that the source distribution matches the target distribution. When it is of interest to indicate systematic difference between source and target distributions, we use the convention of selection [25], where we consider model fit using data drawn from the selected, source population P(X, Y, A S 1) and reason about the properties of those models evaluated on samples drawn from the full, target population without selection P(X,
2.1 Algorithmic Fairness and Robustness
It is common to reason about algorithmic fairness through disaggregated evaluation of model per- formance over subgroups of the population or otherwise testing for some form of independence or conditional independence involving subgroup membership. For disaggregated evaluation, we consider metrics m R x y + I that are decomposable (i.e., can be computed at an instance-level and aggregated), where the induced fairness condition is given by EYm,((, R) A a} = Elm, (Y, R)] for some fixed set of subgroups {a)asA and metrics {my} 1.M
is also popular to operationalize fairness with statistical criteria corresponding to independence
or conditional independence. For example, this includes demographic parity (R 1 A) [41, 42], separation (R .L A Y) [20], equalized odds ( 1 A Y) [20], sufficiency (Y 1 A R?) [37],
and predictive parity (Y 1 A Ý 1) [39]. Some of these criteria be directly interpreted as conditions of equal performance across subgroups. For example, separation a sufficient condition for equalized odds, which corresponds to condition where both true positive error rates (also known as sensitivity or recall) and false positive error rates (also known as 1-specificity) are equal across subgroups. Predictive parity likewise corresponds to a case where the positive predictive value (also known as precision) of the model is equal across subgroups. In contrast, sufficiency is more naturally described as a condition of equal calibration curves, rather than as an equal performance condition.
Other works have framed fairness as form of robustness over subgroups [43, 44], conceptualized in terms of worst-case performance across subgroups, with an optimization objective formulated to maximize worst-case subgroup performance. This perspective is aligned with that implicitly taken in disaggregated evaluation, in that some deviation in model performance across subgroups is taken as evidence of disparities in model quality across subgroups.
A number of prior works have documented theoretical and empirical incompatibilities and trade-offs between the notions of fairness outlined above [2, 37-40]. Particularly relevant to this work are the findings of Liu et al. [37], where it is shown that, in settings where the prevalence of the label
3

---

Table 1: Causal graphs encoding assumptions regarding heterogeneity across subgroups. X indicates covariates, Y binary label, A subgroup membership, and S selection.
Covariate Shift
Outcome Shift
Complex Shift
Selection
Causal Direction
Y
A
A
A
S
Label Shift
Presentation Shift
Complex Shift
Selection
Anticausal
X
X
Direction
differs across subgroups (iee, A Y Y), fitting high-quality predictive model for each subgroup implies satisfaction of calibration and sufficiency, but violation of separation and demographic parity, the sense that empirical risk minimization with covariate set that encodes implies some non-zero lower bound on separation and demographic parity error while minimizing an upper bound on calibration and sufficiency error. Other related works [7, 11, 12] show that when allocation on the basis of a predictor is unconstrained and the utility function has certain structure (e.g., if the benefit of the intervention increases as a function the risk of the outcome the absence of the intervention), maximizing prediction performance for each subgroup is consistent with maximizing the utility of the allocation for each subgroup.
3 Understanding fairness through causal models of distribution shift
In this section, we provide the basis for understanding the fairness properties that can be expected for models that fit the data well under a variety of assumptions on the data generating process. These properties described here serve context for the core theoretical results regarding the stability of performance metries and approach controlled evaluation across subgroups that are described subsequently in Section 4. In Section 3.1 we detail the data generating processes of interest that we study in this work. We do so through the specification graphs that incorporate subgroup structure to describe distributional differences across subgroups. In Section 3.2, we characterize the conditional independence properties of subgroup-awan and subgroup-agnostic Bayes-optimal models fit to data drawn from each of the data generating processes of interest.
collection of causal directed acyelic
3.1 Describing subgroup heterogeneity with causal directed acyclic graphs
We define collection of simple, prototypical data generating processes that we use to study the performance and fairness properties of models under varying assumptions regarding causal structure and heterogeneity across subgroups. For that purpose, we use causal directed acyclic graphs (DAGs) [45]. These graphs involve X, Y, and A and are analogous to those used to describe distribution shift mechanisms in prior works [30-35]. We consider graphs in both the causal and anticausal directions, that is, when X is direct parent of Y and when Y is a direct parent of X [46]. While we include A in these graphs to describe the role of heterogeneity across subgroups, we not consider A to be direct cause of either X Y. Rather, confounder that influences X or Y varies distribution across subgroups [47].
use bidirected edges to describe cases where unobserved
The causal settings that we study are presented in Table 1. In brief, in both the causal and anticausal directions, we consider two simple mechanisms of heterogeneity across subgroups, a complex shift mechanism that combines the two simpler ones, and configurable graph that incorporates selection node to describe distribution shift between source and target distribution. To describe selection mechanisms, we augment each of the causal graphs described thus far with square selection node S.

---

The set of variables with direct arrows into S are considered those that affect the selection process. If the set of variables with direct edges into S are deemed Pa(S), the distribution P(S X Y. A) can be simplified to P(S Pa(S)).
For the remainder of this section, we restrict ourself to setting without selection. In the causal direction, the two simple cases are covariate shift and outcome shift across subgroups. Informally, covariate shift captures the assumption that an outcome Y is generated from covariates X with the same mechanism for all subgroups; outcome shift captures the assumption that the mechanism differs. For example, if X is a set of clinical covariates (e.g.. prior health conditions, lab test results, etc.) and Y is downstream clinical outcome (e.g., cardiovascular event in the next ten years), the two settings correspond different assumptions on whether the probability of the clinical outcome differs across subgroups after holding the covariate values constant. In this example. the choice to represent the edges that involve A as bidirected corresponds to an assumption that differences in the distribution of covariates and outcomes across subgroups are not inherent properties of the subgroups but are rather a consequence of differential exposure to unobserved SDOH variables.
More formally, under covariate shift, P(X) & P(X A) but P(Y P | A), encoding the invariance Y T A X. In other words, the distribution of X differs across subgroups but the conditional distribution Y X is unchanged. Under outcome shift, we assume that P(X) P(X A). but there exists some unobserved confounder that affects Y and is not independent of A, such that P(Y X) + P(Y X, A). The complex shift case combines the two settings such that P(X)= * P(X A) and P(Y X) + P(Y A,
In the anticausal direction, the two simple cases are label shift and presentation shift. An example of a plausible anticausal setting is a case where it is of interest identify a condition Y (e.g.. pneumonia) from a chest X-ray [8. 9]. Informally, label shift captures the assumption that conditioned on a class of Y. the distributions of covariates X are identical across subgroups, while presentation shift allows this relationship to change and holds the prevalence of Y constant. For example, this could correspond to setting where is assumed that the distribution of chest X-rays given the presence/absence of pneumonia is the same across subgroups of differing condition prevalence. Formally, in the label shift case, the prevalence of Y differs across subgroups. but P(X Y) is stable across subgroups (i.e., P(X Y)= P(X Y. A)). In the presentation shift case. the prevalence of Y is the same across subgroups, but P(X | Y) differs across subgroups (i.e., P(X Y) # P(X Y, A)). This could occur, for example, if there were systematic differences in the imaging technology used across subgroups. As in the causal direction, complex anticausal shift can be constructed if we let P(Y) & P(Y A) and P(X YY # P(X A).
3.2
The effect of causal structure on the properties of Bayes-optimal models
In this section, we describe the conditional independence properties of Bayes-optimal models learned and evaluated on data drawn from each of the DAGs described in Table 1. We do so to establish the properties that are expected to be satisfied for models that fit the data well, as a foundation for the investigation into the stability of performance metrics across subgroups in Section 4. To derive the key results, we consider the conditional independencies among X, Y, and A that are directly implied by each of the causal graphical structures. This allows us identify cases where the structure of the causal graph implies that the Bayes-optimal predictor satisfies the sufficiency or separation criteria. We note that we focus on sufficient conditions where Bayes-optimality is sufficient to imply the fairness criteria of interest, which are weaker than impossibility results [37-40].
To evaluate the conditions of interest, we use the conditional independencies implied by the relevant DAG and consider scores R as a deterministic function of Z E {X, {X, A}}. For conditional independence statements of the form R 1 V1 V2, for any other variables V1 and V22 we consider R as a deterministic function of Z and use the result that Z 1 Vi I 1/2 implics R 1 V1 V/2 ([48), Lemma 4.2). For Bayes-optimal models, we additionally have the constraint that Y 1 Z R when R R= f is Bayes-optimal.
We present the key results in Supplementary Table B1. We find that while the sufficiency faimesss criterion must be satisfied by a subgroup Bayes-optimal predictor regardless of the causal graph, population Bayes-optimal prediction implies sufficiency only when Y 1 A X (i.e., in the covariate shift setting) or in the case of extreme subgroup separability (Section 3.2.1). The condition Y 1 A X is further central to understanding the properties of evaluations that control for the distribution of
5

---

Table 2: Stability of model performance metrics across subgroups. / indicates cases in which prediction is sufficient to induce {R. Y) A V; indicates otherwise. f indicates prediction with an arbitrary model. f and Lī indicate the population and subgroup Bayes-optimal models.
Setting
[R,Y]
1 A
V
Setting
Z (X, (X, A}}
Model
V
0
X
R
Covariate shift
X
S
X
J
J
X
7
X
X
{x,
A}
f
X
V
X
V
X
X
X
X
Label shift
f
AXK
XXXX
X
f
X
(X,
X
Other Y y A |
X
f"
X
X
X X
X X
Outcome Shift
X X
X
Presentation Shift
(X,4)
X
Complex Shift
X
X
X
X
X (Section 4.2) and the potential for subgroup-aware modeling improve over subgroup-agnostic modeling. In particular, when the causal graph implies Y A X, the population Bayes-optimal predictor also subgroup Bayes-optimal and there no expected benefit subgroup-aware modeling. In contrast, when / A X, the population Bayes-ope Bayes-optimal predictor is not generally subgroup Bayes-optimal, and subgroup-aware prediction may improve performance.
Analysis of the separation criterion serves as an interesting special case for our general findings regarding the stability of performance metrics across subgroups, given the connection between separation and equalized odds (i.e., the control true positive and false positive error rates across subgroups). We find that the separation criterion is implied only by the label shift graph and then only when prediction is subgroup-agnostie This is consistent with prior findings [37] showing that models that fit the data well for subgroups satisfy sufficiency but do not generally satisfy the separation and equalized odds criteria.
regardless of whether the model is Bayes-optimal or arbitrary.
In Supplementary Section A.2, extend this analysis settings with selection bias, enumerating the conditions analogous to those described thus for each of the base graphs augmented with selection mechanism that depends combinations of X, Y, and A (Supplementary Table B2), For brevity, we defer summary of the results to Supplementary Section A.2.
3.2.1 Subgroup Separability
We highlight the properties of a special case where the distribution of covariates are separable across subgroups [9, 49]. In this setting, there is little overlap in the distribution of X for any pair of subgroups a, and aj. such that the ratio P(X]A=a.). is large for any X where P(X A = a;) is non-trivial. This implies that each region of X with non-trivial density is associated with exactly one subgroup. and A can be predicted with high accuracy from X.
P(X
Under subgroup separability, we observe that the population and subgroup Bayes-optimal models behave similarly, as if Y A X, regardless of the underlying causal structure that generated the data. To see this, consider that EJy X = E E| X, A = a) P(A = X) [49]. As such, for X where P(X A) > O, we have that P(A X) ~ 1 and E|Y X x E|y X, A]. We then have that, in the case of covariate shift, the population Bayes-optimal predictor satisfies sufficiency and control for X is enough to explain differences in performance for any performance metric.
4 Understanding disaggregated evaluations of algorithmic fairness
We now present the core results regarding the stability of performance metrics across subgroups for both optimal and arbitrary subgroup-awan and subgroup-agnoste models. We argue that distri-

---

butional differences across subgroups directly contribute to model performance differences across subgroups, constituting form of confounding, distinct from the effects of estimation error. Critically, even Bayes-optimal models do not generally achieve equal performance across subgroups despite zero estimation error. However, we show in Section 4. that the stability of performance metrics across subgroups can be understood as consequence of causal structure. Finally, in Section 4.2, we show how this characterization motivates an approach to controlled evaluation across subgroups.
4.1 Performance metric stability
To characterize the stability of model performance across subgroups for each of the causal settings discussed thus far, we rely on a sufficient condition for equal average performance across subgroups for arbitrary performance metrics. Specifically, we note that {R, YJ -L A is sufficient induce E[m(R, Y) A] = RR, Y)] for any performance metric m..
A key result is that in none of the settings discussed thus far does {Rt YJ T. A hold (Table 2), implying that in general we should not expect equal performance across subgroups for an arbitrary performance metric. This simple observation regarding instability of model performance across subgroups be enriched if we consider the extent to which differences in model performance metrics across subgroups can be explained by causal structure. We do so here through characterization of the condition {R, Y} A V for a control variable V E {X, Y, R) for each of the causal settings discussed, focusing cases where selection bias is not present. The condition {R, Y) I A V then serves as a sufficient condition for [mm(AA R, Y) | V, A| = E[m ( R, Y) V.. We then say that differences in performance across subgroups are explained by V if we have [[m(R, Y) | A] + EE [[m R, Y) but (R,Y] I A V and V L A.
In Table 2, summarize whether control for V E [X, Y, RJ is sufficient to induce (R, YJ 1 A V based on the conditional independencies implied by each of the graphical settings discussed. These results are relevant to both Bayes-optimal and arbitrary models, with some differences depending on whether optimality is achieved. For models that are subgroup-agnostic, we find that X explains performance differences only for the covariate shift graph, and Y explains performance differences only for the label shift graph. However, these relationships no longer hold if the model is subgroup- aware because now {R. Y 1 A V no longer necessarily holds. As an exception, in the covariate shift graph, subgroup Bayes-optimality implies equal performance conditioned on X.
In graphs where Y Y A| X, performance differences are expected in general and neither X nor Y alone explains those differences. However, note that when V = R, {R. Y) 1 A V reduces to the sufficiency fairness criterion. This implies that for any model satisfying sufficiency, differences in performance can be explained by differences in the distribution of the score R.
4.2 Controlled evaluation
The observation that performance differences across subgroups may be attributed to specific dis- tributional differences across subgroups suggests that evaluations that control for this source of confounding may be constructed through balancing of the confounding variable V in comparisons of model performance across subgroups or between subgroups and the overall population. Here, we present an approach to doing so, building off of Cai et al. [16], and discuss the interpretation of such evaluations for algorithmic fairness. Note that such evaluations do not require an assumption of
even if the interpretation can depend on whether the model is optimal. In Supplementary Table B3, we detail the set of implications and conclusions that can be drawn from controlled evaluations in a setting where the causal graph unknown under various assumptions (i.e., the choice of control and covariate variable sets and whether the model is Bayes-optimall
We consider controlled comparison of each subgroup with the overall population. Specifically, compare the performance of a model for subgroup A = a with the performance M,, on the weighted v A a)de, corresponding sampling V
aggregate distribution Q := =f P(R, Y V P(V from the subgroup distribution P(V) A = a) and {R, Y} from the aggregate population distribution P(R, Y [V) = P(R, Y V,A a') P') = a' VVda'. Concretely, M, can be computed via weighted average in the aggregate population with weights w x P(A = a V):
M. /
P(A = v) m(R, Y P))(d)de = = / w w (H, (R, P(o)de.
P(A -0=0 a)
7

---

Weights: None
Weights:
Weights:
Weights:
Covoriate Shitt
Covariates: X Covariates: XA
Outcome Shift-
Complex Causal
Separable
Setting
Label Shift
Group
Presentation Shift
Complex Anti causel
Covoriate
Dutcome Shift-
Complex Causal
Separable
Presentation Shift
Complex Anti cousal
-0.2
0.0
0.2 -0.2
0.0
0.2 0.2
0.0
0.2
0.0
0.2
Tg: Subgroup log loss weighted estimate
Figure 1: Controlled evaluation for confounding across subgroups. Plotted are the statistics Ta with 95% confidence intervals, corresponding to differences between unweighted disaggregated performance with population performance weighted to match the distribution of X, Y. or on the subgroups. The first row corresponds to cubgroup-agnoste prediction and the second row corresponds to subgroup-aware prediction where A is used as a covariate.
We then define Ta == Elm((R, Y) A A a] as the difference between the performance for subgroup A = and M.. In Supplementary Section A. provide additional details regarding this approach and describe alternative configurations of the weights, including the approach of Cai et al. [16] that maps the data region of shared overlap between pair of distributions. Pseudo-code for calculating Ta in setting where P(A = a V) is given or estimated given in Algorithms and 2, respectively.
This form of controlled evaluation is perhaps best understood as a form of conditional independence testing. Consider that if (R, Y} 1. A V. the distribution that M, is computed with respect to is identical to the subgroup distribution, and thus Ta must be zero. It follows that a hypothesis test for whether T. differs from zero corresponds to conditional independence test against the null hypothesis that {R. } 1 A | V. A few specific configurations of V and Z correspond to notable conditional independence tests. For example, when V X and Z X, the test corresponds to test against the null hypothesis 1. A X. Furthermore, if V = R, regardless of whether Z is X or {X, A), the test corresponds
test for sufficiency, {R, Y) 1 A R simplifies Y 1 A R.
For the causal settings we study in this work, T, expected to be zero when V chosen to a variable marked with / in Table 2. Therefore, in setting where the causal structure is not known, observing that Ta is non-zero provides evidence against the set of assumptions and conditions that imply {R, } T A V. For example, observing that Ta for subgroup-agnostee model is non-zero after controlling for X provides evidence against the hypothesis that the data were generated under the covariate shift graph, regardless of whether the model is Bayes-optimal or arbitrary.
Viewing controlled evaluation as a form of conditional independence testing clarifies some challenges related to interpretation. For example, as is shown in Table 2, any gubgroup-agnostie predictor satisfies {R, Y} 1 A X in a covariate shift setting. The implication is that a weighted evaluation procedure for X in covariate shift setting does not distinguish between differences in model performance that would remain under Bayes-optimality from those that follow from poor fit of the model (e.g., due to underrepresentation or model misspecification) for some values of X of differing prevalence across subgroups. Similarly, an evaluation that controls for R inherits the same limitations as a test for sufficiency, in that it is possible for poorly-fit model to satisfy sufficiency [11, 19].
Experiments
We conduct simulation study and experiment with real-world tabular data. The purpose of these experiments is to empirically verify the properties discussed in Section 3. We briefly describe the design of the experiments here and defer additional details to Supplementary Sections A.3 and A.4. For the simulation study, we generate data corresponding to each of the settings described in Table For the real-world data experiments, we follow Ding et al. [50] to derive prediction tasks from the American Community Survey (ACS) Public Use Microdata Sample (PUMS) provided by the
8

---

U.S. Census Bureau [51]. We use the 'ACSIncome' (prediction of whether an individual's 's income exceeds $50, 000) and 'ACSPubiiCovverage (prediction of whether an individual is enrolled in a public health insurance plan) task definitions [50], stratifying the data by race and ethnicity (cohort characteristics provided in Supplementary Table B4).
The properties we assess mirror those discussed in Section 3: we assess properties related to (1) the informativeness of subgroup membership (Section 3.2), (2) calibration and sufficiency (Section 3.2), and (3) controlled evaluation (Section 4.2). To assess properties with respect to the informativeness of subgroup membership, we assess differences in performance between subgroup-aware and subgroup- agnostic models (as in Liu et al. [52]). In the main text, we report results for subgroup- models that leverage A as a covariate and include additional results with stratified models in the supplementary material. To assess calibration and sufficiency violation, we visualize calibration curves for each subgroup and conduct evaluations that control for R. To verify empirical consistency with the properties presented in Table 2 and Supplementary Table B3, we compute Ma and Ta to control for X, Y, or R in the evaluation of model performance.
Simulation study results: The results of the simulation study generally coincide with those that are expected based on the theoretical analysis. We find that subgroup-aware models generally improve predictive performance overall and for subgroups when Y y A X, ie., in all settings except for covariate shift, with the exception of the case where subgroups are separable based on X (Supplementary Figure B1). Specifically, we observe improvements overall and for subgroups in log-loss and non-negative changes in AUC-ROC. We note that sensitivity and specificity do not strictly improve; rather an increase in one is often paired with a reduction in the other. However, we observe that net benefit [53], a decision-theoreee based on an assumed tradeoff between true positives and false negatives, improves like the log-loss.
metric that combines sensitivity and specificity
The effect of subgroup-awand prediction on calibration and sufficiency mirrors the effects on overall model performance (Supplementary Figure B2). In cases where Y L A | X and prediction is subgroup-agnostic we observe miscalibration for subgroups and sufficiency violation. In all cases, subgroup-aware aware prediction results in calibrated models that satisfy sufficiency. As expected, there is no difference in calibration between uubgroup-agnostie and subgroup- models when there is minimal overlap in X across subgroups.
We conduct a small experiment to verify a subset of the properties expected under selection. We adapt the complex causal shift setting and introduce three selection mechanisms corresponding to cases where the selection mechanism depends on X, Y, or A}. Further methodological details are provided in Supplementary Section 1.3.1. We visualize calibration curves in Supplementary Figure B3, finding that they correspond to the properties presented in Supplcmentary Table B2.
We find the results of controlled evaluation are generally consistent with those presented in Table 2. In general, we find that only for the covariate shift graph or when subgroups are separable, control for X is sufficient to control for differences in the average log loss between the population and subgroups for subgroup-agnostic models, consistent with the interpretation of this form of the controlled evaluation as a conditional independence test for Y .. A X (Figure 1). This same pattern holds for other metrics (Supplementary Figures B5-B10). We further find that performance is not generally stable after control for Y. except in the label shift setting, and then only when prediction is auggroup-agnostic. Consistent with the interpretation of control for R as corresponding to test for sufficiency, we note that the statistic T, takes on a value not statistically significantly different from zero in cases where calibration and sufficiency are satisfied and a non-zero value otherwise. For completeness, we report the absolute values of the performance of each of the models of interest in each setting for each subgroup as well as the weighted population estimates M2 (Supplementary Figures B B-17),, For comparison, we further apply the approach of Cai et al. [16] (Supplementary Figures B B18 B18-B24), finding that the results are largely qualitatively consistent with ours.
Results on ACS PUMS: For the experiments with ACS PUMS, we report the change in perfor- mance attained through subgroup- prediction (Supplementary Figure B25), visualize calibration curves (Supplementary Figure B26), and apply the approach to controlled evaluation (Supplementary Figures B27-B40). While the structure of the causal graph underlying these data is not available, our approach can be used for conditional independence testing and attribution of performance differences to distribution shifts. We generally observe greater evidence against the hypothesis Y 1. A X for the 'ACSPublicCoverage task than we do for the 'ACSIncome' task. Specifically, for 'ACSPublic-

---

Coverage', we find that subgroup- aware prediction outperforms subgroup-agnostee prediction for nearly all race/ethnicity subgroups (Supplementary Figure B25) and that differences in performance between subgroups and the aggregate population are not generally explained by control for X; for 'ACSIncome', improvements performance through subgroup-aware prediction are more minor, and differences in performance are explained to a greater degree by X (Supplementary Figure B27). Furthermore, while the extent of miscalibration appears to be minor for both tasks (Supplementary Figure B26), we do observe evidence of sufficiency violation for subgroup-agnosta prediction for both tasks, affecting the "Other" subgroup for 'ACSIncome' and nearly all subgroups for 'ACSPub- licCoverage'. Stratified prediction, but not subgroup prediction with race/ethnicity covariate mitigates the sufficiency violation for the "Other" subgroup for the 'ACSIncome' task, potentially as result of suboptimality due to relative underrepresenntation of this subgroup in the data.
6 Discussion
Our work highlights the challenges of interpretation of disaggregated evaluations over subgroups, con- sistent with findings prior work studying related challenges for evaluation under distribution shift (e.g., Cai et al. [16] and Liu et al. [52]) and incompatibility among fairness criteria [6, 7, 12, 19, 37- 39J. Through vignettes corresponding to prototypical data generating processes represented as causal DAGs, we show how performance differences across subgroups can arise as a consequence of condi- tional dependencies implied by causal structure. This perspective further allows for understanding the properties of evaluations designed to control for confounding across subgroups, as well as the effects that subgroup prediction and subgroup separability have on the properties of models.
Our findings suggest that disaggregated evaluations may be enriched if combined with controlled procedures that effectively match the distribution of set of control variables across subgroups. We interpret such evaluations as a form of conditional independence test that, when rejected, provides evidence towards falsification of assumptions consistent with the conditional independence criteria of interest (e.g., causal assumptions), More broadly, controlled evaluation allows for assessment of hypotheses related causal structure and aids in the assessment of model fit and fairness violation. We provide further guidance as to the interpretation of such evaluations in Supplementary Table B3.
Our work has several limitations that pose opportunities for future work. As our scope was limited to the analysis of binary outcomes, our findings may not generalize to continuous or multi-class outcomes. Furthermore, while the dependence of several of the findings on an assumption of Bayes- optimality clarifies that the challenges related to the interpretation of disaggregated evaluations remain even under optimal prediction, it may limit the extent to which those findings are relevant in practical settings where Bayes-optimaliy analysis of controlled evaluations assumed access to an oracle weight model P(A V), which must be estimated in practice. There is a need for future work to conduct further theoretical and empirical characterization of the role that estimation error has on the properties that we study here, e.g., by casting the evaluation procedure within semiparametric estimation framework that considers the models for Y and A as imperfectly estimated nuisance functions [54, 55].
cannot be assumed nor directly tested for. Relatedly, our
We offer a concrete conceptual reorientation of analytic practice for algorithmic fairness. First, it is important understand differences in optimal model performance across subgroups as a consequence of distributional differences that may be caused by disparities. Observing model performance differences thus motivates deeper investigation to understand the causes of distributional differences and to disambiguate such differences from observational biases (e.g., selection bias) and estimation error. Second, we argue that fairness (as well as related concepts such as equity or justice) is a downstream property of policy or intervention that may leverage a predictive model [13, 58). Subgroup performance differences are then relevant to fairness to the extent that they reflect a contextually-mennt meaningful notion of fairness in the context of the policy. Our results show that if is of interest to model well outcomes that may be disparate across subgroups, we should not in general expect parity in model performance across subgroups. However, we emphasize that the assumptions underpinning this setting should not be taken for granted, and that there are several unresolved normative questions outside of the scope of this work pertaining to the justification for the prediction of disparate outcomes across populations and control for proxies of disparity in evaluations. Designing effective evaluation procedures that are grounded in understanding of both the societal context contributing to inequities and the capacity for interventions and policies that incorporate predictive models to promote equity and fairness goals is critical area of future work.
10

---

Acknowledgments and Disclosure of Funding
This study was funded by Google LLC and/or subsidiary thereof (Google). SRP, NH, CN, DM, AwD, SS, NE, HC-L,, KH, and AID are or were employees of Google and may own stock as part of standard compensation package. SK acknowledges support by NSF 2046795 and 2205329, IES R305C240046, ARPA-H, the MacArthur Foundation, Schmidt Sciences, Stanford HAI, and Google.
References
[1] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limita- tions and Opportunities. fairmlbook.org 2019. http://wwe fairmlbook...om org.
[2] Stephen R Pfohl, Agata Foryciarz, and Nigam H Shah. An empirical characterization of fair machine learning for clinical risk prediction. Journal of biomedical informatics, 113:103621, 2021.
[3] Richard J Chen, Judy Wang. Drew FK Williamson, Tiffany Chen, Jana Lipkova, Ming Lu, Sharifa Sahai, and Faisal Mahmood. Algorithmic fairness in artificial intelligence for medicine and healthcare. Nature biomedical engineering, 7(6):719-742, 2023.
[4] Irene Y Chen, Emma Pierson, Sherri Rose, Shalmali Joshi, Kadija Ferryman, and Marzyeh Ghassemi. Ethical machine learning healthcare. Annual review of biomedical data science, (1):123-144, 2021.
[5] Noam Barda, Gal Yona, Guy N Rothblum, Philip Greenland, Morton Leibowitz, Ran Balicer, Eitan Bachmat, and Noa Dagan. Addressing bias in prediction models by improving subpopu- lation calibration. Journal of the American Medical Informatics Association, 28(3):549-555 558, 2021.
[6] Agata Foryciarz, Stephen R Pfohl, Birju Patel, and Nigam Shah. Evaluating algorithmic faimesss in the presence of clinical guidelines: the case of atherosclerotic cardiovascular disease risk estimation. BMJ Health & Care Informatics, 29 1004660, 2022.
[7] Stephen Pfohl, Yizhe Xu, Agata Foryciarz, Nikolaos Ignatiadis. Julian Genkins, and Nigam Shah. Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1039-1052, 2022.
[8] Laleh Seyyed-Kalantari, Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27 (112)22116-2118 2021.
Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Chen, and Marzyeh
[9] Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy, John L Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, et al. Ai recognition of patient race in medical imaging: modelling study. The Lancet Digital Health, (6):e406-e414 2022.
[10] Donna Arnett, Roger S Blumenthal, Michelle A Albert, Andrew Buroker, Zachary D Goldberger, Ellen J Hahn, Cheryl Dennison Himmelfarb, Amit Khera, Donald Lloyd-Jones, J William McEvoy, et al. 2019 acc/aha guideline on the primary prevention cardiovascular discase: report of the american college of cardiology/americatt heart association task force clinical practice guidelines. Journal of the American College of cardiology. 74(10)xc17-<<232 2019.
[11] Sam decision making and the cost of fairesss. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pages 797-806. 2017.
Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic
[12] Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern. Sam Corbett- Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, et al. Fairness on the ground: Applying algorithmic fairness approaches to production systems. arXiv 2021.
preprint

---

[13] Melissa D McCradden, Shalmali Joshi, James A Anderson, and Alex John London. A normative framework for artificial intelligence as a sociotechnical system in healthcare. Patterns, 4(11), 2023.
[14] Wouter AC van Amsterdam, Nan van Geloven, Jesse H Krijthe, Rajesh Ranganath, and Giovanni Ciná. When accurate prediction models yield harmful self-fulfiliing prophecies. Patterns, 6(4), 2025.
[15] Melissa D McCradden, Mjaye L Mazwi, and Lauren Bakden-Rayner Can an accurate model be bad? Patterns, 6(4), 2025.
[16] Tiffany Tianhui Cai, Hongseok Namkoong, and Steve Yadlowsky. Diagnosing model perfor- mance under distribution shift. arXiv preprint arXiv:2303.020) 1, 2023.
[17] Zinzi D Bailey, Nancy Krieger, Madina Agénor, Jasmine Graves, Natalia Linos, and Mary Bassett. Structural racism and health inequities in the usa: evidence and interventions. The lancet,
2017.
[18] Zinzi D Bailey, Justin M Feldman, and Mary Bassett, How structural racism works racist policies root cause of us racial health inequities. New England Journal of Medicine, 38)( 768-773, 2021.
[19] Sam Corbett- Davies, Johann D Gaebler, Hamed Nilforoshan, Ravi Shroff, and Sharad Goel. The measure and mismeasure of fairness. Journal of Machine Learning Research, 24(312):
1-117, 2023.
[20] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[21] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference on machine learning, pages 60-69. PMLR, 2018.
[22] Samir Passi and Solon Barocas. Problem formulation and fairness. In Proceedings of the conference on fairness, accountability, and transparency, pages 39-48, 2019.
[23] Abigail Z Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 375-385, 2021.
[24] Matthew Sperrin, Richard D Riley, Gary S Collins, and Glen P Martin. Targeted validation: validating clinical prediction models in their intended population and setting. Diagnostic and prognostic research, 6(1):24, 2022.
[25] Elias Bareinboim and Judea Pearl. Controlling selection bias in causal inference. In Artificial Intelligence and Statistics, pages 100-108. PMLR, 2012.
[26] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447-553, 2019.
[27] Heinrich Jiang and Ofir Nachum. Identifying and correcting label bias in machine learning. In International conference on artificial intelligence and statistics, pages 702-712. PMLR, 2020.
[28] Luke Guerdan, Amanda Coston, Zhiwei Steven Wu, and Kenneth Holstein. Ground (less) truth: A causal framework for proxy labels in human-algorithm decision-making. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pages 688-704, 2023.
[29] Vishwali Mhasawade, Alexander D' Ammour, and Stephen R Pfohl. A causal perspective label bias. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1282-1294, 2024.
[30] Victor Veitch, Alexander D' Amour, Steve Yadlowsky, and Jacob Eisenstein. Counterfactual invariance to spurious correlations in text classification. Advances Neural Information Processing Systems, volume 34, pages 16196-16208, 2021.
12

---

[31] Jessica Schrouff, Natalie Harris, Sanmi Koyejo. Ibrahim M Alabdulmohsin, Eva Schnider, Krista Opsahl-Ong, Alexander Brown, Subhrajit Roy, Diana Mineu, Christina Chen, al. Diagnosing failures of fairness transfer across distribution shift in real-world medical settings. Advances in Neural Information Processing Systems, 15:19984-19318 2022.
[32] Elliot Creager, Been-Ke Jacobsen, and Richard Zemel. Environment inference for invariant learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 2189-2200. PMLR, 18-24 Jul 2021.
[33] Maggie Makar, Ben Packer, Dan Moldovan, Davis Blalock, Yoni Halpem, and Alexander D' Amour. Causally motivated shortcut removal using auxiliary labels. In International Confer- ence on Artificial Intelligence and Statistics, pages 739-766. PMLR, 2022.
[34] Maggie Makar and Alexander D' Amour. Fairness and robustness in anti-causal prediction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
[35] Harvineet Singh, Rina Singh, Vishwali Mhasawade, and Rumi Chunara. Fairness violations and mitigation under covariate shift. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 3-13, 2021.
[36] Charles Jones, Daniel C Castro, Fabio De Sousa Ribeiro, Ozan Oktay, Melissa McCradden, and Ben Glocker. A causal perspective on dataset bias in machine learning for medical imaging. Nature Machine Intelligence, 6(2): (22):1381146 2024.
[37] Lydia T Liu, Max Simchowitz, and Moritz Hardt. The implicit fairness criterion of unconstrained learning. In International Conference on Machine Learning, pages 4051-4060. PMLR, 2019.
[38] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint 1xXiv1669.00500 2016.
[39] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5(2): 2017.
[40] Camelia Simoiu, Sam Corbett-Davies, and Sharad Goel. The problem of infra-marginality in outcome tests for discrimination. The Annals of Applied Staristics, 11(3):1 1900)119911111,, 2017. ISSN 19326157.
[41] Toon Calders, Faisal Kamiran, and Mykola Pechenizkiy. Building classifiers with independency constraints. In 2009 IEEE international conference on data mining workshops, pages 13-18. 18. IEEE, 2009.
[42] Cynthia Dwork. Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[43] Shiori Sagawa Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
[44] Emily Diana, Wesley Gill, Michael Kearns, Krishnaram Kenthapadi, and Aaron Roth. Minimax group fairness: Algorithms and experiments. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 66-76, 2021.
[45] Judea Pearl. Causality. Cambridge university press, 2009.
[46] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, page 459-466, Madison, WI, USA, 2012. Omnipress. ISBN 9781450312851.
[47] Thomas Richardson. Markov properties for acyclic directed mixed graphs. Scandinavian Journal of Statistics, 3451)114 2003.
[48] Philip Dawid. Conditional independence in statistical theory. Journal of the Royal Statistical Society Series B: Statistical Methodology, 41(1):1-15, 1979.
13

---

[49] Charles Jones, Mélanie Roschewitz, and Ben Glocker. The role of subgroup separability in group-fair medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 179-188. Springer, 2023.
[50] Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in Neural Information Processing Systems, 34, 2021.
[51] U.S. Census Bureau. American community survey (ACS) public use microdata sample (PUMS), california 5-year person file, 2018
[52] Jiashuo Liu, Tianyu Wang. Peng Cui, and Hongseok Namkoong. On the need for language de- scribing distribution shifts: Illustrations on tabular datasets. In Advances in Neural Information Processing Systems, volume 36, pages 51371-51408, 2023.
[53] Andrew Vickers and Elena B Elkin. Decision curve analysis: novel method for evaluating prediction models. Medical Decision Making, 26(6):565-55) 5744 2006.
[54] Edward H Kennedy. Semiparametric doubly robust targeted double machine learning: review. Handbook of statistical methods for precision medicine, pages 207-236, 2024.
[55] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, and Whitney Newey. Double/debiaseddnemman machine learning of treatment effects. American Economic Review, 107(5):211-265, 2017.
[56] Lily Hu and Yiling Chen. Fair classification and social welfare. In Proceedings of the 2020 conference on fairesss, accountability, and transparency, pages 535-545, 2020.
[57] Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3150-3158. PMLR, 10-15 Jul 2018.
[58] Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. Participatory problem formulation for fairer machine learning through community based system dynamics. arXiv preprint arXiv:2005.07572, 2020.
[59] Paul R Rosenbaum and Donald B Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1):41-55, 1983.
[60] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F.. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
[61] Edwin B Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Staristical Association, 22(158):209-211 1927.
[62] Skipper Seabold and Josef Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
14

---

Supplementary Material
A
Supplementary methods
A.1 Weighting approaches to controlled evaluation
The approach to weighting model performance metries to control for distributional differences across subgroups builds on general approach to estimating model performance under distribution shift. For a source distribution P and target distribution Q over {R, },, performance on Q can be estimated using data from P with appropriate weights. Formally, this is Eg Eg(mm(Y, [t]] = Ep[w m(Y, R)] (R.)Y)
for w x (R) assuming positivity (Po(R, Y) > + Pp(R. Y) > 0). From some mixture distribution over P and Q, where examples from Q are indexed by D = 1 and those from P by
P(R.YD==1) x P(D = R)
D = 0, we note that the weights can be reformulated as w P(R,Y
As described in Section 4.2, we construct controlled evaluations by setting the distribution of variable V to a reference distribution. We consider a source distribution P and target distribution Q over V such that we are computing the performance in P after fixing the marginal to Pe (V).. The
resulting expectation is given by f w m( (R, Y Pp for weights w x PP(V) x P(D = [V,, where D indicates the identity of the distribution, as before.
Po(V)
In this work, we primarily consider setting p P(R, Y) and Q = P(R,Y -- a) with weights w x P(A a V). This yields a mapping of the marginal population distribution P(V) onto the subgroup distribution P(V A = a). implicitly fixing the conditional distribution of P(R, Y V) to that of the aggregate population. It follows that if {R, Y) 1 V. then Pp Y v) = Po R, Y v) and thus weighted performance Me
w m(R, (), Y u))(v) in P is equal to the marginal performance (R, Y) A = a in Q. If weighted performance in P is not equal to the marginal performance in Q, then {R. ) L VV..
is possible to construct alternative weight configurations that can be used for the same purpose as the approach that we present. For example, if the distributions are set such that P = P(R. Y A = a) and Q = PRR,Y. Y), then the weights are given by w x P(A=a] V) corresponding to a mapping of the subgroup A = a onto the population distribution, analogous to approaches to inverse propensity score weighting for treatment effect estimation [59]. Alternatively, if we consider a pair of subgroups A = a and A = a' where P = P( R, Y A = a) and Q = P(R, Y A = a'), the weights are given P(A-2'IV) by w X V[ V). However, both of these formulations
susceptible to returning unstable and high-variance estimates when small values in the denominator induce extreme weights. We note that our approach is not susceptible to this issue given that P(A = a V) is bounded between O and 1.
Cai et al. [16] presented alternative weighting strategy for pairwise comparisons motivated to ad- dress the extreme weights issue by considering mapping the data to a shared distribution of sufficient overlap between the distributions of covariates for a pair of subgroups. This approach considers target distribution @ x P[V
P(VIA==))(V]A=a)) for two source distributions Pa and Par This has the
effect of defining a target density that takes on a value of zero in cases where the control variable has zero support in either of the subgroup distributions. The weights for this approach are given by P(A=a'IV)
P(A=a w x P(A=a))(A= P1 for A = and w x P(A=a)P(A=o P for A d'
A.2 Properties under selection
In this section, we consider fairness properties under explicit distribution shift through the lens of selection bias [25]. We consider models fit using data drawn from the selected, source population P(X.Y, Y, A S = 1) and reason about the properties of those models evaluated on samples drawn from the full, zarger population without selection P(X, Y, A). Graphically, we represent the selection variable S with square node to indicate conditioning on selection in the observed data, where a directed edge in to S indicates dependence the selection mechanism on the originating node (Table 1). This formulation allows for us to anticipate the fairness properties of models under selection based on the structure of the causal graph and the connectivity of the selection node (Supplementary Table B1).
15

---

Algorithm 1: Calculation of the weighted metric difference Ta with oracle access to P(A a V). This procedure computes the difference between the mean of metric function m evaluated subgroup A and the weighted metric Ma computed on the aggregate data. The weighted metric Ma corresponds to evaluation m with respect a distribution where V ~ P(V | A = a) and Y P(Y V). See Section 4.2 and Supplementary Section A.1 for further details
Data:
Dataset D = (X,, Y, A, R.)
Target subgroup a E A
Metric function m y x R R
Selection function get_ v that selects V from {X, Y, R} Function V > [0, IJ, that returns P(A = a V)
Result: The statistic To begin
//Get weights for all N data points for i to N do
0; f get_v((x Y,, R.) wi g(v))
//Extract features for instance //Apply model g to get weight w;
end
//Calculate weighted and subgroup means
Ma f- ΣN ()..().
/ //Weighted mean metric
Mmean f Eis (Y Ei 1(A,=a)
//Mean metric on subgroup Awa
/ //Compute the final statistic
T. Mamean Maa
return To
end
Intuitively, the model learned in the selected population generalizes to the full population if E[Y Z, E Z). for set of predictor variables Z. To reason about this, consider the sufficient condition that S 1 Y Z, which implies that
P(Y Z) P(Y = P(S = = Z,Y)
P(S = Z)
=1
(1)
The implication is that a Bayes-optimat model will generalize under selection if the variables used as predictors d-separate the selection node from Y. For example, in graph where the subgroup covariate shift assumption holds and selection depends on X and A, Bayes-optimal model fit in the selected population using only X generalizes all subgroups in the full population. However, following the results described previously. average performance will not be stable under selection, generally, or across subgroups the selected or full population.
For cases where the graph structure does not imply that model learned in the selected population is subgroup Bayes-optimal in the full target population, reason about whether sufficiency is implied by reasoning about whether each of the ratios in the following identity are equal to 1:
P(Y R, A) P(Y R) P(Y S=LRR)
P(Y S 1, A,
P(S P 1 RAA)
P(S = Y,
(2)
P
P(S Y,R,A)
YIAR
S SLAIR
SLAAYR
In other words, if the model satisfies sufficiency in the selected population (Y 1 A R) and the selection node d-separated from given and (R, Y). then the model satisfies sufficiency full population. This allows to, for example, claim that population Bayes-optimal model satisfies sufficiency in the subgroup covariate shift graph if selection depends only on Y. such that the model is miscalibrated to the same extent for all subgroups. We note that equation (2) permits re-ordering
16

---

Algorithm 2: Calculation of the weighted metric difference To with cross-fitting, This procedure computes the difference between the mean of metric function m evaluated on subgroup A = and the weighted metric M, computed on the aggregate data. The weighted metric Ma corresponds evaluation of with respect distribution where P(V | A a) and Y ~ P(Y V). See Section 4.2 and Supplementary Section A.1 for further details.
Data:
Dataset D (Xx, Y, A,, R
Target subgroup a E A y R R
Metric function
Number of folds K
Function get foldidi that maps data indices [N) to fold indices j {1, . Selection function get_ that selects V from RR
Algorithm Alg for estimating P(A V)
Result: The statistic T
begin
Split data D into K folds using get_folel sot_foldid: (Dj}حー1 / //Train K models for P(A = a V), one for each fold
for j
do
Duain Umpi Dm
//Use K-1 folds for training
9j Aig(Dmin))
//Train model gj
end
//Get held-out predictions for all N data points for i to N do
0s + get_ (X,, Y, R,) j get_ folddi)) ii + (c,)
//Extract features for instance //Get the fold index for instance / //Apply model 9j to get weight w;
end
//Calculate weighted and subgroup means
Ma Ei (Y.RR)
//Weighted metric
ΣN 1(A:=))
M Mmenn
//Mean metric on subgroup A=a
Σe, 1(4=aa)
//Compute the final statistic
T. + Mmcan - Maa
return Ta
end
that allows for reasoning about different set of conditional independencies:
P(Y R, A)
P(Y S = 1, A) * P(S=1
1 * P Y,
A,
(3)
P(Y R) P(Y 1
YLAR
YLA
SLYIR
SLYIRA
For reasoning about separation, we use analogous logic and reason about the terms in the following two identities:
P(R Y,A)
P(R = P(S=1|Y,A)
P
(4)
P(R YY
P(R Y,S=11 1) P(S=1
RLAAY
RLAIY,S=1
SLATY
SLAYY.
and
P(R Y,A)
Y P(S=1
* Y
(5)
P(R Y)
P(R Y,S
1)
RLAAY
SLRJY
SLRRY.A
To summarize the results that follow (Supplementary Table B2), a Bayes-optimal model generalizes under selection if the variables used covariates d-separate the selection node S from Y in the
17

---

causal graph. For example, when selection depends on only X or A, subgroup Bayes-optimal predictors generalize and satisfy sufficiency in the target domain, regardless of other components of the graph. When selection depends only on Y, subgroup Bayes-optimality implies sufficiency without calibration the target domain. When selection depends on Y and either X or A, subgroup Bayes-optimality in the source domain does not imply sufficiency in the target domain. Furthermore, separation in the target domain is implied by Bayes-optimal prediction only in the label shift graph when the selection mechanism does not depend on A, and then only when the model does not depend on A.
A.3 Simulation study
We conduct a simulation study to verify the properties studied in this work. We construct data generating processes satisfying each of the settings studied in this work. The data generating processes are provided in Supplementary Section 1.3.1. For each data generating process, we sample 70,000 independent samples and use 50,000 for training and 20,000 as held-out testing dataset for evaluation.
All model fitting and evaluation procedures are repeated and conducted separately for cases where prediction of Y conducted with (1) X alone, (2) X and an additional categorical covariate indicating subgroup membership A, and (3) a set of models using X alone fit separately for each subgroup. For model fitting, we use the scikit-learn version [60] 1.6.1 implementation of gradient boosting classification trees (specifically, HistGradion diiistrraientBostiiniii enteenstinoigiie. ing.oossincaaassiieeee.. assifier) with stratified five-fold cross- cross-validatin validation,, with a hyperparameter grid over the maximum number of leaf nodes in {10, 25, 50), refitting the model over the training data using the hyperparameter setting with the minimum average log-loss over the held-out cross- validdaiiootion folds. The refit model then used to make predictions on the held-out testing data.
For the experiment involving selection, we modify the complex causal shift graph with three selection mechanisms (see Supplementary Section A.3.1), and for each data generating process, sample 50,000 samples conditioned on S for training and 20,000 samples from the full population (i.e., without selection). We repeat the same training procedure described above.
To get estimates of P(A| V) for use in weighted estimation of model performance, we fit models to predict A from V. When V is X or Y, the fitting procedure is identical to that used for fitting the models for Y, in that conduct cross-validation with gradient boosting trees on the training data and make predictions with the resulting model on the testing dataset. For cases where V = R, we instead conduct nested coss-validation procedure using only the testing data, similar to cross- fitting partition of the testing data, which,
[55]. Here, we use an outer stratified five-fold gross-validatio for each outer fold, conducts an inner stratified five-fold cross-validation procedure that returns model used to make held-out predictions on the corresponding held-out outer fold. Metrics are then computed the full test set.
For evaluation, we compute unweighted and weighted performance estimates for the log-loss, area under the receiver operating characteristic curve (AUC-ROC), sensitivity (recall), specificity, pre- cision, and net benefit [53]. We compute sensitivity, specificity, and precision using a threshold of 0.5. For net benefit, we use the parametrization presented in Pfohl et al. [7], where the preference trade-off is encoded by a choice of threshold. We use a threshold of 0.5 for both the classifier decision threshold and the preference trade-off threshold. To generate confidence intervals, use the percentile bootstrap with 10,000 bootstrap samples of the testing data. For weighted metrics, the un-normalized sample weights are treated as fixed based the result of the procedure described above and sampled alongside the data elements. The resulting samples are then used for weighted computation of metrics.
To generate calibration curves, we guantile-discrttin the range of scores R into ten bins and take the empirical mean of Y for the data in each bin. We compute confidence intervals for each bin separately using the Wilson Score Interval Method [61] with the implementation provided by the Statsmodels package version 0.12.1 [62].
The simulation study was conducted on machines with 32 CPUs and 32 GB of RAM. Computing the bootstrap confidence intervals for each of the settings and metrics was the most significant contributor to the overall run time, taking approximately two hours per setting (i.e., approximately hours for the seven settings considered in the primary analyses).
18

---

A.3.1 Data gemerating processes
Causal-direction data generating processes This description encompasses the covariate shift, outcome shift and complex causal shift settings. We consider X to univariate, Y be binary, and A to be binary, taking on value of 0 or 1. We use binary latent variable U to encode the relationship between X and A. For the covariate shift setting, we set μáo - -2, μn = 0, 7A = 1, set fi -2 A1 0. 7A = 0, 3ag = 0.5, 3g = - 1. and Ca. = 0.1, aes = O. For the complex causal shift setting, the betting are identical to the outcome shift
encom
Buo 3ag = 0.5, and Cas Boj O. For the outcome shift setting, setting.
except that YA 1, which has the effect of introducing a covariate shift. To verify properties in a setting where the subgroup covariates distributions are
separable, further increase the extent of the covariate shift present in the complex causal shift e by setting 141 = 2.
U ~ Bernoulli(0.5)
X = 0 ~ N(Ho 1)
X = 1 N(m, 1)
A| JaU + (1 (A) (Bernoull
Y A Bernoulli (logit" -1 'Baa0 + (aa))
Y A 1 Bernoulli (logit" -1 (Be, x + crai)
We construct three settings with selection bias through augmentation of the complex causal shift setting. We implement three settings, corresponding to cases where the selection mechanism depends on X, Y, or (Y, A), correspond to Sx, Sy, and Sy A- The selected dataset is constructed by filtering the data to cases where S = 1.
4
Sx ~ Bernoulli(
x² +1) 25
Sy ~ Bernouli(00.Y + +..((1 Y))
Sy a A = 0 ~ Bemoulli + +08((1 - V))
Sy A A = 1 ~ Bemoulli + +0.(11 - -Y)
Anticausal data generating processes This description encompeen the label shift, presentation shift, and complex anticausal shift settings. We consider X to be univariate, Y to be binary, and A to be binary, taking value of 0 or 1. For simplicity, having A-dependent effects, rather than using a latent variable U. For the label shift case, we set Ty, 0.5, ny, 0.1, PAAyYo -1, (AAr'i 1, MA,Y -1, MAA,, 1. For the presentation shift case, we set Ty, = 0.5, τy, = 0.5, MA,% = 1, MADy, = 0, JA,Y = -1, MAAY; = 1. The comppiee anticausal shift setting uses the same parameters as the rrsenntttinn shift setting except Tya 0.5, my; = 0.1.
define this data generating process as
A ~ Bermoulli (0.5)
Y ~ Bermoulli (ATr, (ATr, + (1 A)TT.)
X A, ~ N((aY,
A.4 Experiments with the American Community Survey (ACS) Public Use Microdata Sample (
[51]. We use the 'ACSIncome' and 'ACSPublic DibbbioCCC
As described in the main text, we follow Ding et al. [50] to define prediction tasks from ACS PUMS tasks definitions provided by the folktables Python package version 0.0.12 [50]. For all experiments, use the 5- 5-Year horizon California
19

---

'person' files, encompassing census records from 2013-2018. The 'ACSIncome" task definition is to predict whether an individual's income is greater than $50, 000 per year for adults of age 18 years or older with an income of at least $100 that have worked greater than zero hours in the past twelve months. The ACCSuublicCoveraee" task definition to predict whether an individual enrolled in public health insurance plan, restricted to individuals of age 65 years or younger making less than $30, 000 per year. The standard covariates used for the 'ACSIncome' task are age (AGEP), class of worker (COW), educational attainment (SCHL), marital status (MAR), occupation (OCCP), place of birth (POBP), relationship (RELP), usual hours worked per week past 12 month (WKHP), and race (RAC1P). The standard covariates used for the 'ACSSPubicCovvragee task AGEP, SCHL, MAR, sex, disability (DIS), employment status of parents (ESP), citizenship status (CIT). mobility status (MIG), military service (MIL), ancestry (ANC), nativity (NAT), hearing difficulty (DEAR), vision difficulty (DEYE), cognitive difficulty (DREM), income, employment status (ESR), state (ST), gave birth child within the past 12 months (FER), and RACIP.
To define subgroups, we create a custom combined race and ethnicity field that combines the RAICP field with the hispanic origin flag (HISP) and groups rare categories. The combined race/ethnicity field takes on the value "Hispanic" for individuals of Hispanic origin and the value otherwise. We then combine the American Indian and Alaska Native with the Native Hawaiian and Pacific Islander into a group called "Other". For modeling, we remove RACIP from the covariate set, and use the combined race/ethnicity field for subgroup-
ICPP field
prediction.
For modeling and evaluation, we replicate the procedure used in the simulation study, using gradient boosting trees for classification with the same cross-val validation and hyperparameter selection pro- cedure. As standard for the prediction tasks proposed by Ding et al. [50], we do not directly use the person weights provided in the ACS PUMS data, which implies that the derived estimates and models may not be representative of the underlying populations.
As in the case of the simulation study, we conduct these experiments using machines with 32 CPUs and 32 GB of RAM. Computation of the bootstrap confidence intervals was the most significant contributor to the overall run time, taking approximately 14 hours per setting (i.e., approximately 28 hours for the two tasks considered).
20

---

B Supplementary figures and tables
Supplementary Table B1: Conditional independence properties of Bayes-optimal models. / indicates conditions where Bayes-optimal prediction is sufficient condition for the listed criteria. indicates that Bayes-optimal prediction not a sufficient condition for the property. (2) corresponds to the Bayes-optimal predictor that depends on Z.
Sufficiency (Y T. A l (2))
Separation (f 1 A YY)
Setting
Z = X Z = (X, A}
Z = X
Z = (x,A)
Covariate Shift
/
X
x
Outcome Shift
x
X
x
X
Causal Complex Shift
X
X
Label Shift
X
V
Presentation Shift
X
V
X
Anticausal Complex Shift
X
V
X
Supplementary Table B2: Properties of Bayes-optimal models under selection. / indicates cases where Bayes-optimal prediction in the selected population P(- P 1) is sufficient to induce the listed property in the full population P(.). "Other Y 1 A X"" indicate the outcome shift, presentation shift, and complex causal and anticausal shift graphs.
Sufficiency
Subgroup calibration
Separation
Setting
- X Z (X
- Z (X, A]
Z X
Z {X, A}
Covariate shift
X S
X
XX *
+ S
(X, A) + S
Y +
(X,Y) + S
{A, Y} +
(X, Y, A, S
Label shift
SEKKAA
A
AX +
Y S
(X, Y) +
{A,Y) S
[X, Y. +
X
Other Y XAI
X
S
[X, A] S
Y S
(X,Y) +
YA,Y) +
XXXX
[X,Y Y,A) +
21

---

Supplementary Table B3: Implications of findings of controlled evaluations. Each row responds to a different setting of the control variable, covariates, test result, and assumption of Bayes-optimality. Results assume that the weight model P(A V) is correct.
Control Variable
Covariates
Reject(T. 0)
Bayes-optimal
Implications
X
X
Yes
Yes
Subgroup-dependent error struc-
ture conditioned on X is present. Evidence that data is not gener- ated under covariate shift.
Sufficiency violation may be present.
Subgroup performances differ- ences not explained by covariate shift.
Possible to improve performance with subgroup-awan prediction. Improving prediction the basis of X alone is not possible.
X
X
Yes
No
Same the Bayes-optimal case, with the exception that may be possible to improve prediction the basis of X.
X
X
No
Yes
Results consistent with (i.e., can- not reject) the hypothesis of no subgroup-dependent error struc- ture conditioned on X.
Results consistent with (i.e., can- not reject) the hypothesis that data generated under covariate shift.
Results consistent with (i.e., can- not reject) the hypothesis that all performance differences plained by covariate shift.
X
X
No
No
Results consistent with (i.e., can- not reject) the hypothesis of no subgroup-dee deeeeden eтrоr struc- ture conditioned on X.
Results consistent with (i.e., can- reject) the hypothesis that data generated under covariate shift.
Cannot rule out the hypothesis that performance differences ex- plained by systematic underper- formance for some values of X.
Continued next page
22

---

struc- Supplementary Table B3 continued from previous page
Control Var
Covariates
Reject(Tas
0)
Bayes-optimal
Implications
X
(X, A}
Yes
Yes
Subgroup-dependent
ture conditioned on X is present. Evidence that data is gener- ated under covariate shift. Subgroup performances differ- ences not explained by covariate shift.
X
(X
Yes
No
Subgroup-dependent error struc- ture conditioned on X is present.
X
(X
No
Yes
Results consistent with (i.e., can- not reject) the hypothesis of no subgroup-dependent error struc- ture conditioned on X.
Results consistent with (i.e., can- not reject) the hypothesis that data generated under covariate shift.
Results consistent with (i.e. can- not reject) the hypothesis that all performance differences ex- plained by covariate shift.
X
(X, A}
No
No
Results consistent with (i.e., can- not reject) the hypothesis of no subgroup-dependee error struc- ture conditioned on X.
X
Yes
Yes
Subgroup-dependeet error struc- ture conditioned on is present. Evidence that data is not gener- ated under label shift.
Y
X
Yes
No
Same as the Bayes-optimal case.
Continued on next page
23

---

Supplementary Table B3 continued from previous page
Control Var
Covariates
Reject(Tas
0)
Bayes-optimal
Implications
Y
X
No
Yes
Results consistent with (i.e., can- not reject) the hypothesis of no subgroup-dependent error struc- ture conditioned on Y.
Results consistent with (i.e., can- not reject) the hypothesis that data generated under label shift. Results consistent with (i.e., can- not reject) the hypothesis that the separation and equalized odds cri- teria hold.
If label shift holds, properties im- plied by covariate shift violation may hold (i.e. sufficiency viola- tion and informativeness of sub- group membership).
Y
X
No
No
Same as the Bayes-optimal case.
Y
(X,A)
Yes
Yes
Subgroup-dependent error struc- ture conditioned on is present.
Y
(X,A)
Yes
No
Same as the Bayes-optimal case.
Y
(X
No
Yes
Cannot reject hypothesis of no subgroup-dependent error struc- ture conditioned on Y.
Y
(X
No
No
Same as the Baycs-optimal case.
R
X
Yes
Yes
Evidence of sufficiency violation. Subgroup- prediction likely to improve performance. Evidence that data not generated under covariate shift.
R
X
Yes
No
Evidence of sufficiency violation. Subgroup-aware prediction likely to improve performance.
Evidence that data either not generated under covariate shift or that model underperforms for subgroups.
Continued on next page
24

---

Supplementary Table B3 continued from previous page
Control Var
Covariates
Reject(Tas
0)
Bayes-optimal
Implications
R
X
No
Yes
Results consistent with (i.e, can- not reject) the hypothesis that suf- ficiency satisfied.
Results consistent with (i.e., can- reject) the hypothesis that data generated under covariate shift.
R
X
No
No
Results consistent with (i.e., can- not reject) the hypothesis that suf- ficiency satisfied.
R
(X, A)
Yes
Yes
This state should not possible since subgroup Bayes-optimality implies Tę 0 under control for R.
R
(X, A)
Yes
No
Evidence of sufficiency violation. Indicates that the model could be improved for one or more sub- groups.
R
A)
No
Yes
Results consistent with (i.e., can- not reject) the hypothesis that suf- ficiency satisfied.
R
(X,
No
No
Same as the Bayes-optimal case.
Supplementary Table B4: ACS PUMS cohort characteristics. Shown are the number individuals and the prevalence of the label for each race/ethnicity subgroup for the two tasks derived from the ACS PUMS data.
ACSIncome
ACSPublicCoverage
Count
Prevalence
Count
Prevalence
Asian
151,163
0.449
104,642
0.286
Black
40,764
0.327
41,090
0.507
Hispanic
313,007
0.200
317,534
0.393
Multiracial
23.781
0.374
20,734
0.323
Other
8,910
0.304
8,590
0.397
White
412,572
0.504
236,842
0.299
25

---

Lag loss
AUC-ROC
Sensitvitty
Specificity
Net benefit
Covariate Shift
Setting
Outcome Shift
Group
Complex Causal
Separable
Label Shift
Population
Presentation Shift
Anticausal
-0.20
0.25 0
0.50
0.50
0.20
Difference in performance (subgroup-awark Vs. unaware model)
Supplementary Figure B1: Simulation study: the effeet of subgroup-aware prediction on model performance. We report the difference in performance between models that have access to subgroup membership as an additional covariate as compared to those that do not. Plotted are average differences with 95% confidence intervals for each setting and for several performance metrics.
Covariate Stift
Dutcome Shift
Complex Causal
Separable
Label Shift
Presentation Shift Complex Anticaural
1.0
0.5
Fraction positive
Group
0.5
X satat
0.5
0.0
1.0.00
0.5
100.00
1.0000
0.5 1.00.0
1.0
Score
Supplementary Figure B2: Simulation study: calibration curves. Plotted are calibration curves for each subgroup with 95% confidence intervals. The first corresponds to subgroup-agnostic prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
S
+
A)
+
Cov.: X
0.5
Fraction positive
Cov.: XA
Group
Cov.: Xatrat
0.5
1.0
0.5
0.5
Score
Supplementary Figure B3: Simulation study: calibration with selection bias. Plotted are calibration curves for each subgroup with 95% confidence intervals. Models are in the selected population and evaluated in the full population without selection. The first row corresponds to Sabgroup-agnostee prediction, the second row to prediction with an additional covariate, and the third row to stratified prediction by A.
26

---

Weights: None
Weights:
Weights:
Weights:
Covariate Shift
Outcome Shitt
Complex Causal
Cov:
Separable
Label
Presentation Shift
Complex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation on Shift
Complex Anticausal
Covariate Shift
Outcome Shift
Cov straat
Complex Causal
Separable
Label
Presentation on Shift
Complex lex Anticausal
-02
0.0
-.2
0.2 -0.2
0.2 0.0
0.2
Tg: Subgroup log loss weighted estimate
Supplementary Figure B4: Simulation study: controlled evaluation of log loss. Plotted are the statistics To with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first corresponds to Sabbrouu-agnnosse prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
Weights: None
Weights:
Weights:
Weights:
Covariate Shift
Outcome Shift
Complex Causal
Cov:
Separable
Label
Presentation on Shift
Complex lex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation
Comp lex Anticausal
Covariate
Outcome Shift
Cov: X strra
Complex Causal
Separable
Label
Presentation Shift
Complex lex Anticausal
-02
0.0
0.2
-0.2
0.0
0.2
-0.
0.0
0.2
-0.2
0.2
T,: Subgroup AUC-ROC weighttd estimate
Supplementary Figure B5: Simulation study: controlled evaluation AUC-ROC. Plotted are the statistics T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y. or R on the subgroups. The first row corresponds to subgroup nubgroup-agnostic prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
27

---

Weights: None
Weights:
Weights:
Weights: R
Covariate Shift
Outcome
Complex Causal
Cov:
Separable
Presentation Shift
Complex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation Shift
Complex Anticausal
Covariate Shift
Outcome Shift
Cov strat
Complex Causal
Separable
Label Presentation on Shift
Complex lex Anticausal
-0.25
0.00
-0.25
-0.25
0.00
-0.25 0.00
Tg: Subgroup sensitivity weighted estimate
Supplementary Figure B6: Simulation study: controlled evaluation of sensitivity, Plotted are the statistics To with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first corresponds to subbrroup-annosse prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
Weights: None
Weights:
Weights:
Weights:
Covariate Shift Outcome Shift
Complex Causal
Cov:
Separable
Label
Presentation on Shift
Complex lex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation
Comp lex Anticausal
Covariate
Outcome Shift
Cov: X strra
Complex Causal
Separable
Label
Presentation Shift
Complex Anticausal
-0.55 0.00 0.25
-0.25
0.00
0.25
-0.25
0.00
0.25 -0.25 0.00
0.25
Tj: Subgroup specificity weighted estimate
Supplementary Figure B7: Simulation study: controlled evaluation of specificity, Plotted are the statistics T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first corresponds to subgroup nubgroup-agnostic prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
28

---

Weights: None
Weights:
Weights:
Weights:
Covariate Shift
Outcome Shitt
Complex Causal
Con:
Separable
Label
Presentation Shift
Complex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation Shift
Complex Anticausal
Covariate Shift
Outcome Shift
Cov. X strat
Complex Causal
Separable
Label
Presentation on Shift
Complex lex Anticausal
0.25 0.00 0.25
-0.25
0.00
0.25
-0.25
0.00
0.25
-0.25 0.00 0.25
Ta: Subgroup net benefit weighted estimate
Supplementary Figure B8: Simulation study: controlled evaluation of net benefit. Plotted are the statistics Ta with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first corresponds to culbrroupagnnostee prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
Weights: None
Weights:
Weights:
Weights:
Covariate Shift
Outcome Shift
Complex Causal
Cov:
Separable
Label
Presentation on Shift
Complex lex Anticausal
Setting
Covariate Shitt
Group
Outcome Shift
Complex
Cov: XA
Separable
Label Shift
Presentation
Comp lex Anticausal
Covariate
Outcome Shift
Cov: X strrt
Complex Causal
Separable
Label Presentation Shift
Complex Anticausal
-0.25 0.00 0.25
0.00
0.25
-0.25
0.00
0.25
-0.25 0.00
0.25
Tg: Subgroup precision weighted estimate
Supplementary Figure B9: Simulation study: controlled evaluation of precision. Plotted are the statistics To with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first row corresponds to subgroup uubgroup-agnostic prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
29

---

Weights: None
Weights:
Weights:
Weights:
Covariate Shift
Complex Causal
Cov:
Separable
Label Shift
Presentation Shift
Complex Anticausal
Setting
Covariate Shift
Group
Outcome Shift
Complex Causal
Cov: XA
Separable
Label Shift
Presentation on Shift
Complex Anticausal
Covariate Shift
Outcome
Cov: X strat
Complex Causal
Separable
Presentation
Complex Anticausal
-02
0.0
0.2-0.2
0.0 0.2-0.2
0.0
0.2-0.2
0.0
T3: Subgroup cls rate weighttd estimate
Supplementary Figure B10: Simulation study: controlled evaluation of classification rate. Plotted are the statistics T, with 95% confidence intervals, corresponding differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, R on the subgroups. The first row corresponds to subgroup-agnostic prediction. the second row to prediction with A as an additional covariate, and the third row to stratified prediction by A.
30

---

Features:
Features: XA
Features: X strat
Subgroup
Covariste Shift
X
Y
R-
0.60
Subgroup
Outcome Smift
Y
0.7
Subgroup
Complex ex Causal
Y
Group
Control variable
0.55
0.60
0.65
0.55
0.60
0.65
0.55
0.60
0.65
Subgroup
Separable
Y
8-
D.4
0.5
0.6
0.4
0.5
0.6
0.4
0.5
0.6
Subgroup
Label Shift
X
0.2
0.3
0.4
0.2
0,3
0.4
0.2
0.3
0.4
Subgroup
Presentation Shift
X
Y-
0.4
0.6
0.4
0.6
0.4
9.6
Complex Anticausal
Subgroup
X
Y
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Weighted model performance (log los5)
Supplementary Figure B11: Simulation study: weighted estimation of log loss. Plotted are the weighted estimates of performance M, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or for each subgroups. The entry labeled "subgroup" corresponds the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostic prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
31

---

Features:
Features: XA
Features: X_stt strat
Subgroup
Covariate Shift
X
Y
R-
0.625
0.675
0.625
0.650
0.675
0.650
Subgroup
Outcome Shift
Y
Subgroup
Complex Causal
Y
Group
Control variable
0.65
0.70
0.75
0.65
0.70
0.75
0.65
0.70
0.75
Subgroup
Separoble
Y
8-
0.65
0.70
0.75
0.65
0.70
0.75
0.65
0.70
0.75
Subgroup
Label Shift
X
0.90
0.92
0.94
0.90
0.92
0.94
0.90
0.92
0.94
Subgroup
Presentation Shift
X
Y-
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Complex Anticausal
Subgroup
X
Y
0.6
0.8
0.6
0.8
0.6
0.8
Weighted model performance (AUC-ROC)
Supplementary Figure B12: Simulation study: weighted estimation AUC-ROC. Plotted the weighted estimates of performance M, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or R for each subgroups. The entry labeled "subgroup" corresponds the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostie prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
32

---

Features:
Features: XA
Features: X strat
Subgroup
Covariste Shift
X
Y
R-
0.6
Subgroup
Outcome Shift
Y
Subgroup
Complex Causal
Y
Group
Control variable
0.2
0.4
0.6
0.2
0.4
0.2
0.4
0.6
Subgroup
Separabbe
Y-
8
0,00
0.05
0.10
0.00
0.05
0.10
0.00
0.05
0.10
Subgroup oup
Label Shift
X
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Subgroup
Presentation Shift
X
Y
0,7
0.8
0.9
0.7
0.8
0.9
0.7
0.8
0.9
Complex Anticausal
Subgroup
X
Y
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Weighted model performance (sensitivity)
Supplementary Figure B13: Simulation study: weighted estimation of sensitivity. Plotted are the weighted estimates of performance M, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostic prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
33

---

Features:
Features: XA
Features: X strat
Subgroup
Covariste Shift
X
Y
R-
1.0
Subgroup
Outcome Snift
Y
Subgroup
Complex Causal
Y
R-
Group
Control variable
0.7
0.8
0.9
0.7
0.8
0.9
0.7
0.8
0.9
Subgroup
Separable
Y
8-
0.96
0.98
1.00
0.96
0.98
1.00
0.96
0.98
1.00
Subgroup
-+-
Label Shift
X
0.85
0.90
0.95
0.85
0.90
0.95
0.85
0.90
0.95
Subgroup
Presentation Shift
X
Y-
0.25
0.50
0.75
0.25
0.50
0.75
D.25
0.50
0.75
Complex Anticausal
Subgroup
X
Y
0.8
0.9
0.7
0.8
0.9
1.0
0.7
0.8
0.9
1.0
Weighted model performance (specificity)
Supplementary Figure B14: Simulation study: weighted estimation of specificity. Plotted are the weighted estimates of performance M, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostic prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
34

---

Features:
Features: XA
Features: X strat
Subgroup
Covariste Shift
X
Y
+
R-
0,050
0.100
0,050
0.100
Subgroup
Outcome Shift
Y
0.200
0.200
0.400
0.200
0,400
0.200
0,200
0.400
Subgroup
Complex Causal
Y
Group
Control variable
0,100
0.100
0.100
Subgroup
Separabbe
Y
8-
-0.010
0,010
-0.010
0
0.010
-0.010
0
0.010
Subgroup
Label Shift
X
0.100
0.200 0.300
0
0.100
0,200
0,300
0.100
0.200
0.300
Subgroup
Presentation Shift
X-
Y-
0,200
0
0.200
0.200
Complex Anticausal
Subgroup
X
Y
0.100
0.200
0.100 0.200
0.100 0.200
Weighted model performance (net benefit)
Supplementary Figure B15: Simulation study: weighted estimation of net benefit. Plotted the weighted estimates of performance M,, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X, Y, or R for each subgroups. The entry labeled "subgroup" corresponds the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostie prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
35

---

Features:
Features: XA
Features: X_ strat
Subgroup
Covariste Shift
X
Y
-
R1
Subgroup
Outcome Shift
Y
Subgroup
Complex Causal
Y
R-
Group
Control variable
0.6
0.1
0.6
0.4
Subgroup
Separabbe
Y
8-
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Subgroup
Label Shift
X
0.6
0.8
0.6
0.8
0.6
0.8
Subgroup
Presentation Shift
X
Y-
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
Complex Anticausal
Subgroup
X
Y
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
Weighted model performance (precision)
Supplementary Figure B16: Simulation study: weighted estimation of precision. Plotted are the weighted estimates of performance M, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostic prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
36

---

Features:
Features: XA
Features: X strat
Subgroup
Covariste Shift
X
Y
+
R-
Subgroup
Outcome Smift
Y
Subgroup
Complex Causal
Y
R-
Group
Control variable
0.3
0.4
0.5
0.3
0.4
0.5
0.3
0.4
Subgroup
Separabble
Y
8-
0.15
0.20
0.25
0,30
0.15
0.20
0.25
0.30
0.15
0.20
0.25
0.30
Subgroup pup
Label Shift
X
0.2
0.4
0.2
0.4
0.2
0.4
Subgroup
Presentation Shift
X
Y-
0.475
0.500
0.325
0.475
0.500
0.525
0.475
0.500
0.525
Complex Andicausal
Subgroup
X
Y
0.2
0.4
0.4
0'2
0.4
Weighted model performance (cls rate)
Supplementary Figure B17: Simulation study: weighted estimation classification rate. Plotted the weighted estimates of performance M,, with 95% confidence intervals, corresponding weighted estimates of population performance weighted to match the distribution of X, Y, or R for each subgroups. The entry labeled "subgroup" corresponds the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnostie prediction, the second column to prediction with A an additional covariate, and the third column stratified prediction by A.
37

---

Weights: None
Weights:
Weights: Y
Weights: R
Covanitbe Shift
0.6
0.5
0.7
0.5
Shared space performance (log loss)
0.7
0.6
Group
0.8
Population
0.6
0.3
0.2
0.8
Prosent Shift
0.6
1.00
0.75
0.50
0.25
X.strat
X_strat
X_strat
Xtstrat
Covariate set
Supplementary Figure B18: Simulation study: shared space subgroup log loss. Plotted are the average performance with 95% confidence intervals for subgroup performance following weighting to shared space, using the approach of Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
38

---

Weights: None
Weights:
Weights:
Weights: R
0.7
Coveritte Shift
0.6
0.5
0.4
Shared space performance (AUC-ROC)
0.4
0.7
0.6
0.5
Group
0.4
0.8
1
Population
0.6
0.4
0.950
0.925
++
+
t
0.900
Presentition Siift
0.8
0.4
0.8
0.6
0.4
X_strat
X_strat
X_strat
Xtstrat
Covariate set
Supplementary Figure B19: Simulation study: shared space subgroup AUC-ROC. Plotted
the average performance with 95% confidence intervals for subgroup performance following weighting to a shared space, using the approach Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
39

---

Weights: None
Weights: X
Weights: Y
Weights: R
0.6
0.2
0.0
Shared space performance (sensitivity)
0.75
0.50
0.25
Group
0.75
Population
0.50
0.25
Presermalonn Shift
0.8
0.6
1.0
0.5
0.0
X.strat
X_strat
X_strat
Ststrat
Covariate set
Supplementary Figure B20: Simulation study: shared space subgroup sensitivity. Plotted are
average performance with 95% confidence intervals for subgroup performance following weighting to shared space, using the approach of Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
40

---

Weights: None
Weights:
Weights:
Weights: R
Covartate Shift
0.8
0.6
0.75
0.25
Shared space performance (specificity.
1.00
0.75
0.50
Group
1.00
0.75
Population
0.50
0.25
0.9
0.8
0.75
Presentstion
0.50
0.25
1.0
0.5
0.0
X_strat
X_strat
X_strat
Ststrat
Covariate set
Supplementary Figure B21: Simulation study: shared space subgroup specificity. Plotted are average performance with 95% confidence intervals for subgroup performance following weighting to shared space, using the approach of Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
41

---

Weights: None
Weights: X
Weights: Y
Weights: R
0.10
0.00
0.4
0.2
0.0
Shared space performance (net benefit)
-0.2
0.2
0.0
Group
-0.2
0.4
Population
0.0
0.3
0.2
0.1
0.0
0.50
0.25
0.50
0.00
XA
X.strat
XA
X_strat
XA
X_strat
Ststrat
Covariate set
Supplementary Figure B22: Simulation study: shared space subgroup net benefit. Plotted the average performance with 95% confidence intervals for subgroup performance following weighting to a shared space, using the approach Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
42

---

Weights: None
Weights: X
Weights:
Weights: R
0.7
0.6
1.0
Outcome
0.8
0.6
Shared space performance (precision)
0.8
0.6
Group
0.8
Population
0.4
0.8
Presentst-on Shift
0.8
0.4
0.50
0.25
0.00
X_strat
X_strat
X_strat
Ststrat
Covariate set
Supplementary Figure B23: Simulation study: shared space subgroup precision. Plotted are average performance with 95% confidence intervals for subgroup performance following weighting to shared space, using the approach of Cai et al. [16]. Columns correspond to different conditioning variables used to construct the weights and rows correspond data generating processes.
43

---

Weights: None
Weights: X
Weights: Y
Weights: R
Coverate
0.3
0.2
Outcome SMi
0.4
Shared space performance (cls rate)
0.6
0.4
Group
0.8
Population
0.4
0.2
0.4
SHA
0.2
Presentetton Stiee
0.75
0.50
0.25
X
XA
X_strat
XA
X_strat
XA
X_strat
XA
Xtstrat
Covariate set
Supplementary Figure B24: Simulation study: shared space subgroup classification rate. Plotted are average performance with 95% confidence intervals for subgroup performance following weighting to shared space, using the approach of Cai et al. [16]. Columns correspond to different conditioning variables used construct the weights and rows correspond data generating processes.
Log loss
AUC-ROC
Sensitivity
Specificity
Net benefit
Group
ACSincome
Asian
Task
Black
Hispanic
Multiracial
Other
ACSPublicCoverage
White
0.01 -0.05
-0.03
Difference in performance (Subgroup-aware vs. unaware model)
the effect of subgroup-aware prediction model performance. We report the difference in performance between models that have access to subgroup membership additional covariate compared those that do not. Plotted are average differences with 95% confidence intervals for each setting and for several performance metrics.
Supplementary Figure B25: ACS PUMS:
44

---

ACSincome
ACSPublicCoverag
1.0
0.5
Fraction positive
Group
Asian
Black
0.5
XA
Hispanic
Multiracial
1.0
1903 X 1r0
0.5
Score
Supplementary Figure B26: ACS PUMS: calibration curves. Plotted are calibration curves for each subgroup with 95% confidence intervals. The first row corresponds to subgroup-agnostic prediction, the second row to prediction with as an additional covariate, and the third row to stratified prediction by A.
Weights: None
Weights: X
Weights: Y
Weights: R
ACSincame
Cov: X
ACSPubliCoverrae
Group
Task
ACSincome
Asian
Cov. XA
Black
Hispanic
Multiracial
ACSPubliCoverage
Other
ACSincome
Cov: X. Stsstra
White
AcseublicCoverage
0.0
0.1
0.0
0.1
0.0
0.1
0.0
0.1
T3 Subgroup loss weighted estimate
Supplementary Figure B27: ACS PUMS: controlled evaluation of log loss. Plotted are the statisties T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y. or on the subgroups. The first row corresponds to subgroup-agnostic prediction, the second row to prediction with A as an additional covariate, and the third row to stratified prediction by A.
45

---

Weights: None
Weights:
Weights: Y
Weights:
ACSincome
Cov.:
ACSPublicoveergee
Group
Task
ACSincome
Cov:: XA
Asian
Black
Hispanic
Multiracial
ACSPublcCoveraag
Other
ACSIncome
Cov.: X strat
While
ACSPublicoverrae
0.05 0.00 0.05 -0.05 0.00 0.05 -0.05
0.05 -0.05 0.00 0.05
Ta: Subgroup AUC-ROC- weighted estimate
Supplementary Figure B28: ACS PUMS:
controlled evaluation of AUC-ROC. Plotted are the statistics Ta with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first row corresponds to subgroup-agnostic prediction, the second row prediction with an additional covariate, and the third row stratified prediction by A.
Weights:
Weights:
Weights: Y
Weights:
ACSincome
Cov.:
ACSPublicCoverage
Group
Task
ACSIncome
Cov. XA
Asian
Black
Hispanic
Multiracial
ACSPubicCoverage
Other
Cov.: X strat
White
ACSIncome
ACSPulliCoverage
0.25
0.00
0.25
0.00
0.25
0.00
0.25
0.00
T3 Subgroup sensitivity weighted estimate
Supplementary Figure B29: ACS PUMS: controlled evaluation of sensitivity. Plotted the statistics T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y. or R on the subgroups. The first row corresponds to subgroup aggooscc-agnostic prediction, the second row prediction with A an additional covariate, and the third row stratified prediction by A.
46

---

Weights: None
Weights:
Weights: Y
Weights:
ACSincome
Cov.:
ACSPublicoveernge
Group
Task
ACSincome
Cov:: XA
Asian
Black
Hispanic
Multiracial
ACSPublcCooeraag
Other
ACSIncome
Cov.: X strat
While
ACSPublcCoverrae
0.1
0.0
-0.1 0.0
0.0
-0.1
0.0
Tg: Subgroup specificity - weighted estimate
Supplementary Figure B30: ACS PUMS: controlled evaluation of specificity, Plotted are the statistics Ta with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first row corresponds to subgroup-age aggiosicc prediction, the second row prediction with an additional covariate, and the third row stratified prediction by A.
Weights: None
Weights:
Weights: Y
Weights:
ACSincome
Cov.:
ACSPublicCvverage
Group
Task
ACSIncome
Cov. XA
Asian
Black
Hispanic
Multiracial
ACSPubicCoverage
Other
Cov.: X strat
White
ACSIncome
ACSPubliCoverrae
0.1 0.0
0.1 0.0 0.1
0.1
0.0 0.1
0.0
0.1
Tg: Subgroup net benefit weighted estimate
Supplementary Figure B31: ACS PUMS: controlled evaluation of net benefit. Plotted the statistics T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y. or R on the subgroups. The first row corresponds to subgroup aggooscc-agnostic prediction, the second row prediction with A an additional covariate, and the third row stratified prediction by A.
47

---

Weights:
Weights:
Weights:
Weights:
ACSincome
Cov.:
ACSPublicoverrgee
Group
Task
ACSincome
Cov:: XA
Asian
Black
Hispanic
Multiracial
ACSPublcCooeraaee
Other
ACSIncome
Cov.: X strat
While
ACSPublicoverrae
0.1 0.0 0.1
0.1 0.1
-0.1
0.1 -0.1
Ta: Subgroup precision weighted estimate
Supplementary Figure B32: ACS PUMS: controlled evaluation of precision. Plotted are the statistics Ta with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y, or R on the subgroups. The first row corresponds to subgroup-age aagnosic prediction, the second row prediction with an additional covariate, and the third row stratified prediction by A.
Weights:
Weights:
Weights:
Weights:
ACSincome
Cov.:
ACSPublicCvverage
Group
Task
ACSIncome
Cov. XA
Asian
Black
Hispanic
Multiracial
ACSPubicCoverage
Other
Cov.: X strat
White
ACSIncome
ACSPulliCoverage
0.0
0.2
0.0
0.2
0.0
0.0
Tj Subgroup cls rate weighted estimate
Supplementary Figure B33: ACS PUMS: controlled evaluation of classification rate. Plotted the statistics T, with 95% confidence intervals, corresponding to differences between the unweighted disaggregated performance with the population performance weighted to match the distribution of X, Y. or R on the subgroups. The first row corresponds to subgroup- -agnostic prediction, the second to row prediction with A as an additional covariate, and the third to stratified prediction by A.
48

---

Features:
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
8-
Hispanic
Control variable
Multiracial
0.30
0.35
0.40
0.45
0.30
0.35
0.40
0.45
0.30
0.35
0.40
0.45
Other
White
Subgroup
ACSPublirCoeerge
0.45
0.50
0.55
0.60
0.45
0.50
0.55
0.60
0.45
0.50
0.55
0.60
Weighted model performance (Log loss)
Supplementary Figure B34: ACS PUMS: weighted estimation of log loss. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X, Y, ft for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresponds subgroup-agnosti prediction, the second column to prediction with A as additional covariate, and the third column to stratified prediction by A.
49

---

Features: X
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
R-
Hispanic
Control variable
Multiracial
0.850 0.875 0.900
0,850 0.875 0,900
0.850 0.875 0.900
Other
White
Subgroup
ACSPublirCovergge
0.75
0.80
0.75
0.80
0.75
0.80
Weighted model performance (AUC-ROC)
Supplementary Figure B35: ACS PUMS: weighted estimation of AUC-ROC. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X. Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresoonds to subgroup-agnoste prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
50

---

Features: X
Features:
Features: X strat
Subgroup
X
AcSincome
Group
Asian
Black
Hispanic
Control variable
Multiracial
0.6
0.7
0.8
0.6
0.7
0.8
0.6
0.7
0.8
Other
White
Subgroup
ACSPublicCovergg
0.4
0.6
0.6 0.4 0.6
Weighted model performance (Sensitivity)
Supplementary Figure B36: ACS PUMS: weighted estimation of sensitivity. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X. Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresoonds to subgroup-agnosti prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
51

---

Features: X
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
R-
Hispanic
Control variable
Multiracial
0.80
0.85
0.90
0.95
0.80
0.85
0.90
0.95
0,50
0.85
0.90
0.95
Other
White
Subgroup
ACSPublicCoerrag
0.80
0.85
0.90
0.95
0.80
0.85
0.90
0.95
0.80
0.85
0.90
0.95
Weighted model performance (Specificity)
Supplementary Figure B37: ACS PUMS: weighted estimation of specificity. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X. Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresoonds to subgroup-agnosti prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
52

---

Features: X
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
Hispanic
Control variable
Multiracial
0,100
0.200
0.300
0.100
0,200
0.300
0.100
0,200
0.300
Other
White
Subgroup
ACSPublicCoverage
0.100
0.200
0.100
0.200
0.100
0.200
Weighted model performance (Net benefit)
Supplementary Figure B38: ACS PUMS: weighted estimation of net benefit. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X. Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresoonds to subgroup-agnosti prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
53

---

Features: X
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
8-
Hispanic
Control variable
Multiracial
0.6
0.7
0.8
0.7
0.8
0.6
0,7
0.8
Other
White
Subgroup
ACSPublicCoverag
0.7
0.8 0.7 0.8
0.8
Weighted model performance (Precision)
Supplementary Figure B39: ACS PUMS: weighted estimation of precision. Plotted are the weighted estimates performance M, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X. Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresoonds to subgroup-agnoste prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
54

---

Features: X
Features:
Features: X strat
Subgroup
X
ACSincome
Group
Asian
Black
Hispanic
Control variable
Multiracial
0.2
0.3
0.4
0.5
0.2
0.3
0.4
0.5
0.2
0.3
0.4
0.5
Other
White
Subgroup
ACSPublicCoverag
0.3
0.4
0.5
0.4 0.5
0.3
0.4
0.5
Weighted model performance (Cls rate)
Supplementary Figure B40: ACS PUMS: weighted estimation of classification rate. Plotted are weighted estimates of performance M,, with 95% confidence intervals, corresponding to weighted estimates of population performance weighted to match the distribution of X, Y. or R for each subgroups. The entry labeled "subgroup" corresponds to the unweighted estimate of subgroup performance. The first column corresponds to subgroup-agnosse prediction, the second column to prediction with A as an additional covariate, and the third column to stratified prediction by A.
55

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The claims in the abstract and introduction are specific and are an accurate reflection of the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
56

---

Justification: Yes. The discussion section discusses limitations. In particular, we discuss how the generalizability of our findings may be limited due to the restricted scope of our work (e.g., primarily for binary outcomes and the restriction to Bayes-optimal predictors for some findings).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [No]
We present theoretical results in the main text and supplementary material without structuring them formally as theorems and lemmas. However, we do assert that our presentation is rigorous and appropriate given the results presented.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
57

---

Answer: [Yes]
Justification: The experiments are conducted with simulated data and open, public datasets. The data generating processes for the simulated data are described in full in the paper. The experimental procedures are described with sufficient detail for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code is released and available at a public repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
lublic/guides wotps///Soouuuiiiiiioooiio
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: ott.de/ee/oodeaeebiiiiiiiioioooo
//nips.coo. wwwpc///ppppppeeeeeeeeeeeeeeeeee http://ggiie/eeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
58

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes)
Justification: The training and evaluation details are described in sufficient detail in the detailed methods section provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report confidence intervals for all statistics. As described the paper, these typically bootstrap confidence intervals computed with the percentile bootstrap.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
59

---

Justification:
Yes, the relevant details are provided in the detailed methods section provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: / htt///euriisccccec https///ccppppcc/.cccccceeeeeeee
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics and verified that this true.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impact is integral to the topic of the work. Potential negative impacts due to misinterpretation of the procedures propose diseussed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
60

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No high risk data or models have been released part of this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No existing assets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcom thoode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Released code is appropriately licensed.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
61

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicabll, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No research with human subjects conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No research involving crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs not used.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.LM policy (https: https::///euripss /////nurrs.../// cwt//o/f///ennnccccc /ctp:////Ceeeeecce///2////11/11 for what should or should not be described.
62


Reviews:
[{"review_id": "3mDYDMC0R2", "reviewer": "Reviewer_oWko", "summary": "This paper presents a theoretical framework for evaluating the limitations of disaggregated performance metrics in fairness analysis. Using causal graphical models, particularly DAGs, the authors characterize when performance disparities across subgroups arise from structural data differences versus from model bias. They formalize a taxonomy of distribution shift types (e.g., covariate, outcome, label, presentation), derive conditional independence properties under each, and propose a controlled evaluation procedure using importance weighting to isolate confounding.\nThe theoretical contributions are clear: even Bayes-optimal models can yield group-wise disparities under plausible data-generating assumptions, and common fairness metrics may be misleading without consideration of underlying causal structure. Synthetic and real-world (ACS PUMS) experiments empirically support these findings.", "questions": "- Given that real-world practitioners typically cannot observe or validate the causal structure, what practical guidance can you offer for choosing among the causal assumptions in your taxonomy? Are there diagnostics or heuristics to help evaluate which DAG is plausible?\n- Is there a path forward for extending this framework to multi-class or continuous outcomes? If so, what new assumptions would be required, and what challenges arise?\n- How sensitive is the framework (particularly the weighting-based controlled evaluation) to models that are miscalibrated or underfit? Can misleading conclusions arise in such settings?\n- What are the empirical characteristics of the importance weighting method under data sparsity or high dimensionality? How does the variance of the estimator behave for small subgroups?\n-  For teams working with real-world data, what minimal conditions must be met (in terms of data quality, model fidelity, or covariate richness) to apply this framework responsibly?", "limitations": "The paper touches on some of these issues but could do more to caution against misapplication. Controlled eval procedures can create false confidence if the model is poorly fit or the causal assumptions are incorrect. Practitioners without causal expertise could be misled by surface-level stability.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "First, let's start with the strengths:\nThe causal analysis is rigorous, and the paper provides a clear set of sufficient conditions under which group-level disparities do or do not reflect bias. I'm also a fan of the use of DAGs to formalize distribution shift types (e.g., covariate vs. outcome shift) to add clarity to fairness evaluation debates and help unify previously fragmented observations in the literature. The visualization is quite strong. The tables summarizing which fairness criteria are expected to hold under different assumptions offer a usable diagnostic framework for readers with sufficient causal knowledge. Lastly, the focus on Bayes-optimality enables tractable and interpretable theoretical results.\n\nNow on to the weaknesses in no particular order:\n1. The framework presumes knowledge of the data-generating process or, at a minimum, a defensible causal DAG. In most real-world settings, this is a strong and often unjustifiable assumption. The practical relevance of the framework is directly tied to the feasibility of accurate causal discovery.\n\n2. All theoretical and empirical analysis is restricted to binary classification. Extending to multi-class or continuous outcomes would likely require fundamentally different assumptions and analytical tools- not just technical extensions.\n\n3. While Bayes-optimality enables clean theory, it is an unrealistic assumption in most deployment contexts. In its current form, the framework does not account for model capacity limitations, data scarcity, or learning instability.\n\n4. The empirical evaluation is narrow with only two tabular prediction tasks and does not explore how the framework performs when causal assumptions are violated. Even within those tasks, the framework is not fully stress-tested under common real-world uncertainties.", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oWko for their engaged and enthusiastic review. In light of their comments and questions, we would like to emphasize the following.\n\n**Regarding the need for causal knowledge of the data-generating-process**: \n\nWe clarify that the use of our framework does not necessarily require an analyst to have full knowledge of the data generating process. \n\nIn general our core results are of the form, “if a given causal structure and set of conditions hold, then the property of interest (typically conditional independence) holds”. This type of result immediately implies a diagnostic tool that serves as a form of causal discovery: if we observe evidence against the property of interest then we have evidence towards falsification of the causal structure and set of conditions that imply the property of interest. This follows from the equivalence of a logical statement “If P, then Q” with the contrapositive “If not Q, then not P”. Note that this does not provide the means to confirm any particular hypothesis regarding the underlying causal structure, but it does allow for rejection of a causal graph when we have evidence of violation of a conditional independence property that the causal graph in question satisfies.\n\nA key example of this in our work is reasoning about evidence against the hypothesis that the data were generated under the covariate shift in a setting without selection (which encodes the assumption $Y \\perp A \\mid X$). We show that both (1) evaluation of a subgroup-agnostic predictor with control for the covariates $X$ and (2) an analysis of whether subgroup-aware prediction improves over subgroup-agnostic prediction can be used as evidence to falsify the assumption that the data were generated under the covariate shift graph. In either case, we have evidence that the covariate shift assumption is violated but not confirmation that the data were generated from any of the other graphs considered. Fortunately, the other graphs that violate the covariate shift assumption (label shift, outcome shift, presentation shift, complex shift) behave similarly with respect to the properties that we discuss related to disaggregated evaluation and the informativeness of subgroup membership. The label shift graph is an exception, but we are again fortunate to have the ability to falsify the assumptions of the label shift setting with observations on the basis of evidence against the conditional independence $X \\perp A \\mid Y$ (which corresponds to a controlled evaluation of a subgroup-agnostic model that controls for $Y$).\n\nWe also would like to emphasize that one rhetorical purpose of describing the properties of models fit to data faithful to this set of specific causal graphs is to serve as a reasonably comprehensive set of vignettes demonstrating how subgroup performance instability arises under common causal assumptions regarding heterogeneity across subgroups. The experiments with simulated data serve to verify that the properties of interest hold as expected. The experiments with real-world data serve to demonstrate how the framework can enhance fairness analyses in practical settings where the causal graph is not necessarily known and we cannot assume the model is Bayes-optimal.\n\nA major caveat to the above arguments is that in the case that selection is present, it is necessary to at least have some knowledge of the data generating process for valid inference. This is not a limitation unique to this study, but rather a fundamental inability to perform inference in the absence of assumptions regarding the relationship between the data and the target population or parameter or interest. \n\n**Regarding multi-class or continuous outcomes**:\n\nWe agree that extending the approach to handle multi-class or continuous outcomes may require fundamentally different assumptions and analytical tools. We will update the discussion section to include this as an important area of future research.\n\nTo briefly speculate, we expect that the multi-class classification context is a more straightforward translation from binary classification than the continuous outcome case is. For multi-class classification, the Bayes-optimal multi-class probabilistic predictor fully characterizes the categorical distribution $P(Y | X)$ in the same way that the Bayes-optimal binary probabilistic classifier fully characterizes Bernoulli $P(Y | X)$. As a result, we suspect many of the core definitions and findings to extend to the multi-class setting. \n\nIn the continuous case, the conditional mean estimator $\\mathbb{E}[Y \\mid X]$ is not sufficient to describe arbitrary $P(Y \\mid X)$. As a result, the Bayes-optimal conditional mean regressor does not immediately satisfy the same set of conditional independences satisfied in classification contexts (e.g., sufficiency). However, if we make  assumptions on the data generating process (e.g., if $Y$ is conditionally Gaussian or Poisson), the parameters fully characterizing those distributions can be estimated via maximum likelihood. The implications for disaggregated evaluations may also differ under different noise structures. For example, if the true noise model is homoskedastic (across levels of $X$), the Bayes-optimal conditional mean regressor attains equal subgroup mean squared error under the covariate shift graph, even if this is not true for binary classification or under heteroskedasticity. \n\n**Regarding departures from Bayes-optimality**:\n\nWe would like to clarify that while some of the properties that we describe are specific to the Bayes-optimal case, our framework can still be informative under departures from Bayes-optimality. Please refer to the response to reviewer c4Qe for more details. Please also refer to the response to reviewer Lqbf for further discussion of potential directions for further theoretical and empirical study of the role of model misspecification and estimation error.\n\nRegarding weighting with data sparsity and high dimensionality:\nWe concur that further work would be required to comprehensively extend our evaluation framework to high dimensional and sparse datasets. Both high-dimensionality and sparsity affect our approach by plausibly violating overlap (see e.g., D’Amour 2021), such that one or more regions of the covariate space are uniquely associated with exactly one subgroup. In the present version of the work, we do conduct a brief investigation of the issues that arise in the setting in section 3.5 titled “Subgroup separability” and the corresponding set of simulation experiments for the setting titled “Separable” (e.g., in Figure 1). The primary implication for our approach is the inability to detect covariate shift violation in the separable setting because it is not possible to compare $Y \\mid X$ across subgroups for levels of $X$ that are only observed for a single subgroup. This is a fundamental issue and not a unique limitation of our approach.\n\nIn the causal inference literature, overlap violations often yield weighting estimators with high variance, typically due to the presence of weights with small denominators. Intuitively, this occurs when a weighting estimator projects a distribution onto a region of low probability. We mitigate this issue by projecting the aggregate population distribution onto the distribution of each subgroup. Common support is guaranteed in this context because a subgroup is a subset of the aggregate population.\n\nOur approach yields weights proportional to $P(A \\mid V)$, which is bounded between 0 and 1, and does not induce the same sort of high variance estimates. The limitation of this approach, however, is that under full separability (i.e., $P(A \\mid V)$ = 0 or 1 for all $V$), the weighted population estimate is identical to the unweighted subgroup estimate regardless of the causal structure, implying no ability to compare the subgroups.\n\nIn the present paper, we discuss this in supplementary methods section A.1 and do observe some examples of high variance estimation when applying the alternative weighting approach of Cai et al (Supplementary Figures B18-B24).\n\nD’Amour, Alexander, et al. \"Overlap in observational studies with high-dimensional covariates.\" Journal of Econometrics 221.2 (2021): 644-654.\n\n**Regarding minimal conditions for use with real-world data**:\n\nFor the setting that the majority of this work is conducted, it is necessary to assume that the data used is high-quality and representative of the relevant target population and that the prediction task is well-formulated, such that increases in predictive performance are consistent with increased utility of the model. Arguably, these assumptions underpin most applied uses of machine learning, regardless of whether they are made explicit or are appropriate. One strength of our approach is that we provide a structured way to reason about deviations from this ideal through explicit specification of the causal structure of selection mechanisms. \n* Regarding data quality and richness, similar to our approach taken to reasoning about selection, it is plausible that the causal graphs we present could be augmented to explicitly represent assumptions regarding measurement mechanisms that consider the observed data to be noisy or incomplete views of unobserved constructs of interest. This would allow characterization of model performance and fairness properties under various assumptions regarding those mechanisms and may suggest approaches to testing. We can update the discussion section to mention this as future directions.\n* Regarding model fidelity, please refer to the earlier points about deviations from Bayes-optimality.\n* We intend to update the discussion section to emphasize key practical takeaways, including recommendations for the interpretation of standard disaggregated evaluation, implications of the theoretical analysis and empirical findings, as well as guidance for appropriate use and interpretation of the methodology presented here."}, {"review_id": "u8ZHZRo02t", "reviewer": "Reviewer_c4Qe", "summary": "This paper explores theoretical and empirical properties of the disaggregated performance of Bayes-optimal models under different causal model assumptions. The paper lays out a framework for understanding the different types of causal relationships between labels, covariates, and group membership, and elucidates some theoretical properties of models under these different conditions. The overall argument of the work is that disaggregated metrics in a vacuum are not enough to fully understand whether or not a model is “fair” because some group-wise differences might be expected depending on the causal structure of the data generating process. For experiments, the paper shows a simulation study and measures the difference between subgroup-aware and subgroup-agnostic models on two Census Bureau ACS tasks.", "questions": "- The experiments show that subgroup-aware models have improved predictive performance over models that do not include subgroup membership information. What implications does this have for disaggregated evaluations? I personally had a hard time following the narrative of what we learn about disaggregated metrics from this difference in the models.\n\n- How much does the Bayes optimality assumption affect the generalizability of the results shown here in practice? \n\n- Are there novel metrics or techniques that the authors would propose to alleviate some of the issues that they find in the work?", "limitations": "I have discussed some limitations in the weaknesses above.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The primary strength of this paper is in its motivation and problem framing. I agree with the authors’ main conceit that disaggregated evaluations, on their own, do not necessarily tell the whole story of model performance. The authors have introduced an interesting and rigorous framework for understanding some of the pitfalls that can come from over-reliance on subgroup disaggregation. \n\nThe main weakness of this paper, in my opinion, is the clarity of the presentation and the connection between the experimental results and the overall argument that the paper is making. I see from both the empirical and theoretical results that subgroup-aware models tend to perform better than their agnostic counterparts under a variety of metrics in every causal setting except for covariate shift. I personally had a hard time connecting that empirical point back to the introduction and story of the paper regarding disaggregated evaluations. I have some more specific questions below that I hope will illustrate the confusion further. \n\nAnother weakness of the work is that I find it hard to take concrete recommendations from the work. I do think some useful mathematical properties are illustrated, but I do not see much discussion in the work of suggestions for future evaluators. I think this could be expanded on for the camera-ready version.\n\nOverall this paper seems like sound work, but could benefit from some improvements in clarity to hammer its points home.", "was_revised": true, "final_justification": "After considering the author responses and other reviews, I raised my score to a 5. I think the concerns about clarity that I raised will be sufficiently addressed in the final revision.", "rebuttal": "We thank reviewer c4Qe for their support of the work and their constructive comments. Below, we respond to each of the questions raised:\n\n**Regarding the relationship between disaggregated evaluation and analysis of the informativeness of subgroup membership**:\n\nThank you for this feedback, as we agree that the current presentation is not entirely clear on this point. To clarify, despite the apparent disconnect between these two threads of inquiry (disaggregated evaluation vs. capacity of subgroup-aware modeling to improve prediction performance), there is a deep connection between them through the lens of conditional independence testing. This allows us to view both types of analysis as providing distinct views on the same underlying phenomena. \n\nFor example, in the Bayes-optimal case both of the following findings are evidence against the same conditional independence $Y \\perp A \\mid X$ (i.e., the covariate shift assumption): (1) subgroup-aware modeling improves predictive performance over subgroup-agnostic modeling and (2) control for $X$ in disaggregated evaluation of a subgroup-agnostic model is insufficient to explain subgroup performance differences. Similarly, when we are in a setting where that covariate shift assumption is violated, we expect subgroup-agnostic prediction to induce sufficiency violation and miscalibration for subgroups, and this can be directly assessed with controlled evaluation on the basis of $R$. Notably, such findings with regards to calibration and sufficiency are also a signal that provides signal post-hoc that the model could be improved for subgroups for which it is miscalibrated.\n\nWe intend to make a number of minor changes to the manuscript to aid in clarity. Please see the response to reviewer Lqbf for a detailed point-by-point description of those changes. Relevant to the points you raise, we intend to add additional conceptual scaffolding to the header of section 3 and to the individual subsections in order to better clarify the rhetorical purpose of each section with respect to the aims of the paper as a whole. In the experiments section, we also intend to make minor textual changes to better emphasize the relevance of the subgroup-aware modeling results in relation to the disaggregated evaluation results.\n\n**Regarding the role of Bayes-optimality**:\n\nWe would like to clarify that the use of controlled evaluation procedures does not require Bayes-optimality, but interpretation can differ depending on whether the model is Bayes-optimal. In the original manuscript, this is conveyed by the rows of Table 3 that refer to an arbitrary model $f$ that is not Bayes-optimal. We acknowledge that this could be made clearer in the manuscript and we intend to do so through textual revision to sections 3.3 and 3.4. We have also drafted a new table (to be included in the appendix) that details the set of implications and valid conclusions that can be drawn from controlled evaluations based on the configuration of the weights, the covariate set, the observed statistic $T_a$, and whether the model is Bayes-optimal. Comparison of the set of valid conclusions with and without Bayes-optimality clarifies when and how Bayes-optimality is relevant. At the end of this response, we outline the content of that table.\n\n**Novel metrics and techniques**:\n\nIn future work, we hope to advance the theoretical understanding of the estimators and estimation procedures we propose and systematically evaluate the role of misspecification and estimation error. For this purpose, we expect that techniques from the semiparametric estimation literature that treats the optimal predictive model as a nuisance function will provide insight. Epistemic uncertainty quantification (with e.g., ensembles or Bayesian models) may also prove useful as a practical means of assessing variance in the estimator, as a proxy for closeness to Bayes-optimality. We will update the discussion section to refer to these as promising future directions.\n\n**On practical guidance for practitioners**:\n\nWe intend to make textual revision to sections 3.4 and to the discussion to provide further recommendations for practitioners. The detailed table of guidance regarding interpretation of controlled evaluation provided at the end of this response also serves this purpose. The revisions to the discussion will include both general guidance regarding interpretation of standard uncontrolled disaggregated evaluations as well as the controlled evaluation procedures we propose.\n\n\n**Content for a new table (to be added to the appendix) containing guidance regarding interpretation of controlled evaluations.** Each header corresponds to a setting of control variable, choice of covariates, controlled evaluation finding, and assumption regarding Bayes-optimality. Bulleted content under the headers are the set of implications and conclusions implied by the setting.\n\nControl: $X$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: Yes\n* Subgroup-dependent error structure conditioned on $X$ is present\n* Evidence that data not generated under covariate shift\n* Sufficiency violation likely to be present\n* Subgroup performances differences not explained by covariate shift\n* Possible to improve performance with subgroup-aware prediction\n* Improved subgroup-agnostic prediction is not possible\n\nControl: $X$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: No\n* Same as the Bayes-optimal case, with the exception that some performance differences may be mitigated through improved subgroup-agnostic modeling or data collection\n\nControl: $X$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: Yes\n* Results consistent with (i.e., cannot reject) the hypothesis of no subgroup-dependent error structure conditioned on $X$\n* Results consistent with (i.e., cannot reject) the hypothesis that data generated under covariate shift\n* Results consistent with (i.e., cannot reject) the hypothesis that all performance differences explained by covariate shift\n\nControl: $X$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: No\n* Results consistent with (i.e., cannot reject) the hypothesis of no subgroup-dependent error structure conditioned on $X$\n* Results consistent with (i.e., cannot reject) the hypothesis that data generated under covariate shift\n* Cannot rule out hypothesis that performance differences explained by systematic underperformance across subgroups\n\nControl: $Y$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: Yes\n* Subgroup-dependent error structure conditioned on $Y$ is present\n* Evidence that data not generated under label shift\n\nControl: $Y$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: No\n* Same as the Bayes-optimal case\n\nControl: $Y$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: Yes\n* Results consistent with (i.e., cannot reject) the hypothesis of no subgroup-dependent error structure conditioned on $Y$\n* Results consistent with (i.e., cannot reject) the hypothesis that data generated under label shift\n* Results consistent with (i.e., cannot reject) the hypothesis that the separation and equalized odds criteria hold\n* If label shift holds, properties implied by covariate shift violation may hold (i.e., sufficiency violation and informativeness of subgroup membership)\n\nControl: $Y$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: No\n* Same as the Bayes-optimal case\n\nControl: $Y$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): Yes, Bayes-optimal: Yes\n* Subgroup-dependent error structure conditioned on $Y$ is present\n\nControl: $Y$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): Yes, Bayes-optimal: No\n* Same as the Bayes-optimal case\n\nControl: $Y$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): No, Bayes-optimal: Yes\n* Cannot reject hypothesis of no subgroup-dependent error structure conditioned on $Y$\n\nControl: $Y$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): No, Bayes-optimal: No\n* Same as the Bayes-optimal case\n\nControl: $R$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: Yes\n* Evidence of sufficiency violation\n* Evidence that data not generated under covariate shift\n* Possible to improve performance with subgroup-aware prediction\n\nControl: $R$, Covariates: $X$, Reject($T_a=0$): Yes, Bayes-optimal: No\n* Evidence of sufficiency violation\n* Subgroup-aware prediction likely to improve performance\n* Evidence that data either not generated under covariate shift or model underperforms\n\nControl: $R$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: Yes\n* Results consistent with (i.e., cannot reject) the hypothesis that sufficiency is satisfied\n* Results consistent with (i.e., cannot reject) the hypothesis that data generated under covariate shift\n\nControl: $R$, Covariates: $X$, Reject($T_a=0$): No, Bayes-optimal: No\n* Results consistent with (i.e., cannot reject) the hypothesis that sufficiency is satisfied\n\nControl: $R$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): Yes, Bayes-optimal: Yes\n* This outcome should not be possible if the weights are accurately estimated\n\nControl: $R$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): Yes, Bayes-optimal: No\n* Evidence of sufficiency violation\n* Indicates that the model could be improved for one or more subgroups\n\nControl: $R$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): No, Bayes-optimal: Yes\n* Results consistent with (i.e., cannot reject) the hypothesis that sufficiency is satisfied\n\nControl: $R$, Covariates: $\\{X, A\\}$, Reject($T_a=0$): No, Bayes-optimal: No\n* Same the Bayes-optimal case"}, {"review_id": "1gjgDptegi", "reviewer": "Reviewer_Lqbf", "summary": "The paper investigates why evaluating fairness by comparing model performance across disaggregated groups (e.g., per-race or per-gender accuracy) can be misleading. The authors:\n\n- Analyze multiple causal data-generation processes (e.g., covariate shift) using DAGs to illustrate when disparities reflect underlying population inequities versus algorithmic unfairness.\n- Derive conditions under which Bayes-optimal models satisfy fairness notions such as separation or sufficiency.\n- Propose a controlled-evaluation method using a control variable $V$, combined with a statistic $T_a$ to test whether observed subgroup disparities are confounded.\n- Validate their approach with simulations and experiments on public census data (ACS PUMS), demonstrating that causal structure—rather than raw accuracy differences—often drives conclusions about fairness or unfairness.", "questions": "- **Readability and narrative structure**: Could you consider improving the exposition by:  \n  – Breaking down complex sentences and clarifying transitions?  \n  – Introducing motivating examples early (e.g., a simple causal scenario described in words and illustrated with a DAG)?  \n  – Incrementally layering formalism, while moving dense tables or derivations (e.g., conditional independence relationships) to the appendix?  \n  – Providing a clearer high-level roadmap and smoother transitions across sections?  \n  – Labeling key results as Theorems or Lemmas to help orient the reader?\n\n- **Robustness under model misspecification**: Given that $T_a$ assumes Bayes-optimality, have the authors considered including a qualitative discussion or empirical illustration of how $T_a$ behaves under model misspecification (e.g., due to limited model capacity, regularization, or optimization error)?\n\n- **Guidance for selecting control variable $V$**: Can the authors provide more concrete guidance or examples on how to choose the control variable $V$ in different causal settings? Additionally, would it be possible to illustrate how $T_a$ varies under different reasonable choices of $V$, and how such variation might affect downstream interpretation?\n\n- **Broader empirical evaluation**: Are there plans to extend the empirical evaluation beyond the ACS dataset? Including additional domains—such as healthcare (e.g., MIMIC) or criminal justice (e.g., COMPAS)—or simulating more complex environments involving multiple confounders could help demonstrate the method’s broader applicability and robustness.\n\n- **Stakeholder communication**: Since many stakeholders expect to see raw subgroup disparities, can the authors suggest how conclusions based on $T_a$ might be presented in practice? For example, could templates or visual aids be proposed that show both raw metrics and $T_a$-adjusted insights, to help convey the causal reasoning to non-technical audiences?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- **Strong conceptual contribution**: The paper thoughtfully clarifies why subgroup performance gaps may not reliably indicate unfairness, grounding its arguments in causal reasoning. This represents a meaningful advancement over more simplistic fairness evaluation approaches.\n\n- **Highlights an important challenge in subgroup evaluations**: Many fairness methods rely on detecting metric differences between subpopulations. This work insightfully demonstrates that such disparities may arise from deeper structural confounders, suggesting that raw subgroup gaps should motivate further causal analysis rather than immediate conclusions about unfairness.\n\n- **Diagnostic framework with practical utility**: The controlled evaluation procedure, which conditions away confounding, provides a principled and implementable approach. Its reliance on existing data and plausible causal assumptions makes it accessible to practitioners.\n\n- **Empirical relevance**: The ACS PUMS evaluation effectively illustrates multiple failure modes in fairness assessments, grounding the theoretical insights in real-world data.\n\n**Weaknesses**\n\n- **Clarity and writing style**: The writing is dense and frequently difficult to follow, particularly in the introduction and theoretical sections. Long, multi-clause sentences combine several ideas without clear transitions or structure. The paper makes strong theoretical points, but the dense use of notation and compact causal arguments can hinder readability. Several key ideas would benefit from more intuitive explanation before being formalized. A clearer high-level roadmap and smoother transitions across sections would improve accessibility. This is, in my view, the main weakness of the paper and currently prevents me from increasing my score. However, I would be open to revising my evaluation if the authors significantly improve the exposition.\n\n- **Model misspecification not empirically explored**: Although the paper warns that controlled evaluation may misinterpret results if models are misspecified, it does not empirically assess how robust $T_a$ is with real (non–Bayes-optimal) models. A simple experiment or sensitivity analysis would make the warning more actionable for practitioners.\n\n- **Control variable ($V$) selection remains somewhat abstract**: The authors advise selecting $V$ based on domain knowledge, but practitioners may find this challenging without more concrete guidance. Including specific examples illustrating how to select $V$ in realistic scenarios would make the method more accessible and actionable.\n\n- **Narrow empirical scope**: While the ACS case study is valuable, it is limited to two predictive tasks. Additional experiments in other domains (e.g., healthcare, criminal justice) would strengthen the paper’s generality.\n\n- **Limited procedural guidance**: While the controlled-evaluation concept is generally clear, the paper would benefit from modular pseudo-code or a step-by-step outline describing data preparation, weight computation, and significance testing for $T_a$ to aid reproducibility.", "was_revised": true, "final_justification": "Thank you for the thoughtful and constructive revisions. The added algorithm block, improved clarity, and guidance on selecting V significantly enhance the paper’s accessibility and practical relevance. While the empirical scope is appropriate for the venue, I encourage minor restructuring of the experiment section to make key insights more accessible (e.g., via a summary table). With these changes, I believe the paper is in good shape for acceptance.", "rebuttal": "We thank reviewer Lqbf for their constructive feedback. We particularly appreciate the recommendations as to how to improve the readability and clarity of the work. At the end of the response, we include a detailed list of proposed changes to the manuscript to be made by the time of the camera-ready submission.\n\n**Clarity, writing style, and readability**:\n* We propose to make a series of changes to the manuscript to aid in clarity and readability. This includes breaking up long sentences, adding concrete examples, and adding conceptual scaffolding and transitions to guide the reader through the technical content. \n* As mentioned, we include a detailed description of those changes at the end of this response.\n* To address the concern about procedural reproducibility, we will include an algorithm block describing the computation of the weighted statistic $T_a$ and include code to reproduce the experiments.\n\n**Robustness under model misspecification**:\n\nWe agree that systematic assessment of the role of model misspecification and estimation error would enhance the work. In future work, we hope to advance the theoretical understanding of the estimators and estimation procedures we propose and systematically evaluate the role of misspecification and estimation error. For this purpose, we expect that techniques from the semiparametric estimation literature that treats the optimal predictive model as a nuisance function will provide insight. Epistemic uncertainty quantification (with e.g., ensembles or Bayesian models) may also prove useful as a practical means of assessing variance in the estimator, as a proxy for closeness to Bayes-optimality. We will update the discussion section to refer to these as promising future directions.\n\nWe would also like to clarify that the use of controlled evaluation procedures does not require Bayes-optimality, but interpretation can differ depending on whether the model is Bayes-optimal. In the original manuscript, this is conveyed by the rows of Table 3 that refer to an arbitrary model $f$ that is not Bayes-optimal. We acknowledge that this could be made clearer in the manuscript and we intend to do so through textual revision to sections 3.3 and 3.4. We have also drafted a new table (to be included in the appendix) that details the set of implications and valid conclusions that can be drawn from controlled evaluations based on the configuration of the weights, the covariate set, the observed statistic $T_a$, and whether the model is Bayes-optimal. Comparison of the set of valid conclusions with and without Bayes-optimality clarifies when and how Bayes-optimality is relevant. We include the content of this table in full in the response to reviewer c4Qe.\n\nFinally, we would like to highlight that the experiments with ACS PUMS are plausibly conducted in a practical setting where Bayes-optimality cannot be assumed. As we discuss, interpreting controlled evaluation procedures as conditional independence testing provides the ability to falsify causal assumptions and support understanding of why performance differs across subgroups. In the results section (lines 359-362), we see some deviation from Bayes-optimal behavior through miscalibration of subgroup-aware prediction for the “Other” subgroup in the ACSIncome task. \n\n**Guidance for selecting control variables**:\n\nWe agree that clearer recommendations for control variable choice and interpretation would improve the actionability and accessibility of the work. We intend to include a new paragraph at the end of section 3.4 that includes a clear set of recommendations for use and interpretation of each of the controlled evaluation procedures that we discuss (that is, control for $X$, $Y$, or $R$). We detail the recommendations for interpretation in the new table provided in response to reviewer c4Qe (also mentioned above in the discussion of robustness under model misspecification). We also revise lines 366-370 in the discussion section to clarify that the choice of control variable depends on the research question and inference of interest with the interpretation dependent on causal assumptions.\n\n**Broader empirical evaluation**:\n\nWe agree that further empirical evaluation would strengthen the work. However, we do believe that the present scope is sufficient to address the immediate research questions and appropriate for the space allotted. In future work, we intend to apply our approach in applied contexts (e.g., in healthcare) and conduct a richer theoretical and empirical investigation of methodological considerations not thoroughly addressed here (e.g., model misspecification and finite sample error; separability in high dimensional data).\n\n**Stakeholder communication**:\n\nOur position is that the controlled disaggregated evaluations we propose are complementary to a broader set of analytic practices, including standard uncontrolled evaluations, assessment of calibration, plotting predictive distributions, characterization of the data distribution, reflection on the appropriateness of the problem formulation and assumptions, and transparent reporting. We believe that further interdisciplinary research (e.g., in HCI, human factors, and data visualization) is needed to understand how to best present this broader collection of analyses and visualizations to stakeholders. \n\n**To address this and related concerns, we concretely propose to**:\n* Add additional discussion regarding implications of this work for the general interpretation of disaggregated evaluation.\n* Emphasize the need for future research to understand how to best present controlled evaluation results alongside other findings.\n* Provide new guidance for use and interpretation of controlled evaluations in practical settings, mentioned above and detailed in the response to review c4Qe.\n\n**Here, we outline in detail the proposed changes to aid in clarity**:\n* Introduction\n  * Introduce running examples that can be reused throughout the exposition in the introduction and the presentation of the causal graphs. To expand upon the reference to the use of algorithmic fairness in healthcare, we intend to refer to examples from cardiovascular risk estimation (CVD) and chest X-ray classification (CXR; see refs cited inline). This can be referred to at the end of the first paragraph, again in paragraph two when discussing interventions, and throughout section 3.1 to ground the initial presentation of the causal graphs. \n  * Third paragraph (lines 28-48): \n    * Break up long sentences, including sentences 2 (lines 28-32) and 3 (lines 32-26). \n    * Split this paragraph in two, separately discussing the issues related to subgroup performance stability and misrepresentative data.\n    * Insert a new second sentence (line 29) that presents a high-level, intuitive description of the problem before additional details. \n* Section 3\n  * Add a new paragraph prior to section 3.1 that provides conceptual scaffolding and roadmap for the technical sections that follow.\n  * In section 3.1, present the assumptions of each causal graph in terms of the use cases introduced above (CVD for causal direction graphs and CXR for anticausal).\n  * Section 3.2\n    * Add new exposition to the beginning of the section that clarifies its purpose in relation to the results in the subsequent sections. This can in part be rectified with the new summary section at the beginning of Section 3.\n    * Split up long sentences (lines 184-185, 196-198, 203-208).\n    * Consider moving Table 1 (conditional independence properties of Bayes-optimal models) to the appendix.\n    * Move section 3.5 (subgroup separability) to be a subsection of 3.2. \n  * Section 3.3\n    * Revise the first sentences to emphasize and clarify the purpose of the section. \n    * Clarify that the key findings presented in Table 3 regarding metric stability do not require Bayes-optimality, as Table 3 includes separate rows for arbitrary (not Bayes-optimal) and Bayes-optimal models. Provide explicit guidance about how to use Table 3 for reasoning in practical settings where neither Bayes-optimality nor complete causal structure can be assumed. \n  * Section 3.4\n    * Include an algorithm block that clearly describes the computation of $T_a$ (to include in the appendix).\n    * Include clear guidance for the design and interpretation of controlled disaggregated evaluation results under different conditions (control variable, set of covariates used, value of $T_a$, and Bayes-optimality) without knowledge of the causal graph. This will be incorporated into a revised version of the current final paragraph of this section. This content will be based on and refer to a new table (to be included in the appendix) covering the same content that is outlined in the response to reviewer c4Qe.\n* Experiments\n  * Make minor textual changes to better emphasize the relevance of the subgroup-aware modeling results in relation to the disaggregated evaluation results.\n* Discussion\n  * In the first paragraph, emphasize key practical takeaways. This includes recommendations for interpretation of standard disaggregated evaluations, the implications of the theoretical analysis and empirical findings, as well as guidance for use and interpretation of the methodology presented here.\n  * Revise lines 366-370 to clarify that the choice of control variable depends on the research question and inference of interest with the interpretation dependent on the causal assumptions.\n  * Include mention of additional areas of future research discussed elsewhere in this rebuttal, including the need for further theoretical and empirical investigation into inference under model misspecification and estimation error, extension to broader types of data (continuous, multi-class, high-dimensional, sparse), extension to forms of data misspecification beyond selection (e.g., measurement error), and empirical study in specific contexts of interest (i.e., healthcare)."}]

Author rebuttal:

Meta-review: This paper uses causal models to explore how various settings (such as different types of distribution shift) impact classifiers, and in particular whether Bayes-optimality remains a sufficient condition for the fairness criteria of separation and sufficiency.

Reviewers agreed the paper is well-motivated. They had some concerns about whether it provides practical recommendations, and the authors answered their concerns satisfactorily with proposed revisions.

Reviewers questioned whether focusing on Bayes-optimal classifiers was too narrow a focus and authors agreed to give some additional guidance for the non-optimal case. The paper would also be strengthened by considering criteria other than sufficiency and separation. For example, since causal models have already been introduced, it might not require much additional work to explore a causal fairness definition. This is not a requirement, only an example suggestion. But the current focus on sufficiency and separation is fairly narrow. The fairness literature contains other prominent criteria which are known to be (generally) incompatible with sufficiency/separation, and including any of these would broaden the potential audience for this paper.

The caption for Figure 1 mentions confidence intervals but I only see points. This must be corrected in the revision.
Decision: Accept (poster)