Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
Authors: Drago Plecko
Elias Bareinboim
Venue: NeurIPS 2024 poster (2024)
Keywords: Causal Fairness
Fair Machine Learning
Causal Inference
Trustworthy AI
Primary area: fairness

Abstract: As society increasingly relies on AI-based tools for decision-making in socially sensitive domains, investigating fairness and equity of such automated systems has become a critical field of inquiry. Most of the literature in fair machine learning focuses on defining and achieving fairness criteria in the context of prediction, while not explicitly focusing on how these predictions may be used later on in the pipeline. For instance, if commonly used criteria, such as independence or sufficiency, are satisfied for a prediction score $S$ used for binary classification, they need not be satisfied after an application of a simple thresholding operation on $S$ (as commonly used in practice). 
In this paper, we take an important step to address this issue in numerous statistical and causal notions of fairness. We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation.
We then demonstrate that the marginal difference in the optimal 0/1 predictor $\widehat Y$ between groups, written $P(\hat y \mid x_1) - P(\hat y \mid x_0)$, can be causally decomposed into the influences of $X$ on the $L_2$-optimal prediction score $S$ and the influences of $X$ on the margin complement $M$, along different causal pathways (direct, indirect, spurious). We then show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$. This yields a new decomposition of the disparity in the predictor $\widehat Y$ that allows us to disentangle causal differences inherited from the true outcome $Y$ that exists in the real world vs. those coming from the optimization procedure itself. This observation highlights the need for more regulatory oversight due to the potential for bias amplification, and to address this issue we introduce new notions of weak and strong business necessity, together with an algorithm for assessing whether these notions are satisfied. We apply our method to three real-world datasets and derive new insights on bias amplification in prediction and decision-making.

Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
Drago Plecko and Elias Bareinboim
Causal Artificial Intelligence Lab Columbia University
dp31440columbia..o edu, eb#cs columbia. edu
Abstract
As society increasingly relies Al-based tools for decision-maie makiing in socially sensitive domains, investigating fairness and equity of such automated systems has become critical field of inquiry. Most of the literature in fair machine learning focuses on defining and achieving fairness criteria in the context of prediction, while not explicitly focusing on how these predictions may be used later on in the pipeline. For instance, if commonly used criteria, such as independence or sufficiency, are satisfied for a prediction score S used for binary classification, they need not be satisfied after an application of a simple thresholding operation on S (as commonly used in practice). In this paper, we take an important step to address this issue in numerous statistical and causal notions of fairness. We introduce the notion of margin complement, which measures how much a prediction score S changes due to thresholding operation. We then demonstrate that the marginal difference in the optimal 0/1 predictor Y between groups, written P(ù | x1) P(ù zo), can be causally decomposed into the influences of X on the L2-optimal prediction score S and the influences of X on the margin complement M, along different causal pathways (direct, indirect, spurious). We then show that under suitable causal assumptions, the influences of X on the prediction score S are equal the influences X on the true outcome Y. This yields a new decomposition the disparity in the predictor Ý that allows us to disentangle causal differences inherited from the true outcome Y that exists in the real world vs. those coming from the optimization procedure itself. This observation highlights the need for more regulatory oversight due to the potential for bias amplification, and address this issue we introduce new notions of weak and strong business necessity, together with an algorithm for assessing whether these notions are satisfied We apply our method to three real-world datasets and derive new insights on bias amplification in prediction and -aaiin..
Introduction
Automated systems based on machine learning and artificial intelligence are increasingly used for decision-making a variety of real-world settings. These applications include hiring decisions, university admissions, law enforcement, credit lending and loan approvals, health care interventions, and many other high-stakes scenarios in which the automated system may significantly affect the well-being of individuals [Khandani et al. 2010 Mahoney and Mohen 2007 [Beennan et al.) 2009). In this context, society increasingly concerned about the implications and consequences of using automated systems, compared the currently implemented decision processes. Prior works highlight the potential of automated systems to perpetuate or even amplify inequities between demographic groups, with a range of examples from decision support systems for (among others) senteneing
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Angwin et al. [2016], face-detectoo Buolamwini and Gebru 2018 online advertising Sweeney 2013]. Datta et al. 2015], and authentication |Sanbum| 2015 Notably, issues of unfairness and discrimination are also pervasive in settings in which decisions are made by humans, Some well- studied examples include the gender pay gap, supported by a decades-long literature [Blau and Kahn 1992, 2017], or the racial bias in criminal sentencing (Sweeney and Haney 1992 Pager 2003 J, just to cite few. Therefore, AI systems designed to make decisions may often be trained with data that contains various historical biases and past discriminatory decisions against certain protected groups, constituting a large part of the underlying problem. In this work, we specifically focus on investigating when automated systems may potentially lead to even more discriminatory process, possibly amplifying already existing differences between groups.
Within this context, it is useful to distinguish between different tasks appearing in the growing literature on fair machine learning. One can distinguish three specific and different tasks, namely (1) bias detection and quantification for exisiting outcomes or decision policies; (2) construction of fair predictions of an outcome; (3) construction of fair decision- making policies that are intended to be implemented in the real-world. Interestingly, large portion of the literature in fair ML focuses on the second task of fair prediction, and what is often left unaddressed is how these predictions may be used later in the pipeline, and what kind of consequences they may have. For instance, consider a prediction score S for a binary outcome Y that satisfies well-know fairness criteria, such independence (demographic parity [Darlington] I97I]) or sufficiency (calibration [Chouldechova] 2017)). After a simple thresholding operation, commonly applied in settings with a binary outcome, the resulting predictor is no longer guaranteed to satisfy independence sufficiency, and the previously provided fairness guarantees may be entirely lost. The same behavior can be observed for numerous other measures.
These difficulties do not apply only to statistical measures of fairness, Recently, a growing literature has explored causal approaches to fair machine learning [usnne et al. 2017 Kilbertus et al. 2017
Nabi and Shpitser, 2018 Zhang and Bareinboim 201 al Wu et al.j 2019 Chiappa 2019 Plečko
and Meinshausen 2020 Plečko and Bareinboim| 2024]. which have two major benefits. First, they allow for human understandable and interpretable definitions and metrics of fairesss, which are tied to the causal mechanisms transmitting the change between groups. Secondly, they offer a language that is aligned with the legal notions of discrimination, such as the disparate impact doctrine. In particular, causal approaches allow for considerations of business necessity which aim to ellucidate which covariates may be justifiably used by decision-makers even if their usage implies a disparity between groups. However, causal approaches to fairness also suffer from the above-diseussed issues- namely, a guarantee of absence of causal influence from the protected attribute X onto a predictor S need not hold true after the predictor is thresholded (Plečko and Bareinboim, 2024). Therefore, within the causal approach, there is also a major need for better understanding of how probabilistic predictions are translated into binary predictions or decisions.
In this work, we take an important step in the direction of addressing this issue. We work in a setting with a binary label Y, and the goal is to provide a binary prediction Y or binary decision D. Our approach is particularly suitable for settings in which the utility of the decision is monotonic with respect to the conditional probability of Y being positive, written P(Y | covariates), but the developed tools also have ramifications for more general utilities. Examples that fall under our scope are numerous; for instance, the utility of admitting a student to the university (D) is often monotonic in the probability that the student successfully graduates (Y)). In the context of criminal justice, decisions of detention (D) are used to prevent recidivism, and the utility of the decision is monotonic in the probability that the individual recidivates (Y). Finally, various preventive measures in healthcare (D, such as vaccination, screening tests, etc.) are considered to be best applied to individuals with the highest risk of developing a target disease or suffering a negative outcome (Y). We now provide an illustrative two-variable example for one of the key insights of our paper:
Example 1 (Disparities in Hiring). Consider a company deciding to hire employees using an automated system for the first time. From previous hiring cycle when humans were in charge, the company has access to data on gender X (wo for female, In for male) and the hiring outcome Y (I for being hired, 0 otherwise). The true underlying mechanisms of the system given by:
X + 1(Ux 0.5))
(1)
Y - ( [1UV < po) if = 30 1(Ux < p1) if = =I
(2)
2

---

Not hired (36)
Male (irg)
Hired (yr)
Female (irp)
After (8) Before
Thresholding
Figure Visualization of hiring disparities from Ex.
where Ux,Uy Onif(0. 1]. In words, an applicant is female with a 50% probability, and the probability of being hired as female zo is Po. whereas for males I1 the probability is Pt- The company finds the optimal prediction score S to be S(x) P2. The optimal 0/1 predictor Y, which will also be the company's decision, is given by Y (x) = (() 2 2), meaning that the company will threshold the predictions at 5. Suppose that pa = 0.49, p1 0.51, and consider the visualization in Fig. The probability po 0.49 means that 49 out of 100 females were hired, while 51 were not. After thresholding. Yxx)) = 1 (po 2) 0 for each female applicant, meaning that the thresholding operation maps the prediction of each individual to O, even though 49/100 would have positive outcome (Fig. Diig.rritte right). Similarly, for pl 0.51, we have that 51/100 male applicants would be hired, resulting in thresholded predictor Y (x1) = 1 for all the applicants, even though 49/I00 would not have a positive outcome (Fig. left). Therefore, the gender disparity in hiring after introducing the automated predictor Y is 100%, compared to 2% disparity in the outcome Y before introducing Y.
Formally, the disparity in the optimal 0/1 predictor P( x1) P(ū xo) = 1p) 2 2) 1(p0 2 2 can be decomposed as:
-
P( x1) - P 20) P (1 2 2 - pl) (1 2 2 Pu)
(3)
Term
Term measures the disparity coming from the true outcome Y (2% disparity). which is equal to the disparity in the prediction score S. written P(s 21) - P(s Tо). Term Il measures the contribution coming from thresholding the prediction score to obtain an optimal 0/1 prediction (98% disparity). O
The above example illustrates a canonical point in a simple setting: a small disparity the outcome Y, and consequently the prediction score S, may result in a large disparity in the optimal 0/1 predictor Y. we call bias amplification. Contrary to this, large disparity in S may also result in a small disparity in Y, case call bias amelioration.
In the remainder of the manuscript, our goal is to provide a decomposition of the disparity in a thresholded predictor Y into the disparity in true outcome Y and the disparity originating from optimization procedure, but along each causal pathway between the protected attribute X and the predictor Y. particular, our contributions are the following:
(1) We introduce the notion of margin complement (Def. [ and provide a path-specific decom- position of the disparity the 0/1 predictor Y into its contributions from the optimal score predictor S and the margin complement M (Thm. D
(2) We prove that under suitable assumptions, the causal decomposition of the optimal prediction score S is equivalent with the causal decomposition of the true outcome Y (Thm. 2). This allows us to obtain a new decomposition of the disparity in Y into contributions from Y and the margin complement M (Cor. 3)
(3) Motivated by the above decompositions, we introduce new concept of weak and strong business necessity (Def. 3, highlighting a new need for regulatory instructions in the context of automated systems. We provide an algorithm for assessing fairness under considerations of weak and strong business necessity (Alg.) D..
3

---

(4) We provide identification, estimation, and sample influence results for all of the quantities relevant to the above framework (Props. 4 [.. We evaluate our approach on three real-world
examples (Ex. 2 3)
and provide new empirical insights into bias amplification.
Our work is related to the previous literature on causal fairness and the causal decompositions appearing in this literature (Zhang and Bareinboim] 2018b Plečko and Bareinboim] [024]. It is also related to previous literature on studying business necessity requirements through a causal lens [Kilbertus et al., 2017 Plecko and Bareinboim 2024b]. However, our approach offers an entirely new causal decomposition into contributions from the true outcome Y and the margin complement M. More broadly, our work is also related to the literature on fair recision-making
which analyzes
how prediction scores impact the fairness of decisions [Chouldechova] 2017 Dwork et al. 2020
Chouldechova and Roth 2018), or how disparities evolve over time [Liu et al. 2018]. Recent
results also show that focusing purely on prediction, and ignoring cecision-making aspects, may lead to incquitable outcomes and cause harm to marginalized groups [Plečko and Bareinboim [024]]
Nilforoshan et al.] 2022] Plecko and Bareinboim [024a]. highlighting a need to expand focus from narrow statistical definitions of fair predictions
more comprehensive understanding of equity in algorithmic decisions. Still, many questions remain open in the context of fair decision-makin and more future works required in this area. Finally, we mention that our work is also related to the literature on auditing and assessing fairness of decisions made by humans (Pierson et al. 2021 Kleinberg et al. 2018), and understanding how AI systems may help humans overcome their biases Imai et al. 2023
1.1 Preliminaries
We use the language of structural causal models (SCMs) [Pearl 2000]. An SCM is a tuple M == (V,U, U, P(u)), where V, U are sets of endogenous (observable) and exogenous (latent) variables, respectively, F is a set of functions fv. one for each Uv;) for some pa(V.) V and Uv, C U. The set pa( Vi) is called the parent set of V., P(u) is strictly positive probability measure over U. Each SCM M is associated to a causal diagram G [Bareinboim et al.
7
X
V, E V. where V fv (pa
W
Figure 2: Standard Fairesss Model.
2022 over the node set V where V + V2 if Vú2 is an argument of fv,. and V +--> V, if the corresponding Uv, Uv, are not independent. An instantiation of the exogenous variables U = is called unit. By Yz (z( we denote the potential response of Y when setting X = x for the unit u, which is the solution for Yu)) to the set of equations obtained by evaluating the unit in the submodel MM,,, in which all equations in F associated with X are replaced by X = x. Throughout the paper, we assume a specific cluster causal diagram GSFM known as the standard fairness model (SFM) (Plečko and Bareinboim 2024
over endogenous variables {X, Z, W. Y, Y shown in Fig. 2 (see also [Anand et al. (023)). D. The SFM consists of the following: protected attribute, labeled X gender, race, religion), assumed to be binary; the set of confounding variables Z, which are not causally influenced by the attribute X (e.g., demographic information, zip code); the set of mediator variables W that are possibly causally influenced by the attribute (e.g., educational level or other job-related information); the outcome variable Y (e.g.. GPA, salary); the predictor of the outcome Y (e.g.. predicted GPA, predicted salary). The SFM also encodes the lack-of-confuunding assumptions typically used in the causal inference literature. The availability of the SFM and the implied assumptions are a possible limitation of the paper, while we note that partial identification techniques for bounding effects can be used for relaxing them [Zhang et al. 2022
2 Margin Complements
We begin by introducing a quantity that plays key role in the results of this paper.
Definition 1 (Margin Complement). Let U = be a unit, and let S denote a prediction score for binary outcome Y. Let the subscript C denote a counterfactual clause, so that Ze denotes a potential response. The margin complement M of the score S for the unit U = and threshold t is defined as:
M(u) 1(S(u) 2 1) - S(u).
A potential response of M, labeled Mc, is given by Mc(u) ((So(u) 2 t) - -Sc(u).

---

In words, the margin complement for a unit U = u represents the difference in the score after thresholding vs, the score that would happen naturally.
Example 1 (Disparities in Hiring continued). Consider the hiring example with the SCM in Eqs. 72
with po = 0.49, P1 = 0.51. The unit (Ux, -v) = (0, 0) corresponds to a male applicant (X (X(u) = who was hired (u)
1). We have S(u) p1 0.51, and (u) (((()) 2 0.5)
S(u) = - 0.51 = 0.49. For this u, the margin complement indicates that the predicted outcome
Yuu) = (S) 2 0.5) is 49% greater than the predicted probability S(u). The same computation can be done for a female X(u') = O, in which case M (u') = 1ppo 2 0.5) - -Pxa = -0.49, meaning that the predicted outcome Y (u') ) is 49% smaller than the predicted probability S(u').
Given prediction score S and threshold t. the margin complement M tells us in which direction the thresholded version L (SS())= 2 t) moves compared to the score Suu). A positive margin complement indicates that a thresholded predictor is larger than the probability prediction, and negative margin complement the opposite. A similar reasoning holds for the potential responses of the margin complement Mc: we are interested in what the margin complement would have been for an individual U = u under possibly different, counterfactual conditions described by C. As we demonstrate shortly, margin complements (and their potential responses) play a major role explaining how inequities are generated between groups at the time of decision-making. In this section, our key aim is to analyze the optimal 0/1 predictor Y and provide decomposition of its total variation measure (TV, for short), defined as ao., (ū) = P(ū x1) P zn). When working with the causal diagram in Fig. 2 can notice that the TV measure comprises of three types of variations coming from X: the direct effect X + Y. the mediated effect X + W + Y, and the confounded effect X +-- Z Y. Our goal is to construct a decomposition of the TV measure that allows us to distinguish how much of each of the causal effects due to a difference in the prediction score S, and how much due to margin complements M. To investigate this, we first introduce the known definitions of direct, indirect, and spurious effects from the causal fairness literature:
Definition 2 (1-specific Causal Measures (Zhang and Bareinboim 2018b
Plečko and Bareinboim
2024). ). The z-specific (direct, indirect, spurious) effects of X on are defined as:
x-DExoo.11 (y x) P(Ve1,
x)) P(yeo x)
x- x--EEs, wo (y 2)) P(ynr Wro x)) P(ye, |x)
(6)
x-SE,, .0 (v) P(ya, (F)) P(yx, σ1).
(7)
Armed with these definitions, we can prove the following result:
Theorem 1 (Causal Decomposition of Optimal 0/1 Predictor). Let Ý be the optimal predictor with respect to the 0/1-loss based on covariates X, Z, W. Let S denote the optimal predictor with respect to the L2 loss. The total variation (TV. for short) measure of the predictor Y, written as Pg x1) Plğ zo), can be decomposed into direct, indirect, and spurious effects of X on the score S and the margin complement M as follows:
TV 20.21 (0) = DD-Ezz, (s 200 + D--E (m [oo)
(8)
(I-IEm, no (8 x0) x-IEE, an (ml xo))
(9)
(x-SEz, (s) + 2-SEz,,.0 (m)).
(10)
The above theorem is the first key result of this paper. The disparity between groups with respect to the optimal Ollssoss predictor, measured by ...,, (i) can be decomposed into direct, indirect, and spurious contributions coming from (i) the optimal L2-loss predictor S (c.g.. term x-DE (s zo)), and (ii) the margin complement M (e.g., term x--Ez,,2, (m xo)). This provides a unique capability since for each causal pathway (direct, indirect, spurious) the contribution coming from the probability prediction S can be disentangled from the contribution coming from the optimization procedure itself (i.e., the rounding of the predictor). The former, as we will see shortly, is simply a representation of the bias already existing in the true outcome Y, whereas the latter represents a newly introduced type of bias that is the result of using automated system. The contribution of the margin complement may act to both ameliorate or amplify an existing disparity, a point we investigate later on.
Example 1 (Disparities in Hiring extended). Consider the hiring example from Ex. D eexended with a mediator W indicating whether the applicant has a PhD degree (W = 1) or not (W = =0), Suppose
5

---

the augmented SCM is given by:
X t 1Ux < 0.5)
(11)
W < 1(Uw < 0.5 + AX)
(12)
Y - 1(0 < <.11 ax + 3W),
(13)
and P(Ux, Uw, Uv) is such that Ux, Uw, Uy are independent On 1] random variables. The coefficients satisfy the constraints a, X > 0,0.4 < B 0.-0
1+X The optimal probability predictor
is Sxx, w) = 0.1 + ax + Bw. The TV measure of the optimal 0/1 predictor be computed as TV 1.z (0) a
B Using Thm. 7 we can decompose it as:
TV 21 (7)
(A (A-A3)
(14)
a-DEzo.a1 (s)
w-1Ex1 (s/po)
2-1E=1 Fo (m//aa)
x. x-DEug.er mn (m)za)
Along the direct effect, there is difference of a belween the groups in the probability predictor (ix, w) that amplified by 3 -a a by the margin complement, and since 3 < we always have bias amplification. Along the indirect effect, there is an initial difference of X3, amplified by an additional X - X3, again amplifying bias. The decomposition provides breakdown of the disparity of the optimal 0/1 predictor into its constitutive parts.
We next consider crucial point mentioned above. The disparity observed in the optimal L2 score S is simply a representation of the disparity existing in the real world, under suitable causal assumptions:
Theorem 2 (Relationship of Lc-score and Outcome Decompositions). Let M be an SCM compatible with the Standard Fairness Model, and let S be the optimal L2 prediction score. Then, the causal decompositions of the score and the true outcome Y are symmetric, meaning that x- x- vo (8 xo) = x--E ao (y mo), and x-SEE ao (8) = x-SEE (w))
x- D-DEx (8 x00) = D-DEs (y mo)
(15)
1-1E 1E 20 (s xo)) I--FEa o (y aro)
(16)
x- x-SE (8) = x--EE mo (y).
(17)
Based this result, equivalent to the causal influences from X on the optimal prediction score S. Combining this insight with Thm. Ileads to the following result:
that the causal influences from X on the true outcome Y
Corollary 3 (Causal Decomposition of Optimal 0/1 Predictor). Under the Standard Fairness Model, the TV measure of the optimal 0/1 oll-loss predictor Ý can be decomposed into contributions from the true outcome and the margin complement M:
TV TV ao (T) D-DE (y Hо) x--D (m |x0)
(18)
(I-1Ez, (y) ra) x-11E (m vo))
(19)
(x-SE (2) + x 2-5
(20)
This corollary provides important insight compared to Thm. | the disparity in the optimal predictor Y can be decomposed into the contribution inherited from the true outcome Y and the contribution that arises from the optimization procedure (rounding of the predictor). In Appendix D we describe two important extensions of the main results: (i) to the setting of suboptimal predictors; (ii) the setting of predictors using group-specific thresholds, which is often the case with post-processing methods used in fair machine learning [Kamiran et al. 2012 Hardt et al. 2016
3 Weak and Strong Business Necessity
In law, questions of faimess and discrimination can be interpreted based on the disparate treatment (DT) and disparate impact (DI) doctrines of Title VII of the Civil Rights Act of 1964 [Act] 1964). The DT doctrine, when interpreted causally, can be seen disallowing direct type of effect from X onto the outcome. The doctrine, however, more broad, and may prohibit any from of discrimination (be it direct, indirect, or spurious) that results in large disparity between groups The core of this

---

Algorithm 1: Auditing Weak & Strong Business Necessity
Input: data D, BN-Set S. (DE, IE, SE}, BN-Strength, predictor Y, score S, SFM Output: SUCCESS or FAIL of ensuring that disparate impact and treatment hold under BN
foreach CE (DE, IE, SE) do 2
Compute the effects v-- uy-C((y) s-CCE(m, x-CC(s), x-CH(T)
if CE E Strong- BN then
Assert that xBC((() = x- xyC otherwise FAIL.
else if CE E Weak-Bh BN then
Assert that x-CB(s) = z-CE A x-CE(m) = 0, otherwise FAIL
else
Assert that z-C z-CE m) 0, otherwise FAIL
if not FAIL return SUCCESS
doctrine is the notion of business necessity (BN), which allows certain variables correlated with the protected attribute to he used for prediction due to their relevance to the business itself Els 1993] (or more broadly the utility of the decision-maken Based on the decomposition from Cor new BN
considerations emerge:
Definition 3 (Weak and Strong Business Necessity). Let M be an SCM compatible with the Standard Fairness Model. Let CE denote causal pathway (DE, IE, or SE), and let x, x' be two distinct values of X. Let x" be third, arbitrary value of X. If causal pathway does not fall under business necessity, then we require:
r-CEr x²)) ae x²) =0. -.
(21)
A pathway is said to satisfy weak business necessity if:
x-- CE x²) = x- I-CEz,x x²), x- x-CEE (m 1x") = 0.
(22)
A pathway is said to satisfy strong business necessity if:
z- D--Cr x²) = x- x-CEz, x"), while x- a-CEn,, x²) takes any arbitrary value.
(23)
The above definition distinguishes between three important cases, and sheds light aspect of the concept of business necessity. According to the definition, there are three versions BN considerations:
(1) A causal pathway is not in the BN set, and is considered discriminatory. In this case, both the contribution of the prediction score S and the margin complement M need to be equal to 0 (i.e., no discrimination is allowed along the pathway),
(2) A causal pathway satisfies weak BN, and is not considered discriminatory. In this case, the effect of X on the prediction score S needs equal the effect of X onto the true outcome Y along the same pathway [Plecko and Bareinboim 2024b). However, the contribution of the margin complement M along the pathway needs equal O.
(3) A causal pathway satisfies strong BN, and is not considered discriminatory. Similarly as for weak necessity, the effect of X on needs to equal the effect of X on Y, but in this case, the contribution of the margin complement M is unconstrained.
The distinction between cases (2) and (3) opens the door for new regulatory requirements and specifications. In particular, whenever a causal effect is considered non-discriminatory. the attribute X needs to affect S to the extent to which it does in the real world. However, the system designer also needs to decide whether difference existing in the predicted probabilities S is allowed to be amplified (or ameliorated) by means of rounding. The latter point distinguishes between weak and strong BN, and should be consideration of any system designer issuing binary decisions. In Alg. I we propose formal approach for evaluating considerations of weak and strong BN for any input of predictor y and prediction score S.
7

---

4 Identification, Estimation, and Sample Influence
In Thm. 1 and Cor. 3 the observed disparity in the TV measure of the optimal 0/1 predictor is decomposed into its constitutive components. The quantities appearing in the decomposition are counterfactuals, and thus the question of identification of these quantities needs to be addressed. In other words, we need to understand whether these quantities can be uniquely computed based on the available data and the causal assumptions. The following is a positive answer:
Proposition 4 (Identification and Estimation of Causal Measures). Let M be an SCM compatible with the Standard Fairness Model, and let P(V) be its observational distribution. The z-specific direct, indirect, and spurious effects of X on the outcome Y, predictor Y, prediction score S, and the margin complement M are identifiable (uniquely computable) from P(V) and the SFM. Denote by f(z, z, w) estimator of [[ r, 2, w), and by P(z v') the estimator of the probability P(æ e') v'. for different choices v'. For T E S,, S,MM,, the effects can be estimated as:
P(ro (1,
x- D--E
mo.2r (t z0) =
Σ [f (1, 21) - f(zo, ws. w))
(24)
P(xo)
i=1
x- x-1E
11 (t 200)
=
n1
Σ
f( W/, 2x)
P( ((wo)
We 21)
P Ws, P(xo 2x))
P(ri 1 22)
P(a0)
(25)
1
P(x1 Way 24) (3)21) P(xo (2)) P(11 Wis 24)
x-SEEst 20
(1)
f(1 Wi, zi)
(26)
n1
P(r1 2i)
P(an)
P(x1)
The proof of the proposition, together with the identification expressions for the different quantities can be found in Appendix B We next define the sample influences for the different estimators:
Definition 4 (Sample Influence). The sample influence of the i-th sample on the estimator x- -C of the causal effect CE is given by corresponding term in the sunmations in Egs. 26 For instance, the i-th sample influence on D.D (f |xo))) is given by (and analogously for IE, SE terms):
SI-DE(i) = [f(x,, Ws, z;) [100 Wi, 2]] P(xo Wi, 21)
(27)
P(ro)
The sample influences tell us how each of the samples contributes to the overall estimator of the quantity. These sample-level contributions may be interesting investigate from the point of view of the system designer, including identifying any subpopulations that are discriminated against. For direct sample influences, the following proposition can be proved:
Proposition 5 (Direct Effect Sample Influence). The SI-DE(i) in Eq. 27 is an estimutor of
P(Ws, 23 xo)
BB E[TT,,We, - -uoo Vo. Zsn ws]
(28)
P(w, 24)
where E(T ETT,,,.., Tan To, 21, wi] is the (zo, 24, wx) wx)-specific direct effect of X T.
Prop. (epp.5)dmmoooo demonstrates an important point- - namely that the sample influences along the direct path are not just quantities of statistical interest, but also causally meaningful quantities. In particular, the influence of the i-th sample is proportional to the direct effect of the Zo + Ii transition for the group of units u compatible with the event 2o. 2;, W. The influence is further proportional to
P(w,, P 2)) that measures how much more likely the covariates 2,, we of the i-th sample are in the zo group (for which the discrimination is quantified) vs. the overall population. Therefore, practitioners also have a causal reason for investigating these sample influences.
5 Experiments
We analyze the MIMIC-IV (Ex. D. COMPAS (Fx. and Census (Ex. Appendix datasets.
Example 2 (Acute Care Triage on MIMIC-IV Dataset Johnson et al. 2023 ). Clinicians in the Beth
Israel Deaconess Medical Center in Boston, Massachusetts treat critically ill patients admitted to
8

---

(a)
MIMIC-IV Decomposition
(b) DE Sample Influences
Sample Inft Intuuance aace
Effect Size
0.000
0.025
3.000
CH-DE
DIHE Causal Eneect
10000
Sample Ranking
Component
Outcome
Margin Complement
Component
Ouicome
Margin Complement M
Figure 3: Causal decomposition from Cor. and sample influence on the MIMIC-IV dataset.
the intensive care unit (ICU). For all patients, various physiological and treatment information is collected 24 hours after admission, and the available data consists of (grouped into the Standard Fairness model): protected attribute X, in this case race (Io African-Americon, 21 White), set of confounders Z = [sex, age, chronic health status), set of mediators W =flactate, SOFA score, admission diagnosis, PaOx/FiO, ratio, aspartate aminotransferase).
Clinicians are interested in patients who require closer monitoring. They want to determine the top half of the patients who are the most likely to (i) die during their hospital stay; (ii) have an ICU stay longer than 10 days. This combined outcome is labeled Y. These high-risk patients will remain in the most acute care unit. To predict the outcome, clinicians use the electronic health records (EHR) data of the hospital and construct score predictions S and binary predictor Y (SS w) > Quant(0.5;: S)) that selects the top half of the patients.
To investigate the fairness implications of the new Al-based system, they use the decomposition described in Cor. 3 to investigate different contributions to the resulting disparity. The decomposition shown in Fig. 3aa), and uncovers a number of important effects. Firstly, along the direct effect, x x-DE=..2, (y) and x-DEm (m) are larger than O, meaning that minority group individuals have lower chance of receiving acute care (purely based on race). Along the indirect and spurious effects, the situation is different: x-1Ez1,20 (3) and x-SEz1,za (y) and their respective margin complement contributions are different from and negative -implying that minority group individuals have a larger probability of being given acute care result of confounding and mediating variables. Finally, the direct effect sample influences (Fig. (b) highlight that the margin complements are large for a small minority of individuals, requiring further subgroup investigation by the hospital leam. [
Example 3 (Recidivism Prevention on the COMPAS Dataset [Larson et al., 2016). Courts in Broward County, Florida use machine learning algorithuss, developed by private company called Northpointe, predict whether individuals released on parole are at high risk of re-offending within 2 years (Y). The algorithm is based on the demographic information Z (Zi for gender, Ze for age), race X (xn denoting White, Ii Non- White), juvenile offense counts J. prior offense count P. and degree of charge D. The courts wish to know which individuals are highly likely to recidivate, such that their probability of recidivism is above 50%. The compuny constructs a prediction score SNP and the court subsequently uses this for deciding whether to detain individuals high risk of re- re-offending.
After a court hearing in which it was decided that the indirect and spurious effects fall under business necessity requirements, team from ProPublica wishes to investigate the implications of using the automated predictions SNP. They obtain the relevant data and apply Alg. 1 with the results shown in Fig. 4 The team first compares the decompositions of the true outcome Y and the predictor SNP (Fig. 4ra)). For the spurious effect, they find that 2o0 (y) is not statistically different from I-SEz,, (NNP), in line with BN requirements. For the indirect effects, they find that the indirect effect is lower for the predictor SNP compared to the true outcome Y, indicating no concerning violations. However, for the direct effect, while the x D--Ex (y) is not statistically different from O, the predictor SP has a significant direct effect of X, iee, x-DEa (BNP) + O. This indicates violation of the fairness requirements determined by the court.

---

(8)
COMPAS S
and Decompiciitions
COMPAS Decomposition
Influenccs
004
0.0
Sampl intuerre
Efoct Bize
0.0-
4.03
0H/08
DR-E Caudal Efecc
Oaie
20-B Cousel Effcc
Ch-Se
2000
4000
5002
Sample Taanking
Componant
Companant
Figure 4: Application of Alg. the COMPAS dataset.
After comparing the decompositions of SNP and the team moves onto understanding the contribu- tions of the margin complements (Fig. TD. For each effect, there is pronounced impact of the margin complements, For the direct effect (not under BN), the non-zero margin complement contribution x-DEs1 (m) (m) * O represents violation of fairness requirements. For the indirect and spurious effects, the ProPublica team realizes the court did not specify anything about margin complement contributiions based on this, for the next court hearing they preparing argument showing that the effects 2-1E2,,20 (m) and x-SEzz,2., ) are significantly different from 0, thereby exacerbating the differences between groups. Finally, based sample inffuences (Fig. [4c), they realize that the direct effect is driven by small minority of individuals, and they decide to investigate this further. D
6 CConclusion
In this paper, developed tools for understanding the fairness impacts of transforming a continuous prediction score S into binary predictions Y or binary decisions D. In Thm. | and Cor, we showed that the TV measure of the optimal 0/1 predictor decomposes into direct, indirect, and spurious contributions that are inherited from the true outcome in the real world, and also contributions from the margin complement M (Def. TT arising from the automated optimization procedure. This observation motivated new notions weak and strong business necessity (BN) - in the former case, differences inherited from the true outcome Y are allowed to be propagated into predictions or decisions, while any differences resulting from the optimization procedure are disallowed. In contrast, strong BN allows both of these differences and does not prohibit possible disparity amplification. In Alg. we developed a formal procedure for assessing weak and strong BNN. Finally, real-world examples demonstrated that the tools developed in this paper are of genuine importance in practice, since converting continuous predictions into binary decisions may often result bias amplification in practice highlighting the need for this type of analysis, and the importance of regulatory oversight.
References
Elston Talladega County Bd. of Educ. 997 F:2d 1394 (11th Cir. 1993), 1993. United States Court of Appeals for the Eleventh Circuit.
C. Act. Civil rights act of 1964. Title VII, Equal Employment Opportunities, 1964.
T. V. Anand, A. H. Ribeiro, J. Tian, and E. Bareinboim. Causal effect identification in cluster dags. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(10), pages 12172-12179, 2023.
Angwin, Larson, Mattu, and L. Kirchner. Machine bias: There's soft- ware used across the country to predict future criminals. and it's biased against blacks. ProPublica. 5 2016. URL https: ://p:://wwww wwwpwwpoppblccc...mm wtt.iii...rg/rtiille... machine- bias sees-riikk-ao assessments- in
10

---

E. Bareinboim, J. D. Correa, D. lbeling, and T. Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl, page 507-556. Association for Computing Machinery, New York, NY, USA, 1st edition, 2022.
F. D. Blau and L. M. Kahn. The gender earnings gap: learning from international comparisons. The American Economic Review, 82(2):533-538 1992.
D. Blau and L. M. Kahn. The gender wage gap: Extent, trends, and explanations. Journal of economic literature, (5(3):789-865. 2017.
T. Bremnan, W. Dieterich, and B. Ehret. Evaluating the predictive validity of the compas risk and needs assessment system. Criminal Justice and Behavior, 36(1):21-40, 2009.
J. Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities commercial gender classification. In S. A. Friedler and C. Wilson, editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, pages 77-91, NY, USA, 2018.
S. Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7801-7808, 2019.
A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Technical Report 1rXiv:1703.00056 arXiv.org, 2017.
A. Chouldechova and Roth. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810, 2018.
R. B. Darlington. Another look at "cultural fairness" 1. Journal of educational measurement, 8(2): 71-82, 1971.
A. Datta, M. C. Tschantz, and Datta. Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. Proceedings on Privacy Enhancing Technologies, 2015(1): 92-112, Apr. 2015. doi: 10. 10.1515/popets- 0ttp/poopptt-225-00077
C. Dwork, C. Ilvento, and M. Jagadeesan. Individual fairness pipelines. arXiv:2004.05167, 2020.
arXiv preprint
M. Hardt, E. Price. and N. Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29:3315-3323 2016.
K. Imai, Z. Jiang, D. J. Greiner, R. Halen, and Shin. Experimental evaluation of algorithm- aassiste assisted human decision-making Application to pretrial public safety assessment. Journal of the Royal Statistical Society Series A: Statistics in Society, 186(2):167-189, 2023.
A. E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, B. Moody, B. Gow, L.-w. H. Lehman, et al. Mimic-iv, freely accessible electronic health record dataset. Scientific data, 10(1):1, 2023.
F. Kamiran, A. Karim, and X. Zhang. Decision theory for discrimination-awarr classification. In 2012 IEEE 12th International Conference on Data Mining, pages 924-929. IEEE, 2012.
A. E. Khandani, A. J. Kim, and A. W. Lo. Consumer credit-risk models via machine-learning algorithms. Journal of Banking & Finance, 34(11):2777-2787 2010.
N. Kilbertus, M. Rojus-Carulla, G. Parascandolo, M. Hardt, D. Janzing, and B. Schölkopf. Avoiding discrimination through causal reasoning, arXiv preprint arXiv:1706.0274 2017.
Kleinberg, H. Lakkaraju. Leskovec, J. Ludwig, and S. Mullainathan. Human decisions and machine predictions. The quarterly journal of economics, 133(1):237-293, 2018.
M. J. Kusner, J. Loftus, C. Russell, and R. Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
I Larson, S. Mattu, Kirchner, and J. Angwin. How we analyzed the compas recidivism algorithm. ProPublica (5 2016), 9, 2016.

---

L. T. Liu, S. Dean, E. Rolf, M. Simchowitz, and M. Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning, pages 3150-3158. PMLR, 2018.
Mahoney and J. Mohen. Method and system for loan origination and underwriting, Oct. 23 2007. US Patent 7,287,008.
R. Nabi and I. Shpitser. Fair inference outcomes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
H. Nilforoshan, J. D. Gaebler, R. Shroff, and S. Goel. Causal conceptions of faimess and their consequences. In International Conference on Machine Learning, pages 16848-16887. PMLR, 2022.
D. Pager. The mark of a criminal record. American journal of sociology, 108(5):33--775, 2003.
J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, New York, 2000. 2nd edition, 2009.
E. Pierson, D. M. Cutler, J. Leskovec, S. Mullainathan, and Z. Obermeyer. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nature Medicine, 27(1): 136-140 140, 2021.
D. Plečko and E. Bareinboim. Causal fairness analysis: A causal toolkit for fair machine learning. Foundations and Trends@ in Machine Learning, 17(3):304-589, 2024.
D. Plecko and Bareinboim. Causal fairness for outcome control. Advances Neural Information Processing Systems, 36, 2024a.
D. Plecko and E. Bareinboim. Reconciling predictive and statistical parity: A causal approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38 (13), pages 14625-14632, 2024b.
D. Plečko and N. Meinshausen. Fair data adaptation with quantile preservation. Journal of Machine Learning Research, 21:242, 2020.
J. Sanburn. Facebook thinks some native american names are inauthentic. Time, Feb. 14 2015. URL http://time.cm wwtp:/ee/m///00/////a/a////
nati ve- ameri can names/
Shpitser and J. Pearl. What counterfactuals can be tested. Proceedings of the Twenty-third Conference on Uncertainty in Artificial Intelligence, page 352-359, 2007.
L. Sweeney. Discrimination in online ad delivery. Technical Report 2208240, SSRN, Jan. 28 2013. URL http::/dxx.... http//xxxxooi.....o doi. ooooorg//11 2139/ssrn. 2208240
L. T. Sweeney and C. Haney. The influence of race on sentencing: meta-analytic review of experimental studies. Behavioral Sciences & the Law, 1092):199-195, 1992.
Y. Wu, Zhang, X. Wu, and H. Tong. Pe-fairness: A unified framework for measuring causality- based faimess. Advances in neural information processing systems, 32, 2019.
J. Zhang and F. Bareinboim. Equality of opportunity in classification: A causal approach. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 3671-3681, Montreal, Canada, 2018a. Curran Associates, Inc.
Zhang and E. Barcinboim. Fairness in decision -arioon--aiin----a the causal explanation formula. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018b.
Zhang, Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. In Proceedings of the 39th International Conference on Machine Learning, 2022.
12

---

Technical Appendices for Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
The source code for reproducing all the experiments can be found in our Github code repository https://gtt thub. com/ httpp/ppp.edddddteeeeeeeee
gga, The code also included with the supplemen- All experiments were performed on a MacBook Pro, with
tary materials, in the folder seurce- the M3 Pro chip and 36 GB RAM on macOS 14.1 (Sonoma). All experiments can be run with less than hour of compute on the above-described machine or equivalent.
A Proofs of Key Theorems
Thm. Proof: Notice that by definition we have
Mc(u) = (w) 2 t) - Sc(u) Vu E U.
(29)
Now, let E be any observed event, and let C be counterfactual clause representing a possibly counterfactual intervention. Note that we can write:
c
E
(def)
Polu)P(ul B) = Σ En(So(u) 2 P) E)
(30)
(I(Sc(u) 2 - Sc(u) + Sc( P(u| F))
(31)
Σ (I(SE(w) 2 1) - So(u)) P(u E) + Σ Sc(w)P(ul E)
(32)
Mc(u)P(u E) Σ Sc(u)P(u E))
(33)
E[Me| E] + E|Sc EJ.
(34)
Now, we can expand the TV measure of Ý follows:
TV 20.01 (0) = y x1] y xo]
(35)
EE(Y,, |x|] - IE(",, [o]]
(36)
EE(?,, wo] - IE| p
W =0
[50]
(37)
TF(22, W, [ro]
TF V 250
[no]
(38)
E[[S,] [m]] [F(
mo)
+ [F[
[21] IF[Ma, 37
(39)
+ ES [o] EE B[Saa W
[r0]
+ IE [E[M,, xo] - IE| [E[Ma W Xo]
(40)
EE(S,, Wag
[00] EE
rr) + EE [May Wio [xo]] - IE Ma. Xo]
(41)
= -x-SB 1 20 (8) - SE-SEE (m)
(42)
-I-1EE mn (s [0) x- (m xo)
(43)
+ +D-DE mn m (s |xo) + xD-DE (m xo),
(44)
completing the proof.
Thm. Proof: Let M be an SCM compatible with the SFM in Fig. Ls per theorem assumption. Let the optimal La prediction score S be given by S(x, z, w) = Fy let fs be the structural mechanism of S, taking X, Z, W inputs. The structural mechanism of S is then given by:
x, z, w). More precisely, we can
fs 2, w) E Y x, w].
(45)
Therefore, the score S a deterministic function of X, 7, W, and we can add it to the standard fairness model as shown in Fig, 5a Now, note that for potential outcome E[Sa.w.r x²] we can
13

---

7
T 10
W
S
W
W
(a) SFM extended with predictor
(b) Counterfactual graph the SFM.
Figure 5: Graphs used in proofs of Thm. and Prop.
write:
E S,,, |24]= = Σ W. |x", x²)
(46)
Σ IE B 1(W w) x² 2 x²)
(47)
Σ
IESS
"[,, =w) x" ,=,=()(z)) x")
(48)
Σ BBSSa. P(W = x" P) x²")
(49)
2,00
Σ B P(W = w x² P(2 x²")
(50)
2,00
Σ 1 20 1( = w) 10"=(P(=z x")
(51)
2,30
E|Y,,,,, 22",2|p(2 (PP(2xx") xx") = E[Y,,w., xx]]
(52)
where Eq. 48 is using the independence w W X, Z, and Eq. 51 is using the independence Yz.2.0 11 ILW.. o' X, Z. both of which are implied by the standard fairness model extended with the node S above. Based on the equality F[S,,,,, x=] TF(Y,,,, |x"] for arbitrary values v, x', x".
the claim follows from the appropriate choices: for instance, taking (x = æ1,2 No, x" = 2o and {i = To. x' x0. No). it follows that
x- C-DE (s) = F(S Weo
xo] - E(S Wio
Fo]
(53)
= -[YY,, Wao
ro] - EE BE Wxg
xo] = r- -E (v)-
(54)
Similarly, analogous choices of ar x', x" can be used to show the equality for indirect and spurious effects.
B Proofs related to Identification & Estimation
Prop. 4 Proof: Consider the x-specific (direct, indirect, spurious} effects of X on random variable
T given by:
x- DE .o (t xo) P(te, W [o)= P(tzo xoo)
(55)
m . (1 x00) P((o,, W xo)) P((a xrp)
(56)
SS SE (1) P((.) (n)) P((,, (1))
(57)
We first demonstrate why each of the potential outcomes appearing is identifiable under the Standard
Fairness Model. Consider the potential outcome Tr,..... X .g. For proving its identifiability,
make use of the counterfactual graph [Shpitser and Pearl 2007 in Fig. 5b
We can expand
14

---

P(T =t tt ro) as follows:
P(T = t, W2o = zo)
(Counterfactual Un-nesting)
(58)
Σ P(Tz1,u =t, 2.20 zo)
(Law of Total Probability)
(59)
Σ P(T22,u =t =t| 2, P(WW 2( z0)
(T Z,
(60)
2,W0
Σ P(Tz,,u =t 2, 21) P(W w P( mo)
(Consistency)
(61)
2,10
Σ P(T =
=t 2, 2, u. P(W w P(z ro)
W XX
(62)
3,W
Σ P(T = 2 w. 2 P(W = w 2, P( mo)
I 2)
(63)
2,30
Σ P(T = 2, 2, P(W = w P(z zo) (Consistency)
(64)
2,
Σ
P(t 2,
2, 2. mo).
(65)
2,10
Using a similar but simplified argument, we also obtain that:
P(T., = xo) = Σ P(t p(; (20),
(66)
and since P(T, = 1 x)) = P(L w)), all the terms in Eqs. 55 57 can be computed based on
observational data.
We next move onto the estimation part of the proof. We again focus on the quantity P(T.,.W., Weo = t
To). Suppose that we have:
, 2, E, [[ 21, w) Vz,
(67)
P( 1 w,,2) -4 P(ro w. 2) Vz, w
(68)
P(w,2) 2) - P(w, 2) Vu,
(69)
P( P, P(ro)
(70)
where f, P(00 (. are estimators, and P(w, 2), P(zzo) are the empirical distributions:
1 n
P(w, 2)
Σ
1(Z, 2, W; w)
(71)
n
i=1
1 Σ 1(X, zn).
P(xo)
(72)
n
Now, note that we can re-write the identification expression from Eq. 65 as:
P(xo 2, w)
Σ
P(t z, w. x,, w)
(73)
P(xo)
2,10
By coupling of quantities in Eqs. 67 70 to joint probability space and application of the
continuous mapping theorem we obtain that
[() w)
P(x0 w,
P.
Σ P(t z, w. 2,) w)
P(ro 2, w)
(74)
P(xo)
P(xo)
2,00
2,20
15

---

(a)
Census 2018 Decomposition
(b) DE Sample Influences
Sample Intt ance
Effect Size
0.000
0.085
0.000
CHIDE
DIHE
Causal Enrect
Sample Ranking
Component
Outcome
Margin Complement M
Component
Ouicome
Margin Complement M
Figure 6: Causal decomposition from Cor.
and sample influence on the Census 2018 dataset.
Finally, note that we have
P(ro 2, P(xo)
1(2. = 2, W; =w) P(:o 2,
Σ f(1, w) P(2,, 2,30
(,,,
Σ 2,30
P(xo)
i=1
(75)
P(wo 24, ws)
f(x1, 24, Wi)
(76)
P(xg)
The remaining identification expressions are derived based on the same technique, completing the proof of the proposition.
Prop. 5
Proof: Consider the direct effect sample influence of the i-th sample given by
SI-DE(i) = (x1, we, 2.) f(zo, w4, 2,) P(xo Wa, 24)
(77)
P(xo)
By assumption, we know that
f(x, 2, w) - B[T') 3i, z, w| Vx,
(78)
P( w, -, P(ro 2, Vz,w
(79)
P(w, 2) P, P(w, 2) Vw,=
(80)
P(xo) - P(zo)
(81)
Therefore, by a coupling of the above quantities a joint probability space and an application the continuous mapping theorem, we have that
SI-DE(1) P. E(T Fi, Ws, 2x] - E(T zo, Wi, zi]
P(xo Wi, 2;)
(82)
P(xo)
Note that
P(xo wiyzs)
P( Wiy 21) P(w;, 2;) P(ra)
P(w;, zi (0) P(w;, 2x)
(83)
P(xp)
completing the proof of the proposition.
C Census 2018 Example
In this appendix, we analyze the Census 2018 dataset, demonstrating an application in the context of labor and salary decisions:
16

---

Example 4 (Salary Increase on the Census 2018 Dataset [Plečko and Bareinboim 2024)). The United States Census of 2018 collected broad information about the US Government employees, including demographic information 7. (71 for age, 7.2 for race, Ze for nationality), gender X (ro female, In male), marital and family status M, education information I,, and work-related information R. The US Government wishes to use this data prospectively to decide whether a new employee should receive a bonus starting package upon signing the employment contract. To determine which employees should receive such bonus, a Government department decides to predict which of the employees should earn a salary above the median (which is $50,00 350,000/year). They construct a machine learning prediction score S that predicts above-median earnings (Y), and the department decides to allocate the bonus to all employees who are predicted to be above-median earners with a probability greater than 50% (i.e., Y = (S 2 0.5)).
A team of investigators within the Government is in charge of assessing what the impacts of this Al system are on the gender pay gap. They collect the required data and perform the decomposition from Cor. 3 shown in Fig. ((a). The decomposition indicates strong direct effects z-DE (y) and x- D-DE (m). which imply that men are more likely to receive a starting bonus than women. The indirect effects z- (y), x-IEz, (m) are both negative, with the latter effect of the margin complement not being significant. These effects, however, also mean that men are more likely to receive bonus due mediating variables. Finally. for the spurious effects, the effects are not significantly different from O. Based on these findings, the team decided to return the predictions to the original department, with the requirement that the direct effect of gender on the margin complement, z-DEzo.21 (m), must be reduced to 0, to avoid any possibility of bias amplification. D
D Extending Thm. & Cor.
In this appendix, we discuss two important extensions of the results presented in the main paper. First, we note that Cor. provides a decomposition of the TV measure for the optimal 0/1 predictor. In general, data analysts may be interested in analyzing suboptimal predictors using the same set of tools, which is discussed first (Appendix D.I. Second, we note that the results of the paper consider thresholded predictors Y = 1(S 2 t). In practice, it may be desirable to analyze predictors that use group-specific thresholds, c.g., those of the from
Ý = 11(($2 1SS tz)
(84)
where lg is group-specific threshold that may differ for X = and X = x1 groups. This is the second extension we consider, and it is discussed in Appendix D.2
D.1 Suboptimal Predictors
We now consider how Cor. couud be applied to analyze suboptimal predictors. Cor. 3 was based on Thm. 2 which showed that the causal effects of X on Y were equal to the causal effects of X on the optimal prediction score S. In practice, if a suboptimal prediction score Š is used, the symmetry shown in Thm. 2 can no longer be used. Therefore, if predictor Y is based on a suboptimal score S, the result of Cor. cannot be applied directly.
Still, in this case, variation of Cor. 3or.3cam can be used, stated in the following theorem:
Theorem 6 (Decomposing Suboptimal Predictors). Let Y be any thresholded predictor based on a prediction score S, and let M be its margin complement. Under the Standard Fairness Model in Fig. 2 the TV measure of the predictor Y can be decomposed into contributions from the true outcome Y. the suboptimality of S, and the margin complement M:
TV 7V.0. (0) x- D-D2z..21 (yl xo) + (x-DDFzo.a1 (8 mo) x--DFao.an (s (o0) + x: D-DExg.22 (m xt) (85)
(x-IEz, 20 xo) + (x-1E 20 (3 xo) - -x-1Ea, xp (s xo)) + x x-1E2, (m. za)) (86) Sx-SEz, (v) + (E.SE,,,.0)) (8) - x x-SEE (5)) + x x-SSE, (m)).
(87)
Recall that Cor. 3 was a way decomposition along each causal pathway, into: (i) the contribution arising from the true outcome Y: (ii) contribution from the thresholding, i.e., the margin complement M. In Thm. 6 the decomposition is a tiree-way one. First, as before, there is a contribution of the true outcome Y along the causal pathway in question. Second, there is contribution of suboptimality
17

---

Z X
X
S
W
W
(a) SFM extended with predictor
(b) Counterfactual graph the SFM.
Figure 7: Graphs used understand predictors with group-specific thresholds.
of S compared to S, which is captured by comparing the effects of X S and S along the causal pathway. Finally, there is also the contribution of thresholding. along the margin complement M of S. Therefore, when considering suboptimal predictor, three is an additional term in the decomposition (for each causal pathway) that measures the suboptimality of the prediction score S.
D.2 Predictors with Group-specific Thresholds
In the main text and Appendix D.I considered classifiers of the form
Y 1(52 N t),
(88)
where S is an arbitrary prediction score. In practice, when fairness considerations are taken into account, the decision-maker may use group specific threshold, i.e., a classifier of the form:
1 2 if if = xo
(89)
(1 if x1
In Eq. 89 different thresholds txo, tu,, are used for groups To, 21, respectively. The usage of group- specific thresholds is also widespread in the fair ML literature, for instance Kamiran instance|Kamiran al.) [012 uses post-procsssing approach with group- spep-spe thresholds to construct a predictor that satisfies demographic parity. Similarly, (Hardt al. 2016) uses post-processing approach with group- specific thresholds for achieving equality of odds. In Fig. 7al we present the causal diagram that corresponds single threshold setting, in Eq. 88 We note in this case that all of the effects from X Y (direct, indirect, spurious), are mediated by the prediction score S. When considering group-specifie thresholds, represented graphically in Fig. 7b] clearly the predictor Ý needs to also take X input, on top of $. Therefore, allowing for group-specific thresholds results in additional direct effect X - Y. Motivated by this definition, we introduce the following notion:
Definition (Group-specific Threshold Direct Effect). Let Ý be a thresholded predictor with group- specific thresholds, based on a prediction score S. The group-specific threshold direct effect for unit U = u is defined as:
GSD so.a
(ī) Yor.WeD (w) Y0,S2,,Woo (u)
(90)
= =1(5 (u) 2 (mi) - 11 1( Weo (u) 2 lzo)-
(91)
The x-specific group-specific direct effect is defined as:
D-DEGGTT 10.
(ū x:) = EluuDDGGTT GST (j) X = 2².
(92)
The unit-level group-specific threshold direct effect captures the effect of a change along the pathway X + Y. measures a specific unit U = u is classified differently when considering different thresholds tz,, txy. The z-specific version of the effect simply averages the unit-level effect across all units compatible with X = x. Armed with the above definition, in the following result, we provide new decomposition that can explicitly handle analysis of predictors with group-specific thresholds:
- X = 1
Theorem 7 (Causal Decomposition of Predictor with Group-specific Threshold). Let Ý be thresh- olded predictor with group-specific thresholds as in Eq. 89 based prediction score S. Let M be
18

---

the margin complement of S with respect to the threshold Exo of the X = zo group. The TV measure of Y can be decomposed as:
TV 20 (ÿ) D--E
GST 20,51 (y (D) + z-DE 20 (3 2o) + x x- z- (m 20))
(93)
(x-lEz,,ag 20 ($ 20) x- ao (m zo))
(94)
(x-Skiz (8) + z=SEr,, (m))
(95)
As one can see, the decomposition in Thm. is similar to that appearing in Thm. I However, there is additional term x-DE 1r. (3/ 2o) appearing, which is the consequence of considering group-specific thresholds. The above theorem provides the first result establishing that, causally, post-processing methods with group-specific thresholds change only the direct effect of X on Y.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims are corroborated both theoretically and empirically.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes].
Justification: the main limitation is the availability of the appropriate causal diagram, and the implied assumptions (discussed Sec. I.1
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
20

---

Justification: Appendix A A and provide detailed proofs. Assumptions are stated clearly in each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: the experiments described in detail in Sec. 5 & Appendix
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes].
Justification: An anonymized code repository is included (see the beginning of technical appendices). The source code is also uploaded as supplementary material. All the data used is publicly available. We note that access to the MIMIC-IV dataset requires approval from the data owners.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
guuies//ooeeuubisiiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctaps.cc/publlce hwwpbl/icgggideeeeeeeeeeeeeeeeee guwdes/ http://ooeeuuuaissooooooooo..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: See Sec. 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes].
Justification: Yes, uncertainty is quantified using bootstrap repetitions (see Sec. 5)
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification: Described in detail at the beginning of technical appendices.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http::////aaapppooooooooo////eee
Answer: [Yes].
Justification: The paper introduces a method for serutinizing/analyying bias in prediction and fecision-making
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We discuss the need for regulatory oversight based on our results (see Sec. 6). We not see potential for negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: The paper poses no risk in terms of data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: All data owners are cited (Sec. 5 Appendix C and licenses of use were respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: The anonymized code repository contains README file that provides the documentation (instructions) for reproducing all the experiments.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
No crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "kFNmGcAHpg", "reviewer": "Reviewer_1zLn", "summary": "The paper investigates how thresholding the score of a predictive model as a decision rule influences the fairness of the final decisions. In particular, the authors consider binary decisions or predictions of a true binary outcome in presence of a sensitive binary attribute.  In that context, the authors show that the fairness measure of the total variation depends on the direct, indirect and spurious effects due to the true outcome and due the margin complement; a measure introduced by the authors expressing the difference between the predicted score and the final prediction after thresholding.  Under this new measure, the authors also introduce the notions of weak and strong business necessity (BN); under weak BN, disparities due to the true outcome are considered fair, while disparities due to the thresholding rule are not tolerated—zero marginal complement; under weak BN disparities due to both the outcome and the thresholding rule are tolerated.", "questions": "The causal decomposition of the TV measure are on the optimal 0/1 predictor. How this decomposition would change for a sub-optimal (e.g. near optimal) predictor?", "limitations": "The authors very briefly refer to the limitations due to the assumptions on the standard fairness model. It would be useful perhaps to elaborate concretely on some of these assumptions; e.g., on how the notion of the margin complement  could be extended assuming that the protected attribute takes values in a discrete set or a continuous range.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is well organized, has a clear structure and seem appropriately placed into contemporary literature.\n\nThe technical contributions appear strong, well motivated and clearly presented.  The examples do help the reader to understand the intuition and follow the technical results.  \n\nThe problem of potential bias amplification due to applying a thresholding rule on predicted scores appears quite interesting and seems directly relevant and applicable in high stakes domains as the authors demonstrate through the real world data examples.", "weaknesses": "The abstract appears somewhat convoluted and confusing providing low level technical details, especially in lines 12-21. It might have been more helpful to explain the contribution at a higher level, while focusing more on the intuition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Authors: We thank the reviewer for the time and effort in reviewing our paper. We are quite glad that the reviewer appreciated the ideas appearing in the paper, and considered the technical contributions strong, well motivated, and clearly presented. Below we address the main questions/concerns.\n\n(W1: Abstract) Thank you for this suggestion. Your point is indeed valid, it may be quite difficult to understand some of the technical detail that is mentioned in the abstract. This is now updated, and we provide a more high-level explanation of the developments.\n\n(Q1: Suboptimal predictors) This is a great question, and we thank the reviewer for raising this. In fact, this leads to an important result, which we add to the appendix (see also (P1) of the general response).\n\nFirstly, we note that Theorem 1 holds for any thresholded predictor $\\tilde Y$ based on a score $\\tilde S$. In other words, Theorem 1 is not true just for the optimal 0/1 predictor. \nCrucially, however, Theorem 2 no longer holds for a suboptimal predictor. We can clearly see that if $\\tilde S$ is suboptimal, $E[\\tilde S] = E[Y]$ is not expected to hold, which means that the causal effects of $X$ on $\\tilde S$ need not equal those of $X$ on $Y$.\n\nHaving said that, the suboptimality of $\\tilde S$ can be remedied in a very nice way. In Corollary 3, we have a two-way decomposition along each causal path with contributions: (i) coming from the true $Y$; (ii) coming from the margin complement $M$. When instead of the optimal $S$, we are thresholding a suboptimal $\\tilde S$, this results in three-way decomposition along each path with contributions: (i) coming from the original $Y$, (ii) contribution of the suboptimality (difference along the causal path resulting from using $\\tilde S$ instead of $S$), and (iii) contribution of $M$. We invite the reviewer to check this interesting result in (P1) of the main review response. Thanks once again for pointing us in the right direction!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tn6cZr0PFD", "reviewer": "Reviewer_bCLY", "summary": "The paper studies how much thresholding a predictor affects the disparity in the decisions according to sensitive attributes and formalizes new notions of business necessity based on the causal graph of features, outcome and prediction function.", "questions": "1. What is the key technical contribution in Theorem 1 and 2, other than the novelty of the definitions (which allows you to write TV in a new, \"human interpretable\" way)?\n2. What is the connection between your definitions of business need and counterfactual fairness (e.g. Kusner at al 2017)? \n3. Please explain the corrected version of Definition 3 and what is the intended use case for each of the three definitions?\n4. What is the potential negative societal impact of using the weak and strong business need definitions in a regulatory context?", "limitations": "Incompletely addressed impact of weak and strong business need definitions.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Understanding the amplification of bias along the machine learning and decision making pipeline is an important topic.\n\n- There are some very interesting ideas that could have benefited from better execution in the paper. Having some technical definition of business necessity makes claims of BN testable in the real world, and could have policy impact.\n\n- A relatively comprehensive attempt to study the topic. Results include decomposition, identification, with proofs, as well as examples/experiments. With some significant improvements, it will be a nice contribution.", "weaknesses": "- Technical writing needs improvement and some key definitions are clearly wrong.\n(1) Looking at Definition 3, weak BN implies strong BN according to this definition. Why is the condition for weak BN stronger than for strong BN? I was also unable to make sense of the rest of this definition, e.g. part 1.\n(2) some quantities/notation are not properly defined. E.g. in definition 3, what are s, x'', m, x', y, and where are they defined? There should be no reasonable doubt about what these letters refer to. Same comment holds for Theorem 1 (what is m, s?), Theorem 2 and corollary 2.\n(3) Please use the notation :=, or similar, to denote definitional equivalence (rather than an equality _claim_).\n\n- Theorem 1 and 2 appear to be basic, as far as I can tell from the definitions and the proof. I'm not sure why the decomposition, which follows from definition, needs to be presented as a main theorem. The main novelty appears to be the definition. If this is the case, you might consider making this clear to the reader instead of presenting something as more complex that it needs to be. If this is not the case, is there a way you can make the technical contribution clearer in the proof and writing?\n\n- Example 1 is not a reasonable example. In this case (where you ONLY have gender on which to make a definition, and there is practically no signal from it), there would be no reason to use a predictor at all.\n\n-Related work: The discussion of closely related work is sparse. The connection with the causal fairness literature is relegated to a few sentences in the introduction, and there is no meaningful discussion of the contributions of prior work. Please elaborate on the connection between your definitions of business need and causal fairness? \n\nLack of citations and discussion of prior work. The paper states \"Most of the literature in fair machine learning\n4 focuses on defining and achieving fairness criteria in the context of prediction,\n5 while not explicitly focusing on how these predictions may be used later on in\n6 the pipeline.\"\n\nThis is an inaccurate representation of the literature. Even one of the cited works Chouldechoulva (2016) has an entire section on how scores impact decisions. Minimizing prior work on fairness in decision making is at best unhelpful to the reader, and at worst, misinforms.\n\nFor prior work on fairness in decision making (indeed, how predictions are used later in the pipeline), also see the following and the cited works within:\n\nLiu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2018, July). Delayed impact of fair machine learning. In International Conference on Machine Learning (pp. 3150-3158). PMLR.\n\nChouldechova, Alexandra, and Aaron Roth. \"The frontiers of fairness in machine learning.\" arXiv preprint arXiv:1810.08810 (2018).\n\nDwork, Cynthia, Christina Ilvento, and Meena Jagadeesan. \"Individual fairness in pipelines.\" arXiv preprint arXiv:2004.05167 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Authors: We thank the reviewer for the detailed review. We would like to draw the attention of the reviewer to some misunderstandings. The tone of the review seems much harsher than what we are used to in a venue such as NeurIPS and also how we perform our own reviews, always very respectfully. For instance, calling a definition “clearly wrong”, whereas the definition is correct, calling a theorem basic, saying that an example is not reasonable, comes across as harsh and discouraging. Even if there is a suspected mistake, it may be better to point it out politely, since the authors (including us) are always trying their best when writing. Thus, we hope that the reviewer can reconsider the assessment of the paper based on our detailed response below, especially because the reviewer seems to appreciate some genuine novelty in the paper, and we believe the response below clarifies the existing misunderstandings.\n\n(W1: Technical writing) Definition 3 is correct and “Weak business necessity” is indeed a special case of “Strong business necessity”. The concept of BN allows the predictor designer more flexibility in designing the predictor (fewer constraints). Therefore, strong BN (or stronger/higher flexibility) is indeed a more broad concept than weak BN (or weak/lower flexibility). More concretely, a predictor designer given the freedom to choose any arbitrary value for the $x-CE(m)$ effect, could choose to set this value to $0$ (and thus satisfy the weak BN requirement). However, under strong BN, they do not have to do so necessarily. We hope this clarifies a core confusion. \n\nFurthermore, we would like to clarify some of the notation. Indeed, in Definition 3, we now explicitly state that $x, x’$ are two distinct values of $X$, whereas $x^{’’}$ is an arbitrary value of $X$. We thank the reviewer for pointing this shortcoming out, which is now corrected.\nRegarding $y, m, s$ notation: here, we are following the common notation in the graphical approach to causality. The small letters indicate the random variable which is integrated out, in line with the previous works on causal fairness and the causality textbook (Pearl, 2000).\n\n(W2: Theorems are “basic”) We do not see the mentioned theorems as basic, and view this as a harsh assessment. While the definitions are important, it is apriori not clear that these definitions interact in such a nice & non-parametric way with the decomposition of the TV measure (Theorem 1). Furthermore, Theorem 2 requires replacing the true outcome $Y$ with $S$ in the causal diagram. This, even conceptually, is a non-trivial step that has not been done before, and requires specific causal assumptions (such as the SFM), and may not always hold. We invite the reviewer to check the counterfactual graph in Figure 5(b), and then re-assess whether the claim is trivial. Thus, we believe the results are non-trivial. Furthermore, we also do not think we made any attempt to present the results as more complex than they are. The statements and proofs are stated as clearly and concisely as possible.\n\nTo draw a parallel with some well-known results, think about the Pearl’s decomposition of the total effect (TE) into natural direct and indirect effects (NDE, NIE). While this decomposition is a consequence of the definitions of NDE, NIE, it is by no means trivial, and required almost a decade to appear after the first notions of direct and indirect effects were considered.\n\n(W3: Example 1 is not reasonable) Example 1 is used to illustrate the core concept of the paper in the simplest possible, two-variable case. We now explicitly state this in the introduction. \nA data scientist who has access to just these two variables (and perhaps does not even know the meaning of $X$), may simply implement a predictor without checking the predictive value, right? Once again, we are not trying to argue that the example reflects real-world practice (now stated explicitly before the example), but rather illustrate the concepts in the simplest form. Please see the Experiments section for more realistic, practical examples.\n\n(W4: Related work) Thanks for the suggested references. We have now expanded the part on related work, and reference the works you have shared, referring to “notable exceptions investigating the decision-making aspects of fairness”. \n\nHowever, we do believe it is true that most of the literature looks at prediction, which is also noted in the  second reference that you shared. To be clear, our goal is certainly not to diminish any of the contributions in this area, but rather to argue that more works are needed in this direction. We hope that the addition of the suggested references, and the better placement of the work in a broader context helps address this concern. Please let us know. \n\n---\n\n(Q1: Key technical contribution) Please see (W2) for a detailed response.\n\n(Q2: Connection with counterfactual fairness) This is a great question! It has been shown in the causal fairness literature that counterfactual fairness is simply a notion that constrains the _total (causal) effect)_ of $X$ on $Y$. When doing so, counterfactual fairness does not distinguish between, for instance, direct and indirect effects. As a consequence, counterfactual fairness inherently does not have the flexibility to model business necessity requirements, since it considers a single pathway (total causal effect). We hope this answers the question. \n\n(Q3: Definition 3) Please check the response (W1), the definition is correct. Regarding the intended use cases, please see details in global review response (P3).\n\n(Q4: Societal impact) This is another great question. In terms of negative consequences, one may argue that imposing weak BN or no BN constraints may harm the utility of the classifier, which may cause harm in specific applications. However, this represents a trade-off between utility and fairness, which is acknowledged in the literature. We now added an explicit discussion on this in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TlcJtGN4Hj", "reviewer": "Reviewer_ZV8n", "summary": "This paper studies the impact of downstream thresholding operations on continuous (and possibly fair) prediction scores. The paper argues that inappropriate thresholding can amplify or ameliorate the disparity in predictive performance across groups defined by the protected attribute. Using a causal framework, the paper provides a methodology to separate the disparities in the original predictive score from the disparities introduced due to thresholding.", "questions": "The main questions I have are related to the above points.\n1. Does the paper primarily focus on $t=1/2$ or are the results applicable for other threshold values as well?\n2. Do the issues associated with margin complement persist even when after using post-processing approaches to achieve outcome fairness (like those introduced in Hardt et al. 2016)?\n3. What’s meant by a causal pathway being “unconstrained” when defining strong BN?", "limitations": "Some limitations are acknowledged but the paper could do a better job of expanding on the limitations related to the full knowledge of structural causal models.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. One of the motivations of the paper is to understand the downstream utility of fair predictions. In that regard, I appreciate the more practical focus on investigating the barriers faced by popular fairness during real-world implementations. \n2. The disentanglement of different kinds of business necessity requirements seems interesting by itself and allows for potentially meaningful connections between legal and empirical notions of fairness (although I do have some questions on it that are noted later).", "weaknesses": "1. The writing and presentation are often hard to follow and various parts of the analysis seem under-explained, which makes evaluating the paper’s claims difficult. For example, Definition 1 defines $M$ using a generic $t$-value, whereas the proof of Theorem 1 in the appendix, if I understand correctly, directly uses $t=1/2$. If the theorem is based on a specific $t$ value, then that should be made clear in the theorem and the surrounding text in the main body.\\\n  Similarly, when discussing “strong business necessity”, there’s very little explanation of what it means for a causal pathway to be “unconstrained”. With a lack of a clear explanation of this concept, I am not totally sure about the difference between weak and strong BN. \n\n2. Related to thresholding, the paper never goes into the details of how a certain threshold is chosen. Again, seems like Theorem 1’s proof and Example 1 use $t=1/2$, but that is not the only possible choice of threshold that can be used in practice. In fact, if all the claims in the paper are based on the explicit assumption that $t=1/2$, then I don’t see the claims being generalizable to other threshold classifiers and also not applicable to various kinds of classifiers used in practice.  \n\n3. Additionally, there is a wide post-processing fairness literature that essentially proposes methods to choose group-specific thresholds so that the biases from training data and prediction scores do not propagate to the final binary predictions (e.g., Hardt et al. 2016 and related papers). I would strongly recommend an expanded discussion on this related work, especially comparing the proposed approach to other prior fairness works on choosing appropriate thresholds to guarantee outcome fairness. \n\n4. Minor points and typos.\\\n    a. Potential missing word in Line 73 around “…ramifications more”.\\\n    b. In Example 1, seems like the outcome should be 0 and 1, and not $y_0$ and $y_1$.\\\n    c. Line 138, function $pa$ is not defined. I am assuming it means “parent variable” but needs to be properly introduced and described.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Authors: We thank the reviewer for the time spent on our submission. We would like to state clearly that the results of the paper are not at all constrained to the $t = 1/2$ setting and work for any value of $t$. We therefore hope the reviewer can reconsider the contributions in light of this.\n\n(W1: threshold $t$ value) Thank you for giving us the opportunity to clarify this point. The proof of Theorem 1 was stated for $t=1/2$, but $t=1/2$ can be replaced with an arbitrary threshold $t$, which is now done in the manuscript. In other words, the method can be used for classifiers with arbitrary thresholds. Here, the reviewer is correctly noting that the notion of margin complement changes. We are no longer computing the margin complement with respect to 1/2, but with respect to an arbitrary threshold t. However, the proof remains the same for the arbitrary threshold, and the interpretation of the decomposition remains very similar.  \n\n(W2: Limited generality) Considering (W1) above, we note that the concern raised by the reviewer is not a limitation of the proposed method and the method can be used in general. Furthermore, we also note that our goal is not to suggest how to choose a specific threshold, but to provide a diagnostic tool for an arbitrary choice of the threshold.\nPlease also note that in the experimental section, in the MIMIC-IV example, we are using a threshold different from 1/2. In particular, we are using a threshold that is the median of the predicted probabilities, which is about 0.15 in the data. Still, the analysis yields important insights even in this case, highlighting the generality of the method in practice.\n\nWe hope that this addresses the key concern of the reviewer.\n\n(W3: post-processing methods) Thanks for asking this, this is a really good question. First, we wish to note that (Hardt et. al., 2016) are attempting to provide a post-processing method to satisfy equality of odds (i.e., constructing fair predictions). In our work, we are trying to offer a diagnostic tool, that quantifies the impact of thresholding on the causal fairness measures. In other words, we are solving a slightly different problem (which would be called bias detection/quantification). \n\nHaving said that, the reviewer raises an excellent point. Our analysis is based on classifiers with a single threshold, whereas (Hardt et. al., 2016) and many other methods use two thresholds, one for each group.  In the paper, we now added an appendix that handles the following question: is it possible to adapt the diagnostic tool in this paper to a thresholded classifier in which the threshold $t$ depends on $X$, i.e., $t_x$?\n\nThe answer to this is yes! Please refer to point (P2) of the main response for an updated decomposition of the TV measure in case of group-specific thresholds. In this new decomposition, we see that group-specific thresholds change only the direct effect of $X$ on the predictor while the indirect and spurious effects remain invariant. There is a new explicit term that quantifies the direct effect of using a group-specific threshold (see (P2) of main response for the exact expression). We also remark that this gives the first result establishing that, causally, post-processing methods with group-specific thresholds change only the direct effect of $X$ on the predictor, and gives an explicit expression for this direct effect. We think this makes the paper substantially stronger, and thank the reviewer for pointing us in the right direction! Our method can now handle the post-processing approach in (Hardt et. al, 2016) and many other similar methods in the literature.  \n\n(W4: typos) Thanks a lot for catching this. We have now fixed all of them!\n\n(Q1: t=1/2 or not?) The paper can handle an arbitrary threshold $t$, please see responses (W1), (W2).\n\n(Q2: post-processing relationship) Please see (W3) for an answer. \n\n(Q3: Unconstrained in Strong BN) Thanks for asking this, we have now clarified it in the text accordingly. By unconstrained we mean that the causal effect $x\\\\text{-CE}(m)$ can take an arbitrary value $c$, instead of being constrained to $0$ like in the weak BN setting. We hope this clarification in the writing helps address the question. Please let us know!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful reviews. We would like to mention three exciting updates to the paper that come as a result of some great questions from the reviewers. We believe these updates substantially improve the scope of the tools described in the paper:

(P1) (Reviewer 1zLn, Q1) We now introduce a new causal decomposition, which is a variation of Theorem 1 and Corollary 3. This decomposition handles threshold predictors that are based on arbitrary prediction scores $\tilde S$ (and not just the optimal predictor). The new decomposition is as follows:

$$
\\begin{align}
    \\text{TV}_{x_0, x_1}(\\tilde y) &=  {x\\text{-DE}\_{x_0, x_1}(y\\mid x_0)} + 
 (x\\text{-DE}\_{x_0, x_1}(\\tilde s\\mid x_0) - x\\text{-DE}\_{x_0, x_1}(s\\mid x_0)) + {x\\text{-DE}\_{x_0, x_1}(\\tilde m\\mid x_0)}
         \\\\ &\\quad
        - \\big( {x\\text{-IE}\_{x_1, x_0}(y\\mid x_0)} + (x\\text{-IE}\_{x_1, x_0}(\\tilde s \\mid x_0) - x\\text{-IE}\_{x_1, x_0}(s\\mid x_0)) + {x\\text{-IE}\_{x_1, x_0}(\\tilde m\\mid x_0)} \\big)\\\\
        &\\quad- \\big( {x\\text{-SE}\_{x_1, x_0}(y)} + ({x\\text{-SE}\_{x_1, x_0}(s)} - {x\\text{-SE}\_{x_1, x_0}(\\tilde s)}) + {x\\text{-SE}\_{x_1, x_0}(m)} \\big).
\\end{align}
$$

As the expression indicates, there is an explicit term measuring how much the suboptimality of the prediction score $\tilde S$ contributes to the overall difference along the pathway. This is quantified by comparing the difference of the effect of $X$ on $\tilde S$ (the predictor being analyzed) vs. the effect of $X$ on $S$ (the optimal predictor).

(P2) (Reviewer ZV8n, W3) The reviewer raised the question of handling post-processing methods that use separate thresholds for different groups. As it turns out, our tools can be adapted to this setting as well. Suppose that $\tilde Y$ is a predictor using group-specific thresholds. In particular, a new TV decomposition can be written as follows: 

$$
\\begin{align}
        \\text{TV}\_{x_0, x_1}(\\tilde y) &= x\\text{-DE}^{\\mathrm{GST}}\_{x_0, x_1}(\\tilde y \\mid x_0) +  {x\\text{-DE}\_{x_0, x_1}(\\tilde s\\mid x_0)} + {x\\text{-DE}\_{x_0, x_1}(\\tilde m\\mid x_0)}
        \\\\ &\\quad
        - \\big( {x\\text{-IE}\_{x_1, x_0}(\\tilde s\\mid x_0)} + {x\\text{-IE}\_{x_1, x_0}(\\tilde m\\mid x_0)} \\big)\\\\
        &\\quad- \\big( {x\\text{-SE}\_{x_1, x_0}(\\tilde s)} + {x\\text{-SE}\_{x_1, x_0}(\\tilde m)} \\big).
 \\end{align}
$$


In this new decomposition, we see that group-specific thresholds change only the direct effect of $X$ on $\tilde Y$ while the indirect and spurious effects remain the same. The term $x\text{-DE}^{\mathrm{GST}}_{x_0, x_1}(\tilde y \mid x_0)$ is given by

$$
\\begin{align}
E [ \\mathbb{1} ( \\tilde S_{x_1, W_{x_0}} (u) \\geq t_{x_1} ) - \\mathbb{1} ( \tilde S_{x_1, W_{x_0}} (u) \\geq t_{x_0} ) \\mid X = x_0 ]
\\end{align}
$$
and measures the direct effect of using a group-specific threshold. With this new decomposition, a whole class of methods in the fair ML literature can be analyzed, including reject-option classification (Kamiran & Calders, 2012) and post-processing for equality of odds (Hardt et. al., 2016), among many other methods.

(P3) (Reviewer bCLY, Q3 & Bounded Strong BN) The reviewer asked about different possible use cases for the notions of business necessity in Definition 3. Regarding the intended use cases, we provide the following explanations:

(i) No BN is used for pathways that are considered discriminatory, and when we wish to have no causal effect transmitted along the pathway.

(ii) Weak BN is used for cases where a pathway is important for the utility of the predictor designer, while there is still a need to avoid any kind of bias amplification compared to the current world.

(iii) Strong BN is intended for cases where the utility of the decision-maker along the causal pathway is so important that we are willing to accept even amplification of disparities as a result of improving this utility.

We remark that something in between Weak and Strong BN can be proposed, such as Bounded-Strong BN, where the contribution of the margin complement cannot be arbitrary but has to be bounded by some value $\alpha$. Therefore, we introduce the concept of $\alpha$-bounded Strong BN into the manuscript, which is discussed after Definition 3. We thank the reviewer for raising this question!

Meta-review: The reviewers agreed that the paper studies an important problem of how thresholding prediction scores would influence the fairness of final decisions, and that the results would be of broad interest to the community. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (poster)