Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making

Authors: drago  plecko
elias  bareinboim
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 14:59:24

Abstract: As society increasingly relies on AI-based tools for decision-making in socially sensitive domains, investigating fairness and equity of such automated systems has become a critical field of inquiry. Most of the literature in fair machine learning focuses on defining and achieving fairness criteria in the context of prediction, while not explicitly focusing on how these predictions may be used later on in the pipeline. For instance, if commonly used criteria, such as independence or sufficiency, are satisfied for a prediction score $S$ used for binary classification, they need not be satisfied after an application of a simple thresholding operation on $S$ (as commonly used in practice). 
In this paper, we take an important step to address this issue in numerous statistical and causal notions of fairness. We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation.
We then demonstrate that the marginal difference in the optimal 0/1 predictor $\widehat Y$ between groups, written $P(\hat y \mid x_1) - P(\hat y \mid x_0)$, can be causally decomposed into the influences of $X$ on the $L_2$-optimal prediction score $S$ and the influences of $X$ on the margin complement $M$, along different causal pathways (direct, indirect, spurious). We then show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$. This yields a new decomposition of the disparity in the predictor $\widehat Y$ that allows us to disentangle causal differences inherited from the true outcome $Y$ that exists in the real world vs. those coming from the optimization procedure itself. This observation highlights the need for more regulatory oversight due to the potential for bias amplification, and to address this issue we introduce new notions of weak and strong business necessity, together with an algorithm for assessing whether these notions are satisfied. We apply our method to three real-world datasets and derive new insights on bias amplification in prediction and decision-making.

Main paper: 
 introduction: 
 Automated systems based on machine learning and artificial intelligence are increasingly used for decision-making in a variety of real-world settings. These applications include hiring decisions, university admissions, law enforcement, credit lending and loan approvals, health care interventions, and many other high-stakes scenarios in which the automated system may significantly affect the well-being of individuals [Khandani et al., 2010 , Mahoney and Mohen, 2007 , Brennan et al., 2009] . In this context, society is increasingly concerned about the implications and consequences of using automated systems, compared to the currently implemented decision processes. Prior works highlight the potential of automated systems to perpetuate or even amplify inequities between demographic groups, with a range of examples from decision support systems for (among others) sentencing Angwin et al. [2016] , face-detection Buolamwini and Gebru [2018] , online advertising Sweeney [2013] , Datta et al. [2015] , and authentication Sanburn [2015] . Notably, issues of unfairness and discrimination are also pervasive in settings in which decisions are made by humans. Some wellstudied examples include the gender pay gap, supported by a decades-long literature [Blau and Kahn, 1992, 2017] , or the racial bias in criminal sentencing [Sweeney and Haney, 1992, Pager, 2003] , just to cite a few. Therefore, AI systems designed to make decisions may often be trained with data that contains various historical biases and past discriminatory decisions against certain protected groups, constituting a large part of the underlying problem. In this work, we specifically focus on investigating when automated systems may potentially lead to an even more discriminatory process, possibly amplifying already existing differences between groups.Within this context, it is useful to distinguish between different tasks appearing in the growing literature on fair machine learning. One can distinguish three specific and different tasks, namely (1) bias detection and quantification for exisiting outcomes or decision policies; (2) construction of fair predictions of an outcome; (3) construction of fair decision-making policies that are intended to be implemented in the real-world. Interestingly, a large portion of the literature in fair ML focuses on the second task of fair prediction, and what is often left unaddressed is how these predictions may be used later on in the pipeline, and what kind of consequences they may have. For instance, consider a prediction score S for a binary outcome Y that satisfies well-known fairness criteria, such as independence (demographic parity [Darlington, 1971] ) or sufficiency (calibration [Chouldechova, 2017] ). After a simple thresholding operation, commonly applied in settings with a binary outcome, the resulting predictor is no longer guaranteed to satisfy independence or sufficiency, and the previously provided fairness guarantees may be entirely lost. The same behavior can be observed for numerous other measures. These difficulties do not apply only to statistical measures of fairness. Recently, a growing literature has explored causal approaches to fair machine learning [Kusner et al., 2017 , Kilbertus et al., 2017 , Nabi and Shpitser, 2018 , Zhang and Bareinboim, 2018b ,a, Wu et al., 2019 , Chiappa, 2019 , Plečko and Meinshausen, 2020 , Plečko and Bareinboim, 2024] , which have two major benefits. First, they allow for human-understandable and interpretable definitions and metrics of fairness, which are tied to the causal mechanisms transmitting the change between groups. Secondly, they offer a language that is aligned with the legal notions of discrimination, such as the disparate impact doctrine. In particular, causal approaches allow for considerations of business necessity -which aim to ellucidate which covariates may be justifiably used by decision-makers even if their usage implies a disparity between groups. However, causal approaches to fairness also suffer from the above-discussed issuesnamely, a guarantee of absence of a causal influence from the protected attribute X onto a predictor S need not hold true after the predictor is thresholded [Plečko and Bareinboim, 2024] . Therefore, within the causal approach, there is also a major need for a better understanding of how probabilistic predictions are translated into binary predictions or decisions.In this work, we take an important step in the direction of addressing this issue. We work in a setting with a binary label Y , and the goal is to provide a binary prediction Y or a binary decision D. Our approach is particularly suitable for settings in which the utility of the decision is monotonic with respect to the conditional probability of Y being positive, written P (Y | covariates), but the developed tools also have ramifications for more general utilities. Examples that fall under our scope are numerous; for instance, the utility of admitting a student to the university (D) is often monotonic in the probability that the student successfully graduates (Y ). In the context of criminal justice, decisions of detention (D) are used to prevent recidivism, and the utility of the decision is monotonic in the probability that the individual recidivates (Y ). Finally, various preventive measures in healthcare (D, such as vaccination, screening tests, etc.) are considered to be best applied to individuals with the highest risk of developing a target disease or suffering a negative outcome (Y ). We now provide an illustrative two-variable example for one of the key insights of our paper: Example 1 (Disparities in Hiring). Consider a company deciding to hire employees using an automated system for the first time. From a previous hiring cycle when humans were in charge, the company has access to data on gender X (x 0 for female, x 1 for male) and the hiring outcome Y (1 for being hired, 0 otherwise). The true underlying mechanisms of the system are given by:X ← 1(U X < 0.5) (1) Y ← 1(U Y < p 0 ) if X = x 0 1(U Y < p 1 ) if X = x 1 (2) Male (x 1 ) Female (x 0 ) Before (Y ) After ( Y ) ThresholdingNot hired (y 0 ) Hired (y 1 )Figure 1 : Visualization of hiring disparities from Ex. 1.whereU X , U Y ∼ Unif[0, 1].In words, an applicant is female with a 50% probability, and the probability of being hired as a female x 0 is p 0 , whereas for males x 1 the probability is p 1 . The company finds the optimal prediction score S to be S(x) = p x . The optimal 0/1 predictor Y , which will also be the company's decision, is given by Y(x) = 1(S(x) ≥ 12 ), meaning that the company will threshold the predictions at 1 2 . Suppose that p 0 = 0.49, p 1 = 0.51, and consider the visualization in Fig. 1 . The probability p 0 = 0.49 means that 49 out of 100 females were hired, while 51 were not.After thresholding, Y (x 0 ) = 1(p 0 ≥ 1 2 ) = 0 for each female applicant, meaning that the thresholding operation maps the prediction of each individual to 0, even though 49/100 would have a positive outcome (Fig. 1 right). Similarly, for p 1 = 0.51, we have that 51/100 male applicants would be hired, resulting in a thresholded predictor Y (x 1 ) = 1 for all the applicants, even though 49/100 would not have a positive outcome (Fig. 1 left ). Therefore, the gender disparity in hiring after introducing the automated predictor Y is 100%, compared to a 2% disparity in the outcome Y before introducing Y .Formally, the disparity in the optimal 0/1 predictor P( y | x 1 ) -P ( y | x 0 ) = 1(p 1 ≥ 1 2 ) -1(p 0 ≥ 12 ) can be decomposed as:EQUATION)Term I measures the disparity coming from the true outcome Y (2% disparity), which is equal to the disparity in the prediction score S, written P (s | x 1 ) -P (s | x 0 ). Term II measures the contribution coming from thresholding the prediction score to obtain an optimal 0/1 prediction (98% disparity). □The above example illustrates a canonical point in a simple setting: a small disparity in the outcome Y , and consequently the prediction score S, may result in a large disparity in the optimal 0/1 predictor Y , a case we call bias amplification. Contrary to this, a large disparity in S may also result in a small disparity in Y , a case we call bias amelioration.In the remainder of the manuscript, our goal is to provide a decomposition of the disparity in a thresholded predictor Y into the disparity in true outcome Y and the disparity originating from optimization procedure, but along each causal pathway between the protected attribute X and the predictor Y . In particular, our contributions are the following:(1) We introduce the notion of margin complement (Def. 1), and provide a path-specific decomposition of the disparity in the 0/1 predictor Y into its contributions from the optimal score predictor S and the margin complement M (Thm. 1),(2) We prove that under suitable assumptions, the causal decomposition of the optimal prediction score S is equivalent with the causal decomposition of the true outcome Y (Thm. 2). This allows us to obtain a new decomposition of the disparity in Y into contributions from Y and the margin complement M (Cor. 3),(3) Motivated by the above decompositions, we introduce a new concept of weak and strong business necessity (Def. 3), highlighting a new need for regulatory instructions in the context of automated systems. We provide an algorithm for assessing fairness under considerations of weak and strong business necessity (Alg. 1), (4) We provide identification, estimation, and sample influence results for all of the quantities relevant to the above framework (Props. 4, 5) . We evaluate our approach on three real-world examples (Ex. 2-3) and provide new empirical insights into bias amplification.Our work is related to the previous literature on causal fairness and the causal decompositions appearing in this literature [Zhang and Bareinboim, 2018b, Plečko and Bareinboim, 2024] . It is also related to previous literature on studying business necessity requirements through a causal lens [Kilbertus et al., 2017, Plecko and Bareinboim, 2024b] . However, our approach offers an entirely new causal decomposition into contributions from the true outcome Y and the margin complement M . More broadly, our work is also related to the literature on fair decision-making, which analyzes how prediction scores impact the fairness of decisions [Chouldechova, 2017 , Dwork et al., 2020 , Chouldechova and Roth, 2018] , or how disparities evolve over time [Liu et al., 2018] . Recent results also show that focusing purely on prediction, and ignoring decision-making aspects, may lead to inequitable outcomes and cause harm to marginalized groups [Plečko and Bareinboim, 2024 , Nilforoshan et al., 2022 , Plecko and Bareinboim, 2024a] , highlighting a need to expand focus from narrow statistical definitions of fair predictions to a more comprehensive understanding of equity in algorithmic decisions. Still, many questions remain open in the context of fair decision-making, and more future works are required in this area. Finally, we mention that our work is also related to the literature on auditing and assessing fairness of decisions made by humans [Pierson et al., 2021 , Kleinberg et al., 2018] , and understanding how AI systems may help humans overcome their biases [Imai et al., 2023] . We use the language of structural causal models (SCMs) [Pearl, 2000] . An SCM is a tuple M := ⟨V, U, F, P (u)⟩ , where V , U are sets of endogenous (observable) and exogenous (latent) variables, respectively, F is a set of functions f Vi , one for eachEQUATIONu) is a strictly positive probability measure over U . Each SCM M is associated to a causal diagram G [Bareinboim et al., 2022] over the node set V where V i → V j if V i is an argument of f Vj , and V i V j if the corresponding U Vi , U Vj are not independent. An instantiation of the exogenous variables U = u is called a unit. By Y x (u) we denote the potential response of Y when setting X = x for the unit u, which is the solution for Y (u) to the set of equations obtained by evaluating the unit u in the submodel M x , in which all equations in F associated with X are replaced by X = x. Throughout the paper, we assume a specific cluster causal diagram G SFM known as the standard fairness model (SFM) [Plečko and Bareinboim, 2024] over endogenous variables {X, Z, W, Y, Y } shown in Fig. 2 (see also [Anand et al., 2023] ). The SFM consists of the following: protected attribute, labeled X (e.g., gender, race, religion), assumed to be binary; the set of confounding variables Z, which are not causally influenced by the attribute X (e.g., demographic information, zip code); the set of mediator variables W that are possibly causally influenced by the attribute (e.g., educational level or other job-related information); the outcome variable Y (e.g., GPA, salary); the predictor of the outcome Y (e.g., predicted GPA, predicted salary). The SFM also encodes the lack-of-confounding assumptions typically used in the causal inference literature. The availability of the SFM and the implied assumptions are a possible limitation of the paper, while we note that partial identification techniques for bounding effects can be used for relaxing them [Zhang et al., 2022] . 
margin complements: 
 We begin by introducing a quantity that plays a key role in the results of this paper. Definition 1 (Margin Complement). Let U = u be a unit, and let S denote a prediction score for a binary outcome Y . Let the subscript C denote a counterfactual clause, so that Z C denotes a potential response. The margin complement M of the score S for the unit U = u and threshold t is defined as:M (u) = 1(S(u) ≥ t) -S(u). (4) A potential response of M , labeled M C , is given by M C (u) = 1(S C (u) ≥ t) -S C (u).In words, the margin complement for a unit U = u represents the difference in the score after thresholding vs. the score that would happen naturally. Example 1 (Disparities in Hiring continued). Consider the hiring example with the SCM in Eqs. 1-2 with p 0 = 0.49, p 1 = 0.51. The unit (U X , U Y ) = (0, 0) corresponds to a male applicant (X(u) = 1) who was hired (Y (u) = 1). We have S(u) = p 1 = 0.51, and M (u) = 1(S(u) ≥ 0.5) -S(u) = 1 -0.51 = 0.49. For this u, the margin complement indicates that the predicted outcome Y (u) = 1(S(u) ≥ 0.5) is 49% greater than the predicted probability S(u). The same computation can be done for a female X(u ′ ) = 0, in which case M (u ′ ) = 1(p x0 ≥ 0.5) -p x0 = -0.49, meaning that the predicted outcome Y (u ′ ) is 49% smaller than the predicted probability S(u ′ ). □Given a prediction score S and a threshold t, the margin complement M tells us in which direction the thresholded version 1(S(u) ≥ t) moves compared to the score S(u). A positive margin complement indicates that a thresholded predictor is larger than the probability prediction, and a negative margin complement the opposite. A similar reasoning holds for the potential responses of the margin complement M C : we are interested in what the margin complement would have been for an individual U = u under possibly different, counterfactual conditions described by C. As we demonstrate shortly, margin complements (and their potential responses) play a major role in explaining how inequities are generated between groups at the time of decision-making. In this section, our key aim is to analyze the optimal 0/1 predictor Y and provide a decomposition of its total variation measure (TV, for short), defined as TV x0,x1 ( y) = P ( y | x 1 ) -P ( y | x 0 ). When working with the causal diagram in Fig. 2 , we can notice that the TV measure comprises of three types of variations coming from X: the direct effect X → Y , the mediated effect X → W → Y , and the confounded effect X Z → Y . Our goal is to construct a decomposition of the TV measure that allows us to distinguish how much of each of the causal effects is due to a difference in the prediction score S, and how much due to margin complements M . To investigate this, we first introduce the known definitions of direct, indirect, and spurious effects from the causal fairness literature: Definition 2 (x-specific Causal Measures [Zhang and Bareinboim, 2018b, Plečko and Bareinboim, 2024] ). The x-specific {direct, indirect, spurious} effects of X on Y are defined as:EQUATIONArmed with these definitions, we can prove the following result:Theorem 1 (Causal Decomposition of Optimal 0/1 Predictor). Let Y be the optimal predictor with respect to the 0/1-loss based on covariates X, Z, W . Let S denote the optimal predictor with respect to the L 2 loss. The total variation (TV, for short) measure of the predictor Y , written as P (ŷ | x 1 ) -P (ŷ | x 0 ), can be decomposed into direct, indirect, and spurious effects of X on the score S and the margin complement M as follows:EQUATION)The above theorem is the first key result of this paper. The disparity between groups with respect to the optimal 0/1-loss predictor, measured by TV x0,x1 (ŷ) can be decomposed into direct, indirect, and spurious contributions coming from (i) the optimal L 2 -loss predictor S (e.g., term x-DE x0,x1 (s | x 0 )), and (ii) the margin complement M (e.g., term x-DE x0,x1 (m | x 0 )). This provides a unique capability since for each causal pathway (direct, indirect, spurious) the contribution coming from the probability prediction S can be disentangled from the contribution coming from the optimization procedure itself (i.e., the rounding of the predictor). The former, as we will see shortly, is simply a representation of the bias already existing in the true outcome Y , whereas the latter represents a newly introduced type of bias that is the result of using an automated system. The contribution of the margin complement may act to both ameliorate or amplify an existing disparity, a point we investigate later on.Example 1 (Disparities in Hiring extended). Consider the hiring example from Ex. 1 extended with a mediator W indicating whether the applicant has a PhD degree (W = 1) or not (W = 0). Suppose the augmented SCM is given by:EQUATIONandP (U X , U W , U Y ) is such that U X , U W , U Y are independent Unif[0, 1] random variables.The coefficients satisfy the constraints α, λ > 0, 0.4 < β < 0.9-α 1+λ . The optimal probability predictor is S(x, w) = 0.1 + αx + βw. The TV measure of the optimal 0/1 predictor be computed as TV x0,x1 (ŷ) = α β + λ. Using Thm. 1 we can decompose it as:EQUATION)Along the direct effect, there is a difference of α between the groups in the probability predictor S(x, w) that is amplified by α β -α by the margin complement, and since β < 1 we always have bias amplification. Along the indirect effect, there is an initial difference of λβ, amplified by an additional λ -λβ, again amplifying bias. The decomposition provides a breakdown of the disparity of the optimal 0/1 predictor into its constitutive parts. □We next consider a crucial point mentioned above. The disparity observed in the optimal L 2 score S is simply a representation of the disparity existing in the real world, under suitable causal assumptions: Theorem 2 (Relationship of L 2 -score S and Outcome Y Decompositions). Let M be an SCM compatible with the Standard Fairness Model, and let S be the optimal L 2 prediction score. Then, the causal decompositions of the score S and the true outcome Y are symmetric, meaning thatEQUATION)Based on this result, we can see that the causal influences from X on the true outcome Y are equivalent to the causal influences from X on the optimal prediction score S. Combining this insight with Thm. 1 leads to the following result: Corollary 3 (Causal Decomposition of Optimal 0/1 Predictor). Under the Standard Fairness Model, the TV measure of the optimal 0/1-loss predictor Y can be decomposed into contributions from the true outcome Y and the margin complement M :EQUATION)This corollary provides an important insight compared to Thm. 1: the disparity in the optimal predictor Y can be decomposed into the contribution inherited from the true outcome Y and the contribution that arises from the optimization procedure (rounding of the predictor). In Appendix D, we describe two important extensions of the main results: (i) to the setting of suboptimal predictors; (ii) to the setting of predictors using group-specific thresholds, which is often the case with post-processing methods used in fair machine learning [Kamiran et al., 2012 , Hardt et al., 2016] . 
weak and strong business necessity: 
 In law, questions of fairness and discrimination can be interpreted based on the disparate treatment (DT) and disparate impact (DI) doctrines of Title VII of the Civil Rights Act of 1964 [Act, 1964] . The DT doctrine, when interpreted causally, can be seen as disallowing a direct type of effect from X onto the outcome. The DI doctrine, however, is more broad, and may prohibit any from of discrimination (be it direct, indirect, or spurious) that results in a large disparity between groups. The core of this if CE ∈ Strong-BN then Assert that x-CE(s) = x-CE( y), otherwise FAIL else if CE ∈ Weak-BN then Assert that x-CE(s) = x-CE( y) ∧ x-CE(m) = 0, otherwise FAIL else Assert that x-CE( s) = x-CE( m) = 0, otherwise FAIL if not FAIL return SUCCESSdoctrine is the notion of business necessity (BN), which allows certain variables correlated with the protected attribute to be used for prediction due to their relevance to the business itself [Els, 1993] EQUATION)A pathway is said to satisfy weak business necessity if:EQUATION)A pathway is said to satisfy strong business necessity if:EQUATION)The above definition distinguishes between three important cases, and sheds light on a new aspect of the concept of business necessity. According to the definition, there are three versions of BN considerations:(1) A causal pathway is not in the BN set, and is considered discriminatory. In this case, both the contribution of the prediction score S and the margin complement M need to be equal to 0 (i.e., no discrimination is allowed along the pathway),(2) A causal pathway satisfies weak BN, and is not considered discriminatory. In this case, the effect of X on the prediction score S needs to equal the effect of X onto the true outcome Y along the same pathway [Plecko and Bareinboim, 2024b] . However, the contribution of the margin complement M along the pathway needs to equal 0.(3) A causal pathway satisfies strong BN, and is not considered discriminatory. Similarly as for weak necessity, the effect of X on S needs to equal the effect of X on Y , but in this case, the contribution of the margin complement M is unconstrained.The distinction between cases (2) and (3) opens the door for new regulatory requirements and specifications. In particular, whenever a causal effect is considered non-discriminatory, the attribute X needs to affect S to the extent to which it does in the real world. However, the system designer also needs to decide whether a difference existing in the predicted probabilities S is allowed to be amplified (or ameliorated) by means of rounding. The latter point distinguishes between weak and strong BN, and should be a consideration of any system designer issuing binary decisions. In Alg. 1, we propose a formal approach for evaluating considerations of weak and strong BN for any input of a predictor Y and a prediction score S. 
identification, estimation, and sample influence: 
 In Thm. 1 and Cor. 3 the observed disparity in the TV measure of the optimal 0/1 predictor is decomposed into its constitutive components. The quantities appearing in the decomposition are counterfactuals, and thus the question of identification of these quantities needs to be addressed. In other words, we need to understand whether these quantities can be uniquely computed based on the available data and the causal assumptions. The following is a positive answer: Proposition 4 (Identification and Estimation of Causal Measures). Let M be an SCM compatible with the Standard Fairness Model, and let P (V ) be its observational distribution. The x-specific direct, indirect, and spurious effects of X on the outcome Y , predictor Y , prediction score S, and the margin complement M are identifiable (uniquely computable) from P (V ) and the SFM. Denote by f (x, z, w) estimator of E[T | x, z, w], and by P (x | v ′ ) the estimator of the probabilityP (x | v ′ )for different choices of v ′ . For T ∈ {Y, Y , S, M }, the effects can be estimated as:EQUATION)The proof of the proposition, together with the identification expressions for the different quantities can be found in Appendix B. We next define the sample influences for the different estimators: Definition 4 (Sample Influence). The sample influence of the i-th sample on the estimator x-CE est of the causal effect CE is given by corresponding term in the summations in Eqs. 24-26. For instance, the i-th sample influence on x-DE est x0,x1 (t | x 0 ) is given by (and analogously for IE, SE terms):EQUATION)The sample influences tell us how each of the samples contributes to the overall estimator of the quantity. These sample-level contributions may be interesting to investigate from the point of view of the system designer, including identifying any subpopulations that are discriminated against. For direct sample influences, the following proposition can be proved: Proposition 5 (Direct Effect Sample Influence). The SI-DE(i) in Eq. 27 is an estimator ofEQUATIONwhereE[T x1,Wx 0 -T x0 | x 0 , z i , w i ] is the (x 0 , z i , w i )-specific direct effect of X on T .Prop. 5 demonstrates an important point -namely that the sample influences along the direct path are not just quantities of statistical interest, but also causally meaningful quantities. In particular, the influence of the i-th sample is proportional to the direct effect of the x 0 → x 1 transition for the group of units u compatible with the event x 0 , z i , w i . The influence is further proportional to P (w i , z i | x 0 )/P (w i , z i ) that measures how much more likely the covariates z i , w i of the i-th sample are in the X = x 0 group (for which the discrimination is quantified) vs. the overall population. Therefore, practitioners also have a causal reason for investigating these sample influences. 
experiments: 
 We analyze the MIMIC-IV (Ex. 2), COMPAS (Ex. 3), and Census (Ex. 4, Appendix C) datasets.Example 2 (Acute Care Triage on MIMIC-IV Dataset [Johnson et al., 2023] ). Clinicians in the Beth Israel Deaconess Medical Center in Boston, Massachusetts treat critically ill patients admitted to Example 3 (Recidivism Prevention on the COMPAS Dataset [Larson et al., 2016] ). Courts in Broward County, Florida use machine learning algorithms, developed by a private company called Northpointe, to predict whether individuals released on parole are at high risk of re-offending within 2 years (Y ). The algorithm is based on the demographic information Z (Z 1 for gender, Z 2 for age), race X (x 0 denoting White, x 1 Non-White), juvenile offense counts J, prior offense count P , and degree of charge D. The courts wish to know which individuals are highly likely to recidivate, such that their probability of recidivism is above 50%. The company constructs a prediction score ŜNP and the court subsequently uses this for deciding whether to detain individuals at high risk of re-offending.After a court hearing in which it was decided that the indirect and spurious effects fall under business necessity requirements, a team from ProPublica wishes to investigate the implications of using the automated predictions ŜNP . They obtain the relevant data and apply Alg. 1, with the results shown in Fig. 4 . The team first compares the decompositions of the true outcome Y and the predictor ŜNP (Fig. 4 (a)). For the spurious effect, they find that x-SE x1,x0 (y) is not statistically different from x-SE x1,x0 (ŝ N P ), in line with BN requirements. For the indirect effects, they find that the indirect effect is lower for the predictor ŜNP compared to the true outcome Y , indicating no concerning violations. However, for the direct effect, while the x-DE x0,x1 (y) is not statistically different from 0, the predictor ŜNP has a significant direct effect of X, i.e., x-DE x0,x1 (ŝ N P ) ̸ = 0. This indicates a violation of the fairness requirements determined by the court. After comparing the decompositions of ŜNP and Y , the team moves onto understanding the contributions of the margin complements (Fig. 4b ). For each effect, there is a pronounced impact of the margin complements. For the direct effect (not under BN), the non-zero margin complement contribution x-DE x1,x0 (m) ̸ = 0 represents a violation of fairness requirements. For the indirect and spurious effects, the ProPublica team realizes the court did not specify anything about margin complement contributions -based on this, for the next court hearing they are preparing an argument showing that the effects x-IE x1,x0 (m) and x-SE x1,x0 (m) are significantly different from 0, thereby exacerbating the differences between groups. Finally, based on sample influences (Fig. 4c ), they realize that the direct effect is driven by a small minority of individuals, and they decide to investigate this further. □ 
conclusion: 
 In this paper, we developed tools for understanding the fairness impacts of transforming a continuous prediction score S into binary predictions Y or binary decisions D. In Thm. 1 and Cor. 3 we showed that the TV measure of the optimal 0/1 predictor decomposes into direct, indirect, and spurious contributions that are inherited from the true outcome Y in the real world, and also contributions from the margin complement M (Def. 1) arising from the automated optimization procedure. This observation motivated new notions of weak and strong business necessity (BN) -in the former case, differences inherited from the true outcome Y are allowed to be propagated into predictions or decisions, while any differences resulting from the optimization procedure are disallowed. In contrast, strong BN allows both of these differences and does not prohibit possible disparity amplification. In Alg. 1, we developed a formal procedure for assessing weak and strong BN. Finally, real-world examples demonstrated that the tools developed in this paper are of genuine importance in practice, since converting continuous predictions into binary decisions may often result in bias amplification in practice -highlighting the need for this type of analysis, and the importance of regulatory oversight. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] .Justification: The paper poses no risk in terms of data or models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] .Justification: All data owners are cited (Sec. 5, Appendix C) and licenses of use were respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The source code for reproducing all the experiments can be found in our Github code repository https://github.com/dplecko/mind-the-gap. The code is also included with the supplementary materials, in the folder source-code. All experiments were performed on a MacBook Pro, with the M3 Pro chip and 36 GB RAM on macOS 14.1 (Sonoma). All experiments can be run with less than 1 hour of compute on the above-described machine or equivalent.Thm. 1 Proof: Notice that by definition we haveEQUATION)Now, let E be any observed event, and let C be a counterfactual clause representing a possibly counterfactual intervention. Note that we can write:EQUATIONNow, we can expand the TV measure of Y as follows:EQUATIONcompleting the proof.Thm. 2 Proof: Let M be an SCM compatible with the SFM in Fig. 2 as per theorem assumption. Let the optimal L 2 prediction score S be given by S(x, z, w , z, w] . More precisely, we can let f S be the structural mechanism of S, taking X, Z, W as inputs. The structural mechanism of S is then given by: write:) = E[Y | xEQUATIONEQUATIONwhere Eq. 48 is using the independence S x,z,w ⊥ ⊥W x ′ | X, Z, and Eq. 51 is using the independence Y x,z,w ⊥ ⊥W x ′ | X, Z, both of which are implied by the standard fairness model extended with the node S above. Based on the equalityE[S x,W x ′ | x ′′ ] = E[Y x,W x ′ | x ′′ ]for arbitrary values x, x ′ , x ′′ , the claim follows from the appropriate choices: for instance, taking {x = x 1 , x ′ = x 0 , x ′′ = x 0 } and {x = x 0 , x ′ = x 0 , x ′′ = x 0 }, it follows thatx-DE x0,x1 (s) = E[S x1,Wx 0 | x 0 ] -E[S x0,Wx 0 | x 0 ] (53) = E[Y x1,Wx 0 | x 0 ] -E[Y x0,Wx 0 | x 0 ] = x-DE x0,x1 (y). (54)Similarly, analogous choices of x, x ′ , x ′′ can be used to show the equality for indirect and spurious effects.Prop. 4 Proof: Consider the x-specific {direct, indirect, spurious} effects of X on a random variable T given by:EQUATION)We first demonstrate why each of the potential outcomes appearing is identifiable under the Standard Fairness Model. Consider the potential outcome T x1,Wx 0 | X = x 0 . For proving its identifiability, we make use of the counterfactual graph [Shpitser and Pearl, 2007] in Fig. 5b . We can expand P (T x1,Wx 0 = t | x 0 ) as follows:EQUATION= z,wEQUATIONUsing a similar but simplified argument, we also obtain that:EQUATIONand since P (T x = t | x) = P (t | x), all the terms in Eqs. 55-57 can be computed based on observational data.We next move onto the estimation part of the proof. We again focus on the quantity P (T x1,Wx 0 = t | x 0 ). Suppose that we have:EQUATIONP (x 0 | w, z)EQUATIONP (w, z)EQUATIONEQUATIONwhere f, P (x 0 | w, z) are estimators, and P (w, z), P (x 0 ) are the empirical distributions:EQUATIONEQUATIONNow, note that we can re-write the identification expression from Eq. 65 as:z,wEQUATION)By a coupling of quantities in Eqs. 67-70 to a joint probability space and an application of the continuous mapping theorem we obtain that z,w f (x 1 , z, w) P (z, w) P (x 0 | z, w)EQUATION)Example 4 (Salary Increase on the Census 2018 Dataset [Plečko and Bareinboim, 2024] ). The United States Census of 2018 collected broad information about the US Government employees, including demographic information Z (Z 1 for age, Z 2 for race, Z 3 for nationality), gender X (x 0 female, x 1 male), marital and family status M , education information L, and work-related information R. The US Government wishes to use this data prospectively to decide whether a new employee should receive a bonus starting package upon signing the employment contract. To determine which employees should receive such a bonus, a Government department decides to predict which of the employees should earn a salary above the median (which is $50,000/year). They construct a machine learning prediction score S that predicts above-median earnings (Y ), and the department decides to allocate the bonus to all employees who are predicted to be above-median earners with a probability greater than 50% (i.e., Y = 1(S ≥ 0.5)).A team of investigators within the Government is in charge of assessing what the impacts of this AI system are on the gender pay gap. They collect the required data and perform the decomposition from Cor. 3, shown in Fig. 6 (a). The decomposition indicates strong direct effects x-DE x0,x1 (y) and x-DE x0,x1 (m), which imply that men are more likely to receive a starting bonus than women. The indirect effects x-IE x1,x0 (y), x-IE x1,x0 (m) are both negative, with the latter effect of the margin complement not being significant. These effects, however, also mean that men are more likely to receive a bonus due to mediating variables. Finally, for the spurious effects, the effects are not significantly different from 0. Based on these findings, the team decided to return the predictions to the original department, with the requirement that the direct effect of gender on the margin complement, x-DE x0,x1 (m), must be reduced to 0, to avoid any possibility of bias amplification. □In this appendix, we discuss two important extensions of the results presented in the main paper. First, we note that Cor. 3 provides a decomposition of the TV measure for the optimal 0/1 predictor.In general, data analysts may be interested in analyzing suboptimal predictors using the same set of tools, which is discussed first (Appendix D.1). Second, we note that the results of the paper consider thresholded predictors Y = 1(S ≥ t). In practice, it may be desirable to analyze predictors that use group-specific thresholds, e.g., those of the fromEQUATIONwhere t x is a group-specific threshold that may differ for X = x 0 and X = x 1 groups. This is the second extension we consider, and it is discussed in Appendix D.2.We now consider how Cor. 3 could be applied to analyze suboptimal predictors. Cor. 3 was based on Thm. 2, which showed that the causal effects of X on Y were equal to the causal effects of X on the optimal prediction score S. In practice, if a suboptimal prediction score S is used, the symmetry shown in Thm. 2 can no longer be used. Therefore, if a predictor Ỹ is based on a suboptimal score S, the result of Cor. 3 cannot be applied directly.Still, in this case, a variation of Cor. 3 can be used, stated in the following theorem:Theorem 6 (Decomposing Suboptimal Predictors). Let Ỹ be any thresholded predictor based on a prediction score S, and let M be its margin complement. Under the Standard Fairness Model in Fig. 2 , the TV measure of the predictor Ỹ can be decomposed into contributions from the true outcome Y , the suboptimality of S, and the margin complement M :EQUATIONRecall that Cor. 3 was a two-way decomposition along each causal pathway, into: (i) the contribution arising from the true outcome Y ; (ii) contribution from the thresholding, i.e., the margin complement M . In Thm. 6, the decomposition is a three-way one. First, as before, there is a contribution of the true outcome Y along the causal pathway in question. Second, there is a contribution of suboptimality of S compared to S, which is captured by comparing the effects of X on S and S along the causal pathway. Finally, there is also the contribution of thresholding, along the margin complement M of S.Therefore, when considering a suboptimal predictor, three is an additional term in the decomposition (for each causal pathway) that measures the suboptimality of the prediction score S.In the main text and Appendix D.1, we considered classifiers of the formEQUATION)where S is an arbitrary prediction score. In practice, when fairness considerations are taken into account, the decision-maker may use a group specific threshold, i.e., a classifier of the form:EQUATIONIn Eq. 89, different thresholds t x0 , t x1 are used for groups x 0 , x 1 , respectively. The usage of groupspecific thresholds is also widespread in the fair ML literature, for instance Kamiran et al. [2012] uses a post-processing approach with group-specific thresholds to construct a predictor that satisfies demographic parity. Similarly, [Hardt et al., 2016] uses a post-processing approach with groupspecific thresholds for achieving equality of odds. In Fig. 7a , we present the causal diagram that corresponds to a single threshold setting, as in Eq. 88. We note in this case that all of the effects from X to Ỹ (direct, indirect, spurious), are mediated by the prediction score S. When considering group-specific thresholds, represented graphically in Fig. 7b , clearly the predictor Ỹ needs to also take X as an input, on top of S. Therefore, allowing for group-specific thresholds results in an additional direct effect X → Ỹ . Motivated by this definition, we introduce the following notion:Definition 5 (Group-specific Threshold Direct Effect). Let Ỹ be a thresholded predictor with groupspecific thresholds, based on a prediction score S. The group-specific threshold direct effect for a unit U = u is defined as:EQUATIONThe x-specific group-specific direct effect is defined as:EQUATIONThe unit-level group-specific threshold direct effect captures the effect of a X = x 0 → X = x 1 change along the pathway X → Ỹ . It measures if a specific unit U = u is classified differently when considering different thresholds t x1 , t x0 . The x-specific version of the effect simply averages the unit-level effect across all units compatible with X = x. Armed with the above definition, in the following result, we provide a new decomposition that can explicitly handle analysis of predictors with group-specific thresholds:Theorem 7 (Causal Decomposition of Predictor with Group-specific Threshold). Let Ỹ be a thresholded predictor with group-specific thresholds as in Eq. 89, based on a prediction score S. Let M be the margin complement of S with respect to the threshold t x0 of the X = x 0 group. The TV measure of Ỹ can be decomposed as:EQUATIONAs one can see, the decomposition in Thm. 7 is similar to that appearing in Thm. 1. However, there is an additional term x-DE GST x0,x1 (ỹ | x 0 ) appearing, which is the consequence of considering group-specific thresholds. The above theorem provides the first result establishing that, causally, post-processing methods with group-specific thresholds change only the direct effect of X on Ỹ .Answer: [Yes] . Justification: An anonymized code repository is included (see the beginning of technical appendices). The source code is also uploaded as supplementary material. All the data used is publicly available. We note that access to the MIMIC-IV dataset requires approval from the data owners. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The paper introduces a method for scrutinizing/analyzing bias in prediction and decision-making. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] . Justification: We discuss the need for regulatory oversight based on our results (see Sec. 3 and 6) . We do not see potential for negative societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The reviewers agreed that the paper studies an important problem of how thresholding prediction scores would influence the fairness of final decisions, and that the results would be of broad interest to the community. However, the reviewers also raised several concerns and questions in their initial reviews. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses, which helped in answering their key questions. The reviewers have an overall positive assessment of the paper, and there is a consensus for acceptance. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.

Reviews: review: paper_summary: paper_summary: The paper investigates how thresholding the score of a predictive model as a decision rule influences the fairness of the final decisions. In particular, the authors consider binary decisions or predictions of a true binary outcome in presence of a sensitive binary attribute.  In that context, the authors show that the fairness measure of the total variation depends on the direct, indirect and spurious effects due to the true outcome and due the margin complement; a measure introduced by the authors expressing the difference between the predicted score and the final prediction after thresholding.  Under this new measure, the authors also introduce the notions of weak and strong business necessity (BN); under weak BN, disparities due to the true outcome are considered fair, while disparities due to the thresholding rule are not tolerated—zero marginal complement; under weak BN disparities due to both the outcome and the thresholding rule are tolerated.
strength_weakness: strength_weakness: The abstract appears somewhat convoluted and confusing providing low level technical details, especially in lines 12-21. It might have been more helpful to explain the contribution at a higher level, while focusing more on the intuition.
questions: questions: The causal decomposition of the TV measure are on the optimal 0/1 predictor. How this decomposition would change for a sub-optimal (e.g. near optimal) predictor?
limitations: limitations: The authors very briefly refer to the limitations due to the assumptions on the standard fairness model. It would be useful perhaps to elaborate concretely on some of these assumptions; e.g., on how the notion of the margin complement  could be extended assuming that the protected attribute takes values in a discrete set or a continuous range.
paper_summary: paper_summary: The paper studies how much thresholding a predictor affects the disparity in the decisions according to sensitive attributes and formalizes new notions of business necessity based on the causal graph of features, outcome and prediction function.
strength_weakness: strength_weakness: - Technical writing needs improvement and some key definitions are clearly wrong.
(1) Looking at Definition 3, weak BN implies strong BN according to this definition. Why is the condition for weak BN stronger than for strong BN? I was also unable to make sense of the rest of this definition, e.g. part 1.
(2) some quantities/notation are not properly defined. E.g. in definition 3, what are s, x'', m, x', y, and where are they defined? There should be no reasonable doubt about what these letters refer to. Same comment holds for Theorem 1 (what is m, s?), Theorem 2 and corollary 2.
(3) Please use the notation :=, or similar, to denote definitional equivalence (rather than an equality _claim_).

- Theorem 1 and 2 appear to be basic, as far as I can tell from the definitions and the proof. I'm not sure why the decomposition, which follows from definition, needs to be presented as a main theorem. The main novelty appears to be the definition. If this is the case, you might consider making this clear to the reader instead of presenting something as more complex that it needs to be. If this is not the case, is there a way you can make the technical contribution clearer in the proof and writing?

- Example 1 is not a reasonable example. In this case (where you ONLY have gender on which to make a definition, and there is practically no signal from it), there would be no reason to use a predictor at all.

-Related work: The discussion of closely related work is sparse. The connection with the causal fairness literature is relegated to a few sentences in the introduction, and there is no meaningful discussion of the contributions of prior work. Please elaborate on the connection between your definitions of business need and causal fairness? 

Lack of citations and discussion of prior work. The paper states "Most of the literature in fair machine learning
4 focuses on defining and achieving fairness criteria in the context of prediction,
5 while not explicitly focusing on how these predictions may be used later on in
6 the pipeline."

This is an inaccurate representation of the literature. Even one of the cited works Chouldechoulva (2016) has an entire section on how scores impact decisions. Minimizing prior work on fairness in decision making is at best unhelpful to the reader, and at worst, misinforms.

For prior work on fairness in decision making (indeed, how predictions are used later in the pipeline), also see the following and the cited works within:

Liu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2018, July). Delayed impact of fair machine learning. In International Conference on Machine Learning (pp. 3150-3158). PMLR.

Chouldechova, Alexandra, and Aaron Roth. "The frontiers of fairness in machine learning." arXiv preprint arXiv:1810.08810 (2018).

Dwork, Cynthia, Christina Ilvento, and Meena Jagadeesan. "Individual fairness in pipelines." arXiv preprint arXiv:2004.05167 (2020).
questions: questions: 1. What is the key technical contribution in Theorem 1 and 2, other than the novelty of the definitions (which allows you to write TV in a new, "human interpretable" way)?
2. What is the connection between your definitions of business need and counterfactual fairness (e.g. Kusner at al 2017)? 
3. Please explain the corrected version of Definition 3 and what is the intended use case for each of the three definitions?
4. What is the potential negative societal impact of using the weak and strong business need definitions in a regulatory context?
limitations: limitations: Incompletely addressed impact of weak and strong business need definitions.
paper_summary: paper_summary: This paper studies the impact of downstream thresholding operations on continuous (and possibly fair) prediction scores. The paper argues that inappropriate thresholding can amplify or ameliorate the disparity in predictive performance across groups defined by the protected attribute. Using a causal framework, the paper provides a methodology to separate the disparities in the original predictive score from the disparities introduced due to thresholding.
strength_weakness: strength_weakness: 1. The writing and presentation are often hard to follow and various parts of the analysis seem under-explained, which makes evaluating the paper’s claims difficult. For example, Definition 1 defines $M$ using a generic $t$-value, whereas the proof of Theorem 1 in the appendix, if I understand correctly, directly uses $t=1/2$. If the theorem is based on a specific $t$ value, then that should be made clear in the theorem and the surrounding text in the main body.\
  Similarly, when discussing “strong business necessity”, there’s very little explanation of what it means for a causal pathway to be “unconstrained”. With a lack of a clear explanation of this concept, I am not totally sure about the difference between weak and strong BN. 

2. Related to thresholding, the paper never goes into the details of how a certain threshold is chosen. Again, seems like Theorem 1’s proof and Example 1 use $t=1/2$, but that is not the only possible choice of threshold that can be used in practice. In fact, if all the claims in the paper are based on the explicit assumption that $t=1/2$, then I don’t see the claims being generalizable to other threshold classifiers and also not applicable to various kinds of classifiers used in practice.  

3. Additionally, there is a wide post-processing fairness literature that essentially proposes methods to choose group-specific thresholds so that the biases from training data and prediction scores do not propagate to the final binary predictions (e.g., Hardt et al. 2016 and related papers). I would strongly recommend an expanded discussion on this related work, especially comparing the proposed approach to other prior fairness works on choosing appropriate thresholds to guarantee outcome fairness. 

4. Minor points and typos.\
    a. Potential missing word in Line 73 around “…ramifications more”.\
    b. In Example 1, seems like the outcome should be 0 and 1, and not $y_0$ and $y_1$.\
    c. Line 138, function $pa$ is not defined. I am assuming it means “parent variable” but needs to be properly introduced and described.
questions: questions: The main questions I have are related to the above points.
1. Does the paper primarily focus on $t=1/2$ or are the results applicable for other threshold values as well?
2. Do the issues associated with margin complement persist even when after using post-processing approaches to achieve outcome fairness (like those introduced in Hardt et al. 2016)?
3. What’s meant by a causal pathway being “unconstrained” when defining strong BN?
limitations: limitations: Some limitations are acknowledged but the paper could do a better job of expanding on the limitations related to the full knowledge of structural causal models.

Comments: title: Thank you for constructive feedback
comment: I would like to thank the authors for their reply that has sufficiently addressed my question. The authors should consider adding the discussion and results in (P1) in the revised version of their paper.
Dear Reviewer bCLY,
 
In the spirit of the constructive discussion period we usually have at NeurIPS, we would like to double-check if there are any issues that were not sufficiently addressed in our rebuttal. We would be happy to engage and elaborate further on any questions that you may have.

Thank you again for the time spent reading our work,

Authors of paper #3972
We wish to thank the reviewer for the opportunity to engage and provide important clarifications on the manuscript. We also appreciate you sharing constructive suggestions, which we took very seriously, and which helped to improve our manuscript and led us to a new result. 

Related to the discussion on previous literature, as we mentioned to Reviewer bCLY, we already revised the discussion of related literature in fair decision-making. We added the suggested references within this context, and expanded the discussion on related works accordingly.

We once again thank the reviewer for the constructive review process.
Thanks for the detailed response and clarifications! I am satisfied with the response on the ability to handle arbitrary thresholds and will increase my score.
I also appreciate the response on group-specific thresholds, but agree with Reviewer bCLY that the discussion on prior related work needs to be expanded and more thorough.