Title: Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making
Venue: NeurIPS 2024
Accepted: yes
Keywords: Causal Fairness

Fair Machine Learning

Causal Inference

Trustworthy AI
Abstract: As society increasingly relies on AI-based tools for decision-making in socially sensitive domains, investigating fairness and equity of such automated systems has become a critical field of inquiry. Most of the literature in fair machine learning focuses on defining and achieving fairness criteria in the context of prediction, while not explicitly focusing on how these predictions may be used later on in the pipeline. For instance, if commonly used criteria, such as independence or sufficiency, are satisfied for a prediction score $S$ used for binary classification, they need not be satisfied after an application of a simple thresholding operation on $S$ (as commonly used in practice). 
In this paper, we take an important step to address this issue in numerous statistical and causal notions of fairness. We introduce the notion of a margin complement, which measures how much a prediction score $S$ changes due to a thresholding operation.
We then demonstrate that the marginal difference in the optimal 0/1 predictor $\widehat Y$ between groups, written $P(\hat y \mid x_1) - P(\hat y \mid x_0)$, can be causally decomposed into the influences of $X$ on the $L_2$-optimal prediction score $S$ and the influences of $X$ on the margin complement $M$, along different causal pathways (direct, indirect, spurious). We then show that under suitable causal assumptions, the influences of $X$ on the prediction score $S$ are equal to the influences of $X$ on the true outcome $Y$. This yields a new decomposition of the disparity in the predictor $\widehat Y$ that allows us to disentangle causal differences inherited from the true outcome $Y$ that exists in the real world vs. those coming from the optimization procedure itself. This observation highlights the need for more regulatory oversight due to the potential for bias amplification, and to address this issue we introduce new notions of weak and strong business necessity, together with an algorithm for assessing whether these notions are satisfied. We apply our method to three real-world datasets and derive new insights on bias amplification in prediction and decision-making.

Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making

Abstract

As society increasingly relies on AI-based tools for decision-making in socially sensitive domains, investigating fairness and equity of such automated systems has become a critical field of inquiry. Most of the literature in fair machine learning focuses on defining and achieving fairness criteria in the context of prediction, while not explicitly focusing on how these predictions may be used later on in the pipeline. For instance, if commonly used criteria, such as independence or sufficiency, are satisfied for a prediction score S used for binary classification, they need not be satisfied after an application of a simple thresholding operation on S (as commonly used in practice). In this paper, we take an important step to address this issue in numerous statistical and causal notions of fairness. We introduce the notion of a margin complement, which measures how much a prediction score S changes due to a thresholding operation. We then demonstrate that the marginal difference in the optimal 0/1 predictor ̂ Y between groups, written P (ˆ y | x 1 ) -P (ˆ y | x 0 ) , can be causally decomposed into the influences of X on the L 2 -optimal prediction score S and the influences of X on the margin complement M , along different causal pathways (direct, indirect, spurious). We then show that under suitable causal assumptions, the influences of X on the prediction score S are equal to the influences of X on the true outcome Y . This yields a new decomposition of the disparity in the predictor ̂ Y that allows us to disentangle causal differences inherited from the true outcome Y that exists in the real world vs. those coming from the optimization procedure itself. This observation highlights the need for more regulatory oversight due to the potential for bias amplification, and to address this issue we introduce new notions of weak and strong business necessity, together with an algorithm for assessing whether these notions are satisfied. We apply our method to three real-world datasets and derive new insights on bias amplification in prediction and decision-making.

1 Introduction

Automated systems based on machine learning and artificial intelligence are increasingly used for decision-making in a variety of real-world settings. These applications include hiring decisions, university admissions, law enforcement, credit lending and loan approvals, health care interventions, and many other high-stakes scenarios in which the automated system may significantly affect the well-being of individuals [Khandani et al., 2010, Mahoney and Mohen, 2007, Brennan et al., 2009]. In this context, society is increasingly concerned about the implications and consequences of using automated systems, compared to the currently implemented decision processes. Prior works highlight the potential of automated systems to perpetuate or even amplify inequities between demographic groups, with a range of examples from decision support systems for (among others) sentencing

Angwin et al. [2016], face-detection Buolamwini and Gebru [2018], online advertising Sweeney [2013], Datta et al. [2015], and authentication Sanburn [2015]. Notably, issues of unfairness and discrimination are also pervasive in settings in which decisions are made by humans. Some wellstudied examples include the gender pay gap, supported by a decades-long literature [Blau and Kahn, 1992, 2017], or the racial bias in criminal sentencing [Sweeney and Haney, 1992, Pager, 2003], just to cite a few. Therefore, AI systems designed to make decisions may often be trained with data that contains various historical biases and past discriminatory decisions against certain protected groups, constituting a large part of the underlying problem. In this work, we specifically focus on investigating when automated systems may potentially lead to an even more discriminatory process, possibly amplifying already existing differences between groups.

Within this context, it is useful to distinguish between different tasks appearing in the growing literature on fair machine learning. One can distinguish three specific and different tasks, namely (1) bias detection and quantification for exisiting outcomes or decision policies; (2) construction of fair predictions of an outcome; (3) construction of fair decision-making policies that are intended to be implemented in the real-world. Interestingly, a large portion of the literature in fair ML focuses on the second task of fair prediction, and what is often left unaddressed is how these predictions may be used later on in the pipeline, and what kind of consequences they may have. For instance, consider a prediction score S for a binary outcome Y that satisfies well-known fairness criteria, such as independence (demographic parity [Darlington, 1971]) or sufficiency (calibration [Chouldechova, 2017]). After a simple thresholding operation, commonly applied in settings with a binary outcome, the resulting predictor is no longer guaranteed to satisfy independence or sufficiency, and the previously provided fairness guarantees may be entirely lost. The same behavior can be observed for numerous other measures.

These difficulties do not apply only to statistical measures of fairness. Recently, a growing literature has explored causal approaches to fair machine learning [Kusner et al., 2017, Kilbertus et al., 2017, Nabi and Shpitser, 2018, Zhang and Bareinboim, 2018b,a, Wu et al., 2019, Chiappa, 2019, Pleˇ cko and Meinshausen, 2020, Pleˇ cko and Bareinboim, 2024], which have two major benefits. First, they allow for human-understandable and interpretable definitions and metrics of fairness, which are tied to the causal mechanisms transmitting the change between groups. Secondly, they offer a language that is aligned with the legal notions of discrimination, such as the disparate impact doctrine. In particular, causal approaches allow for considerations of business necessity - which aim to ellucidate which covariates may be justifiably used by decision-makers even if their usage implies a disparity between groups. However, causal approaches to fairness also suffer from the above-discussed issues namely, a guarantee of absence of a causal influence from the protected attribute X onto a predictor S need not hold true after the predictor is thresholded [Pleˇ cko and Bareinboim, 2024]. Therefore, within the causal approach, there is also a major need for a better understanding of how probabilistic predictions are translated into binary predictions or decisions.

In this work, we take an important step in the direction of addressing this issue. We work in a setting with a binary label Y , and the goal is to provide a binary prediction ̂ Y or a binary decision D . Our approach is particularly suitable for settings in which the utility of the decision is monotonic with respect to the conditional probability of Y being positive, written P ( Y | covariates ) , but the developed tools also have ramifications for more general utilities. Examples that fall under our scope are numerous; for instance, the utility of admitting a student to the university ( D ) is often monotonic in the probability that the student successfully graduates ( Y ). In the context of criminal justice, decisions of detention ( D ) are used to prevent recidivism, and the utility of the decision is monotonic in the probability that the individual recidivates ( Y ). Finally, various preventive measures in healthcare ( D , such as vaccination, screening tests, etc.) are considered to be best applied to individuals with the highest risk of developing a target disease or suffering a negative outcome ( Y ). We now provide an illustrative two-variable example for one of the key insights of our paper:

Example 1 (Disparities in Hiring) . Consider a company deciding to hire employees using an automated system for the first time. From a previous hiring cycle when humans were in charge, the company has access to data on gender X ( x 0 for female, x 1 for male) and the hiring outcome Y (1 for being hired, 0 otherwise). The true underlying mechanisms of the system are given by:

Male (

x

1

)

Female (

x

0

)

Before (

Y

)

After (

̂

Y

)

Thresholding

Not hired (

y

0

)

Hired (

y

1

)

where U X , U Y ∼ Unif [0 , 1] . In words, an applicant is female with a 50% probability, and the probability of being hired as a female x 0 is p 0 , whereas for males x 1 the probability is p 1 . The company finds the optimal prediction score S to be S ( x ) = p x . The optimal 0/1 predictor ̂ Y , which will also be the company's decision, is given by ̂ Y ( x ) = ✶ ( S ( x ) ≥ 1 2 ) , meaning that the company will threshold the predictions at 1 2 . Suppose that p 0 = 0 . 49 , p 1 = 0 . 51 , and consider the visualization in Fig. 1. The probability p 0 = 0 . 49 means that 49 out of 100 females were hired, while 51 were not. After thresholding, ̂ Y ( x 0 ) = ✶ ( p 0 ≥ 1 2 ) = 0 for each female applicant, meaning that the thresholding operation maps the prediction of each individual to 0 , even though 49/100 would have a positive outcome (Fig. 1 right). Similarly, for p 1 = 0 . 51 , we have that 51/100 male applicants would be hired, resulting in a thresholded predictor ̂ Y ( x 1 ) = 1 for all the applicants, even though 49/100 would not have a positive outcome (Fig. 1 left). Therefore, the gender disparity in hiring after introducing the automated predictor ̂ Y is 100%, compared to a 2% disparity in the outcome Y before introducing ̂ Y . Formally, the disparity in the optimal 0/1 predictor P ( ̂ y | x 1 ) -P ( ̂ y | x 0 ) = ✶ ( p 1 ≥ 1 2 ) -✶ ( p 0 ≥ 1 2 ) can be decomposed as:

Term I measures the disparity coming from the true outcome Y (2% disparity), which is equal to the disparity in the prediction score S , written P ( s | x 1 ) -P ( s | x 0 ) . Term II measures the contribution coming from thresholding the prediction score to obtain an optimal 0/1 prediction (98% disparity). □

The above example illustrates a canonical point in a simple setting: a small disparity in the outcome Y , and consequently the prediction score S , may result in a large disparity in the optimal 0/1 predictor ̂ Y , a case we call bias amplification . Contrary to this, a large disparity in S may also result in a small disparity in ̂ Y , a case we call bias amelioration .

In the remainder of the manuscript, our goal is to provide a decomposition of the disparity in a thresholded predictor ̂ Y into the disparity in true outcome Y and the disparity originating from optimization procedure, but along each causal pathway between the protected attribute X and the predictor ̂ Y . In particular, our contributions are the following:

We introduce the notion of margin complement (Def. 1), and provide a path-specific decomposition of the disparity in the 0/1 predictor ̂ Y into its contributions from the optimal score predictor S and the margin complement M (Thm. 1),

Weprove that under suitable assumptions, the causal decomposition of the optimal prediction score S is equivalent with the causal decomposition of the true outcome Y (Thm. 2). This allows us to obtain a new decomposition of the disparity in ̂ Y into contributions from Y and the margin complement M (Cor. 3),

Motivated by the above decompositions, we introduce a new concept of weak and strong business necessity (Def. 3), highlighting a new need for regulatory instructions in the context of automated systems. We provide an algorithm for assessing fairness under considerations of weak and strong business necessity (Alg. 1),

We provide identification, estimation, and sample influence results for all of the quantities relevant to the above framework (Props. 4, 5). We evaluate our approach on three real-world examples (Ex. 2-3) and provide new empirical insights into bias amplification.

Our work is related to the previous literature on causal fairness and the causal decompositions appearing in this literature [Zhang and Bareinboim, 2018b, Pleˇ cko and Bareinboim, 2024]. It is also related to previous literature on studying business necessity requirements through a causal lens [Kilbertus et al., 2017, Plecko and Bareinboim, 2024b]. However, our approach offers an entirely new causal decomposition into contributions from the true outcome Y and the margin complement M . More broadly, our work is also related to the literature on fair decision-making, which analyzes how prediction scores impact the fairness of decisions [Chouldechova, 2017, Dwork et al., 2020, Chouldechova and Roth, 2018], or how disparities evolve over time [Liu et al., 2018]. Recent results also show that focusing purely on prediction, and ignoring decision-making aspects, may lead to inequitable outcomes and cause harm to marginalized groups [Pleˇ cko and Bareinboim, 2024, Nilforoshan et al., 2022, Plecko and Bareinboim, 2024a], highlighting a need to expand focus from narrow statistical definitions of fair predictions to a more comprehensive understanding of equity in algorithmic decisions. Still, many questions remain open in the context of fair decision-making, and more future works are required in this area. Finally, we mention that our work is also related to the literature on auditing and assessing fairness of decisions made by humans [Pierson et al., 2021, Kleinberg et al., 2018], and understanding how AI systems may help humans overcome their biases [Imai et al., 2023].

1.1 Preliminaries

We use the language of structural causal models (SCMs) [Pearl, 2000]. An SCM is a tuple M := ⟨ V, U, F , P ( u ) ⟩ , where V , U are sets of endogenous (observable) and exogenous (latent) variables, respectively, F is a set of functions f V i , one for each V i ∈ V , where V i ← f V i (pa( V i ) , U V i ) for some pa( V i ) ⊆ V and U V i ⊆ U . The set pa( V i ) is called the parent set of V i . P ( u ) is a strictly positive probability measure over U . Each

Z

X

W

Y

̂

Y

SCM M is associated to a causal diagram G [Bareinboim et al., 2022] over the node set V where V i → V j if V i is an argument of f V j , and V i glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] V j if the corresponding U V i , U V j are not independent. An instantiation of the exogenous variables U = u is called a unit . By Y x ( u ) we denote the potential response of Y when setting X = x for the unit u , which is the solution for Y ( u ) to the set of equations obtained by evaluating the unit u in the submodel M x , in which all equations in F associated with X are replaced by X = x . Throughout the paper, we assume a specific cluster causal diagram G SFM known as the standard fairness model (SFM) [Pleˇ cko and Bareinboim, 2024] over endogenous variables { X,Z,W,Y, ̂ Y } shown in Fig. 2 (see also [Anand et al., 2023]). The SFM consists of the following: protected attribute , labeled X (e.g., gender, race, religion), assumed to be binary; the set of confounding variables Z , which are not causally influenced by the attribute X (e.g., demographic information, zip code); the set of mediator variables W that are possibly causally influenced by the attribute (e.g., educational level or other job-related information); the outcome variable Y (e.g., GPA, salary); the predictor of the outcome ̂ Y (e.g., predicted GPA, predicted salary). The SFM also encodes the lack-of-confounding assumptions typically used in the causal inference literature. The availability of the SFM and the implied assumptions are a possible limitation of the paper, while we note that partial identification techniques for bounding effects can be used for relaxing them [Zhang et al., 2022].

2 Margin Complements

We begin by introducing a quantity that plays a key role in the results of this paper.

Definition 1 (Margin Complement) . Let U = u be a unit, and let S denote a prediction score for a binary outcome Y . Let the subscript C denote a counterfactual clause, so that Z C denotes a potential response. The margin complement M of the score S for the unit U = u and threshold t is defined as:

M ( u ) = ✶ ( S ( u ) ≥ t ) -S ( u ) . (4) A potential response of M , labeled M C , is given by M C ( u ) = ✶ ( S C ( u ) ≥ t ) -S C ( u ) .

In words, the margin complement for a unit U = u represents the difference in the score after thresholding vs. the score that would happen naturally.

Example 1 (Disparities in Hiring continued) . Consider the hiring example with the SCM in Eqs. 1-2 with p 0 = 0 . 49 , p 1 = 0 . 51 . The unit ( U X , U Y ) = (0 , 0) corresponds to a male applicant ( X ( u ) = 1 ) who was hired ( Y ( u ) = 1 ). We have S ( u ) = p 1 = 0 . 51 , and M ( u ) = ✶ ( S ( u ) ≥ 0 . 5) -S ( u ) = 1 -0 . 51 = 0 . 49 . For this u , the margin complement indicates that the predicted outcome ̂ Y ( u ) = ✶ ( S ( u ) ≥ 0 . 5) is 49% greater than the predicted probability S ( u ) . The same computation can be done for a female X ( u ′ ) = 0 , in which case M ( u ′ ) = ✶ ( p x 0 ≥ 0 . 5) -p x 0 = -0 . 49 , meaning that the predicted outcome ̂ Y ( u ′ ) is 49% smaller than the predicted probability S ( u ′ ) . □

Given a prediction score S and a threshold t , the margin complement M tells us in which direction the thresholded version ✶ ( S ( u ) ≥ t ) moves compared to the score S ( u ) . A positive margin complement indicates that a thresholded predictor is larger than the probability prediction, and a negative margin complement the opposite. A similar reasoning holds for the potential responses of the margin complement M C : we are interested in what the margin complement would have been for an individual U = u under possibly different, counterfactual conditions described by C . As we demonstrate shortly, margin complements (and their potential responses) play a major role in explaining how inequities are generated between groups at the time of decision-making. In this section, our key aim is to analyze the optimal 0/1 predictor ̂ Y and provide a decomposition of its total variation measure (TV, for short), defined as TV x 0 ,x 1 ( ̂ y ) = P ( ̂ y | x 1 ) -P ( ̂ y | x 0 ) . When working with the causal diagram in Fig. 2, we can notice that the TV measure comprises of three types of variations coming from X : the direct effect X → ̂ Y , the mediated effect X → W → ̂ Y , and the confounded effect X glyph[arrowaxisleft]glyph[axisshort]glyph[axisshort]glyph[arrowaxisright] Z → ̂ Y . Our goal is to construct a decomposition of the TV measure that allows us to distinguish how much of each of the causal effects is due to a difference in the prediction score S , and how much due to margin complements M . To investigate this, we first introduce the known definitions of direct, indirect, and spurious effects from the causal fairness literature:

Definition 2 ( x -specific Causal Measures [Zhang and Bareinboim, 2018b, Pleˇ cko and Bareinboim, 2024]) . The x -specific {direct, indirect, spurious} effects of X on Y are defined as:

Armed with these definitions, we can prove the following result:

Theorem 1 (Causal Decomposition of Optimal 0/1 Predictor) . Let ̂ Y be the optimal predictor with respect to the 0/1-loss based on covariates X,Z,W . Let S denote the optimal predictor with respect to the L 2 loss. The total variation (TV, for short) measure of the predictor ̂ Y , written as P (ˆ y | x 1 ) -P (ˆ y | x 0 ) , can be decomposed into direct, indirect, and spurious effects of X on the score S and the margin complement M as follows:

The above theorem is the first key result of this paper. The disparity between groups with respect to the optimal 0/1-loss predictor, measured by TV x 0 ,x 1 (ˆ y ) can be decomposed into direct, indirect, and spurious contributions coming from (i) the optimal L 2 -loss predictor S (e.g., term x -DE x 0 ,x 1 ( s | x 0 ) ), and (ii) the margin complement M (e.g., term x -DE x 0 ,x 1 ( m | x 0 ) ). This provides a unique capability since for each causal pathway (direct, indirect, spurious) the contribution coming from the probability prediction S can be disentangled from the contribution coming from the optimization procedure itself (i.e., the rounding of the predictor). The former, as we will see shortly, is simply a representation of the bias already existing in the true outcome Y , whereas the latter represents a newly introduced type of bias that is the result of using an automated system. The contribution of the margin complement may act to both ameliorate or amplify an existing disparity, a point we investigate later on.

Example 1 (Disparities in Hiring extended) . Consider the hiring example from Ex. 1 extended with a mediator W indicating whether the applicant has a PhD degree ( W = 1 ) or not ( W = 0 ). Suppose

the augmented SCM is given by:

and P ( U X , U W , U Y ) is such that U X , U W , U Y are independent Unif [0 , 1] random variables. The coefficients satisfy the constraints α, λ > 0 , 0 . 4 < β < 0 . 9 -α 1+ λ . The optimal probability predictor is S ( x, w ) = 0 . 1 + αx + βw . The TV measure of the optimal 0/1 predictor be computed as TV x 0 ,x 1 (ˆ y ) = α β + λ . Using Thm. 1 we can decompose it as:

Along the direct effect, there is a difference of α between the groups in the probability predictor S ( x, w ) that is amplified by α β -α by the margin complement, and since β < 1 we always have bias amplification. Along the indirect effect, there is an initial difference of λβ , amplified by an additional λ -λβ , again amplifying bias. The decomposition provides a breakdown of the disparity of the optimal 0/1 predictor into its constitutive parts. □

We next consider a crucial point mentioned above. The disparity observed in the optimal L 2 score S is simply a representation of the disparity existing in the real world, under suitable causal assumptions:

Theorem 2 (Relationship of L 2 -score S and Outcome Y Decompositions) . Let M be an SCM compatible with the Standard Fairness Model, and let S be the optimal L 2 prediction score. Then, the causal decompositions of the score S and the true outcome Y are symmetric, meaning that x -IE x 1 ,x 0 ( s | x 0 ) = x -IE x 1 ,x 0 ( y | x 0 ) , and x -SE x 1 ,x 0 ( s ) = x -SE x 1 ,x 0 ( y ) .

Based on this result, we can see that the causal influences from X on the true outcome Y are equivalent to the causal influences from X on the optimal prediction score S . Combining this insight with Thm. 1 leads to the following result:

Corollary 3 (Causal Decomposition of Optimal 0/1 Predictor) . Under the Standard Fairness Model, the TV measure of the optimal 0/1-loss predictor ̂ Y can be decomposed into contributions from the true outcome Y and the margin complement M :

This corollary provides an important insight compared to Thm. 1: the disparity in the optimal predictor ̂ Y can be decomposed into the contribution inherited from the true outcome Y and the contribution that arises from the optimization procedure (rounding of the predictor). In Appendix D, we describe two important extensions of the main results: (i) to the setting of suboptimal predictors; (ii) to the setting of predictors using group-specific thresholds, which is often the case with post-processing methods used in fair machine learning [Kamiran et al., 2012, Hardt et al., 2016].

3 Weak and Strong Business Necessity

In law, questions of fairness and discrimination can be interpreted based on the disparate treatment (DT) and disparate impact (DI) doctrines of Title VII of the Civil Rights Act of 1964 [Act, 1964]. The DT doctrine, when interpreted causally, can be seen as disallowing a direct type of effect from X onto the outcome. The DI doctrine, however, is more broad, and may prohibit any from of discrimination (be it direct, indirect, or spurious) that results in a large disparity between groups. The core of this

doctrine is the notion of business necessity (BN), which allows certain variables correlated with the protected attribute to be used for prediction due to their relevance to the business itself [Els, 1993] (or more broadly the utility of the decision-maker). Based on the decomposition from Cor. 3, new BN considerations emerge:

Definition 3 (Weak and Strong Business Necessity) . Let M be an SCM compatible with the Standard Fairness Model. Let CE denote a causal pathway (DE, IE, or SE), and let x, x ′ be two distinct values of X . Let x ′′ be a third, arbitrary value of X . If a causal pathway does not fall under business necessity, then we require:

A pathway is said to satisfy weak business necessity if:

A pathway is said to satisfy strong business necessity if:

The above definition distinguishes between three important cases, and sheds light on a new aspect of the concept of business necessity. According to the definition, there are three versions of BN considerations:

A causal pathway is not in the BN set, and is considered discriminatory. In this case, both the contribution of the prediction score S and the margin complement M need to be equal to 0 (i.e., no discrimination is allowed along the pathway),

A causal pathway satisfies weak BN, and is not considered discriminatory. In this case, the effect of X on the prediction score S needs to equal the effect of X onto the true outcome Y along the same pathway [Plecko and Bareinboim, 2024b]. However, the contribution of the margin complement M along the pathway needs to equal 0 .

A causal pathway satisfies strong BN, and is not considered discriminatory. Similarly as for weak necessity, the effect of X on S needs to equal the effect of X on Y , but in this case, the contribution of the margin complement M is unconstrained.

The distinction between cases (2) and (3) opens the door for new regulatory requirements and specifications. In particular, whenever a causal effect is considered non-discriminatory, the attribute X needs to affect S to the extent to which it does in the real world. However, the system designer also needs to decide whether a difference existing in the predicted probabilities S is allowed to be amplified (or ameliorated) by means of rounding. The latter point distinguishes between weak and strong BN, and should be a consideration of any system designer issuing binary decisions. In Alg. 1, we propose a formal approach for evaluating considerations of weak and strong BN for any input of a predictor ̂ Y and a prediction score S .

4 Identification, Estimation, and Sample Influence

In Thm. 1 and Cor. 3 the observed disparity in the TV measure of the optimal 0/1 predictor is decomposed into its constitutive components. The quantities appearing in the decomposition are counterfactuals, and thus the question of identification of these quantities needs to be addressed. In other words, we need to understand whether these quantities can be uniquely computed based on the available data and the causal assumptions. The following is a positive answer:

Proposition 4 (Identification and Estimation of Causal Measures) . Let M be an SCM compatible with the Standard Fairness Model, and let P ( V ) be its observational distribution. The x -specific direct, indirect, and spurious effects of X on the outcome Y , predictor ̂ Y , prediction score S , and the margin complement M are identifiable (uniquely computable) from P ( V ) and the SFM. Denote by f ( x, z, w ) estimator of ❊ [ T | x, z, w ] , and by ˆ P ( x | v ′ ) the estimator of the probability P ( x | v ′ ) for different choices of v ′ . For T ∈ { Y, ̂ Y , S, M } , the effects can be estimated as:

The proof of the proposition, together with the identification expressions for the different quantities can be found in Appendix B. We next define the sample influences for the different estimators:

Definition 4 (Sample Influence) . The sample influence of the i -th sample on the estimator x -CE est of the causal effect CE is given by corresponding term in the summations in Eqs. 24-26. For instance, the i -th sample influence on x -DE est x 0 ,x 1 ( t | x 0 ) is given by (and analogously for IE, SE terms):

The sample influences tell us how each of the samples contributes to the overall estimator of the quantity. These sample-level contributions may be interesting to investigate from the point of view of the system designer, including identifying any subpopulations that are discriminated against. For direct sample influences, the following proposition can be proved:

Proposition 5 (Direct Effect Sample Influence) . The SI-DE ( i ) in Eq. 27 is an estimator of

where ❊ [ T x 1 ,W x 0 -T x 0 | x 0 , z i , w i ] is the ( x 0 , z i , w i ) -specific direct effect of X on T . Prop. 5 demonstrates an important point - namely that the sample influences along the direct path are not just quantities of statistical interest, but also causally meaningful quantities. In particular, the influence of the i -th sample is proportional to the direct effect of the x 0 → x 1 transition for the group of units u compatible with the event x 0 , z i , w i . The influence is further proportional to P ( w i , z i | x 0 ) /P ( w i , z i ) that measures how much more likely the covariates z i , w i of the i -th sample are in the X = x 0 group (for which the discrimination is quantified) vs. the overall population. Therefore, practitioners also have a causal reason for investigating these sample influences.

5 Experiments

We analyze the MIMIC-IV (Ex. 2), COMPAS (Ex. 3), and Census (Ex. 4, Appendix C) datasets.

Example 2 (Acute Care Triage on MIMIC-IV Dataset [Johnson et al., 2023]) . Clinicians in the Beth Israel Deaconess Medical Center in Boston, Massachusetts treat critically ill patients admitted to

(a)

MIMIC-IVDecomposition

(b) DE Sample Influences

3

0.075

Sample Influence

2

Size

Effects

0.050

0

0.025

1

-2

0.000

Ctf-DE

Ctf-IE

0

2500

5000

7500

10000

Causal Effect

Sample Ranking

Component

OutcomeY

Margin Complement M

Component

OutcomeY

Margin Complement M

the intensive care unit (ICU). For all patients, various physiological and treatment information is collected 24 hours after admission, and the available data consists of (grouped into the Standard Fairness model): protected attribute X , in this case race ( x 0 African-American, x 1 White), set of confounders Z = {sex, age, chronic health status}, set of mediators W = {lactate, SOFA score, admission diagnosis, PaO 2 /FiO 2 ratio, aspartate aminotransferase}.

Clinicians are interested in patients who require closer monitoring. They want to determine the top half of the patients who are the most likely to (i) die during their hospital stay; (ii) have an ICU stay longer than 10 days. This combined outcome is labeled Y . These high-risk patients will remain in the most acute care unit. To predict the outcome, clinicians use the electronic health records (EHR) data of the hospital and construct score predictions S and a binary predictor ̂ Y = ( S ( x, z, w ) > Quant(0 . 5; S )) that selects the top half of the patients.

✶ To investigate the fairness implications of the new AI-based system, they use the decomposition described in Cor. 3 to investigate different contributions to the resulting disparity. The decomposition is shown in Fig. 3(a), and uncovers a number of important effects. Firstly, along the direct effect, x -DE x 0 ,x 1 ( y ) and x -DE x 0 ,x 1 ( m ) are larger than 0 , meaning that minority group individuals have a lower chance of receiving acute care (purely based on race). Along the indirect and spurious effects, the situation is different: x -IE x 1 ,x 0 ( y ) and x -SE x 1 ,x 0 ( y ) and their respective margin complement contributions are different from 0 and negative - implying that minority group individuals have a larger probability of being given acute care as a result of confounding and mediating variables. Finally, the direct effect sample influences (Fig. 3(b)) highlight that the margin complements are large for a small minority of individuals, requiring further subgroup investigation by the hospital team. □

Example 3 (Recidivism Prevention on the COMPAS Dataset [Larson et al., 2016]) . Courts in Broward County, Florida use machine learning algorithms, developed by a private company called Northpointe, to predict whether individuals released on parole are at high risk of re-offending within 2 years ( Y ). The algorithm is based on the demographic information Z ( Z 1 for gender, Z 2 for age), race X ( x 0 denoting White, x 1 Non-White), juvenile offense counts J , prior offense count P , and degree of charge D . The courts wish to know which individuals are highly likely to recidivate, such that their probability of recidivism is above 50%. The company constructs a prediction score ˆ S NP and the court subsequently uses this for deciding whether to detain individuals at high risk of re-offending.

̸

After a court hearing in which it was decided that the indirect and spurious effects fall under business necessity requirements, a team from ProPublica wishes to investigate the implications of using the automated predictions ˆ S NP . They obtain the relevant data and apply Alg. 1, with the results shown in Fig. 4. The team first compares the decompositions of the true outcome Y and the predictor ˆ S NP (Fig. 4(a)). For the spurious effect, they find that x -SE x 1 ,x 0 ( y ) is not statistically different from x -SE x 1 ,x 0 (ˆ s NP ) , in line with BN requirements. For the indirect effects, they find that the indirect effect is lower for the predictor ˆ S NP compared to the true outcome Y , indicating no concerning violations. However, for the direct effect, while the x -DE x 0 ,x 1 ( y ) is not statistically different from 0 , the predictor ˆ S NP has a significant direct effect of X , i.e., x -DE x 0 ,x 1 (ˆ s NP ) = 0 . This indicates a violation of the fairness requirements determined by the court.

(a)

COMPASS

ANP

andYDecompositions

(q)

COMPASDecomposition

(c)DE Sample Influences

0.10

0.04

Sample Influence

0.05

0

0.02

F00°0

F000

-1

-0.05

-0.02

Ctf-DE

Ctf-IE

Ctf-DE

Ctf-IE

0

2000

4000

6000

CausalEffect

Causal Effect

Sample Ranking

Outcome

Component

OutcomeY

Margin Complement M

Component

OutcomeY

Margin Complement M

̸

After comparing the decompositions of ˆ S NP and Y , the team moves onto understanding the contributions of the margin complements (Fig. 4b). For each effect, there is a pronounced impact of the margin complements. For the direct effect (not under BN), the non-zero margin complement contribution x -DE x 1 ,x 0 ( m ) = 0 represents a violation of fairness requirements. For the indirect and spurious effects, the ProPublica team realizes the court did not specify anything about margin complement contributions - based on this, for the next court hearing they are preparing an argument showing that the effects x -IE x 1 ,x 0 ( m ) and x -SE x 1 ,x 0 ( m ) are significantly different from 0 , thereby exacerbating the differences between groups. Finally, based on sample influences (Fig. 4c), they realize that the direct effect is driven by a small minority of individuals, and they decide to investigate this further. □

6 Conclusion

In this paper, we developed tools for understanding the fairness impacts of transforming a continuous prediction score S into binary predictions ̂ Y or binary decisions D . In Thm. 1 and Cor. 3 we showed that the TV measure of the optimal 0/1 predictor decomposes into direct, indirect, and spurious contributions that are inherited from the true outcome Y in the real world, and also contributions from the margin complement M (Def. 1) arising from the automated optimization procedure. This observation motivated new notions of weak and strong business necessity (BN) - in the former case, differences inherited from the true outcome Y are allowed to be propagated into predictions or decisions, while any differences resulting from the optimization procedure are disallowed. In contrast, strong BN allows both of these differences and does not prohibit possible disparity amplification. In Alg. 1, we developed a formal procedure for assessing weak and strong BN. Finally, real-world examples demonstrated that the tools developed in this paper are of genuine importance in practice, since converting continuous predictions into binary decisions may often result in bias amplification in practice - highlighting the need for this type of analysis, and the importance of regulatory oversight.

References

Elston v. Talladega County Bd. of Educ. 997 F.2d 1394 (11th Cir. 1993), 1993. United States Court of Appeals for the Eleventh Circuit.

R. Act. Civil rights act of 1964. Title VII, Equal Employment Opportunities , 1964.

V. Anand, A. H. Ribeiro, J. Tian, and E. Bareinboim. Causal effect identification in cluster dags. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37(10), pages 12172-12179, 2023.

Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias: There's software used across the country to predict future criminals. and it's biased against blacks. ProPublica , 5 2016. URL https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing .

Bareinboim, J. D. Correa, D. Ibeling, and T. Icard. On pearl's hierarchy and the foundations of causal inference. In Probabilistic and Causal Inference: The Works of Judea Pearl , page 507-556. Association for Computing Machinery, New York, NY, USA, 1st edition, 2022.

D. Blau and L. M. Kahn. The gender earnings gap: learning from international comparisons. The American Economic Review , 82(2):533-538, 1992.

D. Blau and L. M. Kahn. The gender wage gap: Extent, trends, and explanations. Journal of economic literature , 55(3):789-865, 2017.

Brennan, W. Dieterich, and B. Ehret. Evaluating the predictive validity of the compas risk and needs assessment system. Criminal Justice and Behavior , 36(1):21-40, 2009.

Buolamwini and T. Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In S. A. Friedler and C. Wilson, editors, Proceedings of the 1st Conference on Fairness, Accountability and Transparency , volume 81 of Proceedings of Machine Learning Research , pages 77-91, NY, USA, 2018.

Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 7801-7808, 2019.

Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Technical Report arXiv:1703.00056, arXiv.org, 2017.

Chouldechova and A. Roth. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 , 2018.

B. Darlington. Another look at 'cultural fairness' 1. Journal of educational measurement , 8(2): 71-82, 1971.

Datta, M. C. Tschantz, and A. Datta. Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. Proceedings on Privacy Enhancing Technologies , 2015(1): 92-112, Apr. 2015. doi: 10.1515/popets-2015-0007.

Dwork, C. Ilvento, and M. Jagadeesan. Individual fairness in pipelines. arXiv preprint arXiv:2004.05167 , 2020.

Hardt, E. Price, and N. Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29:3315-3323, 2016.

Imai, Z. Jiang, D. J. Greiner, R. Halen, and S. Shin. Experimental evaluation of algorithm-assisted human decision-making: Application to pretrial public safety assessment. Journal of the Royal Statistical Society Series A: Statistics in Society , 186(2):167-189, 2023.

E. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, T. J. Pollard, B. Moody, B. Gow, L.-w. H. Lehman, et al. Mimic-iv, a freely accessible electronic health record dataset. Scientific data , 10(1):1, 2023.

Kamiran, A. Karim, and X. Zhang. Decision theory for discrimination-aware classification. In 2012 IEEE 12th International Conference on Data Mining , pages 924-929. IEEE, 2012.

E. Khandani, A. J. Kim, and A. W. Lo. Consumer credit-risk models via machine-learning algorithms. Journal of Banking & Finance , 34(11):2767-2787, 2010.

Kilbertus, M. Rojas-Carulla, G. Parascandolo, M. Hardt, D. Janzing, and B. Schölkopf. Avoiding discrimination through causal reasoning. arXiv preprint arXiv:1706.02744 , 2017.

Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan. Human decisions and machine predictions. The quarterly journal of economics , 133(1):237-293, 2018.

J. Kusner, J. Loftus, C. Russell, and R. Silva. Counterfactual fairness. Advances in neural information processing systems , 30, 2017.

Larson, S. Mattu, L. Kirchner, and J. Angwin. How we analyzed the compas recidivism algorithm. ProPublica (5 2016) , 9, 2016.

T. Liu, S. Dean, E. Rolf, M. Simchowitz, and M. Hardt. Delayed impact of fair machine learning. In International Conference on Machine Learning , pages 3150-3158. PMLR, 2018.

F. Mahoney and J. M. Mohen. Method and system for loan origination and underwriting, Oct. 23 2007. US Patent 7,287,008.

Nabi and I. Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Nilforoshan, J. D. Gaebler, R. Shroff, and S. Goel. Causal conceptions of fairness and their consequences. In International Conference on Machine Learning , pages 16848-16887. PMLR, 2022.

Pager. The mark of a criminal record. American journal of sociology , 108(5):937-975, 2003.

Pearl. Causality: Models, Reasoning, and Inference . Cambridge University Press, New York, 2000. 2nd edition, 2009.

Pierson, D. M. Cutler, J. Leskovec, S. Mullainathan, and Z. Obermeyer. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nature Medicine , 27(1): 136-140, 2021.

Pleˇ cko and E. Bareinboim. Causal fairness analysis: A causal toolkit for fair machine learning. Foundations and Trends® in Machine Learning , 17(3):304-589, 2024.

Plecko and E. Bareinboim. Causal fairness for outcome control. Advances in Neural Information Processing Systems , 36, 2024a.

Plecko and E. Bareinboim. Reconciling predictive and statistical parity: A causal approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38 (13), pages 14625-14632, 2024b.

Pleˇ cko and N. Meinshausen. Fair data adaptation with quantile preservation. Journal of Machine Learning Research , 21:242, 2020.

Sanburn. Facebook thinks some native american names are inauthentic. Time , Feb. 14 2015. URL http://time.com/3710203/facebook-native-american-names/ .

Shpitser and J. Pearl. What counterfactuals can be tested. In Proceedings of the Twenty-third Conference on Uncertainty in Artificial Intelligence , page 352-359, 2007.

Sweeney. Discrimination in online ad delivery. Technical Report 2208240, SSRN, Jan. 28 2013. URL http://dx.doi.org/10.2139/ssrn.2208240 .

T. Sweeney and C. Haney. The influence of race on sentencing: A meta-analytic review of experimental studies. Behavioral Sciences & the Law , 10(2):179-195, 1992.

Wu, L. Zhang, X. Wu, and H. Tong. Pc-fairness: A unified framework for measuring causalitybased fairness. Advances in neural information processing systems , 32, 2019.

Zhang and E. Bareinboim. Equality of opportunity in classification: A causal approach. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 , pages 3671-3681, Montreal, Canada, 2018a. Curran Associates, Inc.

Zhang and E. Bareinboim. Fairness in decision-making-the causal explanation formula. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018b.

Zhang, J. Tian, and E. Bareinboim. Partial counterfactual identification from observational and experimental data. In Proceedings of the 39th International Conference on Machine Learning , 2022.

Technical Appendices for Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making

The source code for reproducing all the experiments can be found in our Github code repository https://github.com/dplecko/mind-the-gap . The code is also included with the supplementary materials, in the folder source-code . All experiments were performed on a MacBook Pro, with the M3 Pro chip and 36 GB RAM on macOS 14.1 (Sonoma). All experiments can be run with less than 1 hour of compute on the above-described machine or equivalent.

A Proofs of Key Theorems

Thm. 1 Proof: Notice that by definition we have

Now, let E be any observed event, and let C be a counterfactual clause representing a possibly counterfactual intervention. Note that we can write:

❊ ❊ Now, we can expand the TV measure of ̂ Y as follows:

Thm. 2 Proof: Let M be an SCM compatible with the SFM in Fig. 2 as per theorem assumption. Let the optimal L 2 prediction score S be given by S ( x, z, w ) = ❊ [ Y | x, z, w ] . More precisely, we can let f S be the structural mechanism of S , taking X,Z,W as inputs. The structural mechanism of S is then given by:

f S ( x, z, w ) = ❊ [ Y | x, z, w ] . (45) Therefore, the score S is a deterministic function of X,Z,W , and we can add it to the standard fairness model as shown in Fig. 5a. Now, note that for a potential outcome ❊ [ S x,W x ′ | x ′′ ] we can

write:

Z

X

W

Y

S

Z

X

W

T

x

1

,w

W

x

0

x

1

x

0

w

(a) SFM extended with predictor S .

(b) Counterfactual graph of the SFM.

where Eq. 48 is using the independence S x,z,w ⊥ ⊥ W x ′ | X,Z , and Eq. 51 is using the independence Y x,z,w ⊥ ⊥ W x ′ | X,Z , both of which are implied by the standard fairness model extended with the node S above. Based on the equality ❊ [ S x,W x ′ | x ′′ ] = ❊ [ Y x,W x ′ | x ′′ ] for arbitrary values x, x ′ , x ′′ , the claim follows from the appropriate choices: for instance, taking { x = x 1 , x ′ = x 0 , x ′′ = x 0 } and { x = x 0 , x ′ = x 0 , x ′′ = x 0 } , it follows that

❊ ❊ Similarly, analogous choices of x, x ′ , x ′′ can be used to show the equality for indirect and spurious effects.

B Proofs related to Identification & Estimation

Prop. 4 Proof: Consider the x -specific {direct, indirect, spurious} effects of X on a random variable T given by:

We first demonstrate why each of the potential outcomes appearing is identifiable under the Standard Fairness Model. Consider the potential outcome T x 1 ,W x 0 | X = x 0 . For proving its identifiability, we make use of the counterfactual graph [Shpitser and Pearl, 2007] in Fig. 5b. We can expand

P ( T x 1 ,W x 0 = t | x 0 ) as follows:

Using a similar but simplified argument, we also obtain that:

and since P ( T x = t | x ) = P ( t | x ) , all the terms in Eqs. 55-57 can be computed based on observational data.

We next move onto the estimation part of the proof. We again focus on the quantity P ( T x 1 ,W x 0 = t | x 0 ) . Suppose that we have:

where f, ˆ P ( x 0 | w,z ) are estimators, and ˆ P ( w,z ) , ˆ P ( x 0 ) are the empirical distributions:

Now, note that we can re-write the identification expression from Eq. 65 as:

By a coupling of quantities in Eqs. 67-70 to a joint probability space and an application of the continuous mapping theorem we obtain that

(a)

Census2018Decomposition

(b) DE Sample Influences

3

0.075

Sample Influence

2

Size

Effects

0.050

0

0.025

1

-2

0.000

Ctf-DE

Ctf-IE

0

2500

5000

7500

10000

Causal Effect

Sample Ranking

Component

OutcomeY

Margin Complement M

Component

OutcomeY

Margin Complement M

Finally, note that we have

The remaining identification expressions are derived based on the same technique, completing the proof of the proposition.

Prop. 5 Proof: Consider the direct effect sample influence of the i -th sample given by

By assumption, we know that

Therefore, by a coupling of the above quantities to a joint probability space and an application of the continuous mapping theorem, we have that

Note that

completing the proof of the proposition.

C Census 2018 Example

In this appendix, we analyze the Census 2018 dataset, demonstrating an application in the context of labor and salary decisions:

Example 4 (Salary Increase on the Census 2018 Dataset [Pleˇ cko and Bareinboim, 2024]) . The United States Census of 2018 collected broad information about the US Government employees, including demographic information Z ( Z 1 for age, Z 2 for race, Z 3 for nationality), gender X ( x 0 female, x 1 male), marital and family status M , education information L , and work-related information R . The US Government wishes to use this data prospectively to decide whether a new employee should receive a bonus starting package upon signing the employment contract. To determine which employees should receive such a bonus, a Government department decides to predict which of the employees should earn a salary above the median (which is $50,000/year). They construct a machine learning prediction score S that predicts above-median earnings ( Y ), and the department decides to allocate the bonus to all employees who are predicted to be above-median earners with a probability greater than 50% (i.e., ̂ Y = ( S ≥ 0 . 5) ).

✶ A team of investigators within the Government is in charge of assessing what the impacts of this AI system are on the gender pay gap. They collect the required data and perform the decomposition from Cor. 3, shown in Fig. 6(a). The decomposition indicates strong direct effects x -DE x 0 ,x 1 ( y ) and x -DE x 0 ,x 1 ( m ) , which imply that men are more likely to receive a starting bonus than women. The indirect effects x -IE x 1 ,x 0 ( y ) , x -IE x 1 ,x 0 ( m ) are both negative, with the latter effect of the margin complement not being significant. These effects, however, also mean that men are more likely to receive a bonus due to mediating variables. Finally, for the spurious effects, the effects are not significantly different from 0 . Based on these findings, the team decided to return the predictions to the original department, with the requirement that the direct effect of gender on the margin complement, x -DE x 0 ,x 1 ( m ) , must be reduced to 0 , to avoid any possibility of bias amplification. □

D Extending Thm. 1 & Cor. 3

In this appendix, we discuss two important extensions of the results presented in the main paper. First, we note that Cor. 3 provides a decomposition of the TV measure for the optimal 0/1 predictor. In general, data analysts may be interested in analyzing suboptimal predictors using the same set of tools, which is discussed first (Appendix D.1). Second, we note that the results of the paper consider thresholded predictors ̂ Y = 1 ( S ≥ t ) . In practice, it may be desirable to analyze predictors that use group-specific thresholds , e.g., those of the from

where t x is a group-specific threshold that may differ for X = x 0 and X = x 1 groups. This is the second extension we consider, and it is discussed in Appendix D.2.

D.1 Suboptimal Predictors

We now consider how Cor. 3 could be applied to analyze suboptimal predictors. Cor. 3 was based on Thm. 2, which showed that the causal effects of X on Y were equal to the causal effects of X on the optimal prediction score S . In practice, if a suboptimal prediction score ˜ S is used, the symmetry shown in Thm. 2 can no longer be used. Therefore, if a predictor ˜ Y is based on a suboptimal score ˜ S , the result of Cor. 3 cannot be applied directly.

Still, in this case, a variation of Cor. 3 can be used, stated in the following theorem:

Theorem 6 (Decomposing Suboptimal Predictors) . Let ˜ Y be any thresholded predictor based on a prediction score ˜ S , and let ˜ M be its margin complement. Under the Standard Fairness Model in Fig. 2, the TV measure of the predictor ˜ Y can be decomposed into contributions from the true outcome Y , the suboptimality of ˜ S , and the margin complement ˜ M :

Recall that Cor. 3 was a two-way decomposition along each causal pathway, into: (i) the contribution arising from the true outcome Y ; (ii) contribution from the thresholding, i.e., the margin complement M . In Thm. 6, the decomposition is a three-way one. First, as before, there is a contribution of the true outcome Y along the causal pathway in question. Second, there is a contribution of suboptimality

Z

X

W

Y

˜

S

˜

Y

(a) SFM extended with predictor S .

Z

X

W

Y

˜

S

˜

Y

(b) Counterfactual graph of the SFM.

of ˜ S compared to S , which is captured by comparing the effects of X on ˜ S and S along the causal pathway. Finally, there is also the contribution of thresholding, along the margin complement ˜ M of ˜ S . Therefore, when considering a suboptimal predictor, three is an additional term in the decomposition (for each causal pathway) that measures the suboptimality of the prediction score ˜ S .

D.2 Predictors with Group-specific Thresholds

In the main text and Appendix D.1, we considered classifiers of the form

where ˜ S is an arbitrary prediction score. In practice, when fairness considerations are taken into account, the decision-maker may use a group specific threshold, i.e., a classifier of the form:

In Eq. 89, different thresholds t x 0 , t x 1 are used for groups x 0 , x 1 , respectively. The usage of groupspecific thresholds is also widespread in the fair ML literature, for instance Kamiran et al. [2012] uses a post-processing approach with group-specific thresholds to construct a predictor that satisfies demographic parity. Similarly, [Hardt et al., 2016] uses a post-processing approach with groupspecific thresholds for achieving equality of odds. In Fig. 7a, we present the causal diagram that corresponds to a single threshold setting, as in Eq. 88. We note in this case that all of the effects from X to ˜ Y (direct, indirect, spurious), are mediated by the prediction score ˜ S . When considering group-specific thresholds, represented graphically in Fig. 7b, clearly the predictor ˜ Y needs to also take X as an input, on top of ˜ S . Therefore, allowing for group-specific thresholds results in an additional direct effect X → ˜ Y . Motivated by this definition, we introduce the following notion:

Definition 5 (Group-specific Threshold Direct Effect) . Let ˜ Y be a thresholded predictor with groupspecific thresholds, based on a prediction score ˜ S . The group-specific threshold direct effect for a unit U = u is defined as:

The x -specific group-specific direct effect is defined as:

x -DE GST x 0 ,x 1 (˜ y | x ) = ❊ [ u -DE GST x 0 ,x 1 (˜ y ) | X = x ] . (92) The unit-level group-specific threshold direct effect captures the effect of a X = x 0 → X = x 1 change along the pathway X → ˜ Y . It measures if a specific unit U = u is classified differently when considering different thresholds t x 1 , t x 0 . The x -specific version of the effect simply averages the unit-level effect across all units compatible with X = x . Armed with the above definition, in the following result, we provide a new decomposition that can explicitly handle analysis of predictors with group-specific thresholds:

Theorem 7 (Causal Decomposition of Predictor with Group-specific Threshold) . Let ˜ Y be a thresholded predictor with group-specific thresholds as in Eq. 89, based on a prediction score ˜ S . Let ˜ M be

the margin complement of ˜ S with respect to the threshold t x 0 of the X = x 0 group. The TV measure of ˜ Y can be decomposed as:

As one can see, the decomposition in Thm. 7 is similar to that appearing in Thm. 1. However, there is an additional term x -DE GST x 0 ,x 1 (˜ y | x 0 ) appearing, which is the consequence of considering group-specific thresholds. The above theorem provides the first result establishing that, causally, post-processing methods with group-specific thresholds change only the direct effect of X on ˜ Y .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Main claims are corroborated both theoretically and empirically.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: the main limitation is the availability of the appropriate causal diagram, and the implied assumptions (discussed in Sec. 1.1).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes] .

Justification: Appendix A and B provide detailed proofs. Assumptions are stated clearly in each theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: the experiments are described in detail in Sec. 5 & Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: An anonymized code repository is included (see the beginning of technical appendices). The source code is also uploaded as supplementary material. All the data used is publicly available. We note that access to the MIMIC-IV dataset requires approval from the data owners.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes] .

Justification: See Sec. 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: Yes, uncertainty is quantified using bootstrap repetitions (see Sec. 5).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: Described in detail at the beginning of technical appendices.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: The paper introduces a method for scrutinizing/analyzing bias in prediction and decision-making.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] .

Justification: We discuss the need for regulatory oversight based on our results (see Sec. 3 and 6). We do not see potential for negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The paper poses no risk in terms of data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: All data owners are cited (Sec. 5, Appendix C) and licenses of use were respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] .

Justification: The anonymized code repository contains a README file that provides the documentation (instructions) for reproducing all the experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: No crowdsourcing or human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: No research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the impact of downstream thresholding operations on continuous (and possibly fair) prediction scores. The paper argues that inappropriate thresholding can amplify or ameliorate the disparity in predictive performance across groups defined by the protected attribute. Using a causal framework, the paper provides a methodology to separate the disparities in the original predictive score from the disparities introduced due to thresholding.\n\nSTRENGTHS:\n1. One of the motivations of the paper is to understand the downstream utility of fair predictions. In that regard, I appreciate the more practical focus on investigating the barriers faced by popular fairness during real-world implementations. \n2. The disentanglement of different kinds of business necessity requirements seems interesting by itself and allows for potentially meaningful connections between legal and empirical notions of fairness (although I do have some questions on it that are noted later).\n\nWEAKNESSES:\n1. The writing and presentation are often hard to follow and various parts of the analysis seem under-explained, which makes evaluating the paper’s claims difficult. For example, Definition 1 defines $M$ using a generic $t$-value, whereas the proof of Theorem 1 in the appendix, if I understand correctly, directly uses $t=1/2$. If the theorem is based on a specific $t$ value, then that should be made clear in the theorem and the surrounding text in the main body.\\\n  Similarly, when discussing “strong business necessity”, there’s very little explanation of what it means for a causal pathway to be “unconstrained”. With a lack of a clear explanation of this concept, I am not totally sure about the difference between weak and strong BN. \n\n2. Related to thresholding, the paper never goes into the details of how a certain threshold is chosen. Again, seems like Theorem 1’s proof and Example 1 use $t=1/2$, but that is not the only possible choice of threshold that can be used in practice. In fact, if all the claims in the paper are based on the explicit assumption that $t=1/2$, then I don’t see the claims being generalizable to other threshold classifiers and also not applicable to various kinds of classifiers used in practice.  \n\n3. Additionally, there is a wide post-processing fairness literature that essentially proposes methods to choose group-specific thresholds so that the biases from training data and prediction scores do not propagate to the final binary predictions (e.g., Hardt et al. 2016 and related papers). I would strongly recommend an expanded discussion on this related work, especially comparing the proposed approach to other prior fairness works on choosing appropriate thresholds to guarantee outcome fairness. \n\n4. Minor points and typos.\\\n    a. Potential missing word in Line 73 around “…ramifications more”.\\\n    b. In Example 1, seems like the outcome should be 0 and 1, and not $y_0$ and $y_1$.\\\n    c. Line 138, function $pa$ is not defined. I am assuming it means “parent variable” but needs to be properly introduced and described.\n\nQUESTIONS:\nThe main questions I have are related to the above points.\n1. Does the paper primarily focus on $t=1/2$ or are the results applicable for other threshold values as well?\n2. Do the issues associated with margin complement persist even when after using post-processing approaches to achieve outcome fairness (like those introduced in Hardt et al. 2016)?\n3. What’s meant by a causal pathway being “unconstrained” when defining strong BN?\n\nLIMITATIONS:\nSome limitations are acknowledged but the paper could do a better job of expanding on the limitations related to the full knowledge of structural causal models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how much thresholding a predictor affects the disparity in the decisions according to sensitive attributes and formalizes new notions of business necessity based on the causal graph of features, outcome and prediction function.\n\nSTRENGTHS:\n- Understanding the amplification of bias along the machine learning and decision making pipeline is an important topic.\n\n- There are some very interesting ideas that could have benefited from better execution in the paper. Having some technical definition of business necessity makes claims of BN testable in the real world, and could have policy impact.\n\n- A relatively comprehensive attempt to study the topic. Results include decomposition, identification, with proofs, as well as examples/experiments. With some significant improvements, it will be a nice contribution.\n\nWEAKNESSES:\n- Technical writing needs improvement and some key definitions are clearly wrong.\n(1) Looking at Definition 3, weak BN implies strong BN according to this definition. Why is the condition for weak BN stronger than for strong BN? I was also unable to make sense of the rest of this definition, e.g. part 1.\n(2) some quantities/notation are not properly defined. E.g. in definition 3, what are s, x'', m, x', y, and where are they defined? There should be no reasonable doubt about what these letters refer to. Same comment holds for Theorem 1 (what is m, s?), Theorem 2 and corollary 2.\n(3) Please use the notation :=, or similar, to denote definitional equivalence (rather than an equality _claim_).\n\n- Theorem 1 and 2 appear to be basic, as far as I can tell from the definitions and the proof. I'm not sure why the decomposition, which follows from definition, needs to be presented as a main theorem. The main novelty appears to be the definition. If this is the case, you might consider making this clear to the reader instead of presenting something as more complex that it needs to be. If this is not the case, is there a way you can make the technical contribution clearer in the proof and writing?\n\n- Example 1 is not a reasonable example. In this case (where you ONLY have gender on which to make a definition, and there is practically no signal from it), there would be no reason to use a predictor at all.\n\n-Related work: The discussion of closely related work is sparse. The connection with the causal fairness literature is relegated to a few sentences in the introduction, and there is no meaningful discussion of the contributions of prior work. Please elaborate on the connection between your definitions of business need and causal fairness? \n\nLack of citations and discussion of prior work. The paper states \"Most of the literature in fair machine learning\n4 focuses on defining and achieving fairness criteria in the context of prediction,\n5 while not explicitly focusing on how these predictions may be used later on in\n6 the pipeline.\"\n\nThis is an inaccurate representation of the literature. Even one of the cited works Chouldechoulva (2016) has an entire section on how scores impact decisions. Minimizing prior work on fairness in decision making is at best unhelpful to the reader, and at worst, misinforms.\n\nFor prior work on fairness in decision making (indeed, how predictions are used later in the pipeline), also see the following and the cited works within:\n\nLiu, L. T., Dean, S., Rolf, E., Simchowitz, M., & Hardt, M. (2018, July). Delayed impact of fair machine learning. In International Conference on Machine Learning (pp. 3150-3158). PMLR.\n\nChouldechova, Alexandra, and Aaron Roth. \"The frontiers of fairness in machine learning.\" arXiv preprint arXiv:1810.08810 (2018).\n\nDwork, Cynthia, Christina Ilvento, and Meena Jagadeesan. \"Individual fairness in pipelines.\" arXiv preprint arXiv:2004.05167 (2020).\n\nQUESTIONS:\n1. What is the key technical contribution in Theorem 1 and 2, other than the novelty of the definitions (which allows you to write TV in a new, \"human interpretable\" way)?\n2. What is the connection between your definitions of business need and counterfactual fairness (e.g. Kusner at al 2017)? \n3. Please explain the corrected version of Definition 3 and what is the intended use case for each of the three definitions?\n4. What is the potential negative societal impact of using the weak and strong business need definitions in a regulatory context?\n\nLIMITATIONS:\nIncompletely addressed impact of weak and strong business need definitions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how thresholding the score of a predictive model as a decision rule influences the fairness of the final decisions. In particular, the authors consider binary decisions or predictions of a true binary outcome in presence of a sensitive binary attribute.  In that context, the authors show that the fairness measure of the total variation depends on the direct, indirect and spurious effects due to the true outcome and due the margin complement; a measure introduced by the authors expressing the difference between the predicted score and the final prediction after thresholding.  Under this new measure, the authors also introduce the notions of weak and strong business necessity (BN); under weak BN, disparities due to the true outcome are considered fair, while disparities due to the thresholding rule are not tolerated—zero marginal complement; under weak BN disparities due to both the outcome and the thresholding rule are tolerated.\n\nSTRENGTHS:\nThe paper is well organized, has a clear structure and seem appropriately placed into contemporary literature.\n\nThe technical contributions appear strong, well motivated and clearly presented.  The examples do help the reader to understand the intuition and follow the technical results.  \n\nThe problem of potential bias amplification due to applying a thresholding rule on predicted scores appears quite interesting and seems directly relevant and applicable in high stakes domains as the authors demonstrate through the real world data examples.\n\nWEAKNESSES:\nThe abstract appears somewhat convoluted and confusing providing low level technical details, especially in lines 12-21. It might have been more helpful to explain the contribution at a higher level, while focusing more on the intuition.\n\nQUESTIONS:\nThe causal decomposition of the TV measure are on the optimal 0/1 predictor. How this decomposition would change for a sub-optimal (e.g. near optimal) predictor?\n\nLIMITATIONS:\nThe authors very briefly refer to the limitations due to the assumptions on the standard fairness model. It would be useful perhaps to elaborate concretely on some of these assumptions; e.g., on how the notion of the margin complement  could be extended assuming that the protected attribute takes values in a discrete set or a continuous range.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}