SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jacy Reese Anthis $^{1,2,3*}$ , Victor Veitch $^{1}$
<sup>1</sup>University of Chicago, <sup>2</sup>University of California, Berkeley, <sup>3</sup>Sentience Institute
Counterfactual fairness requires that a person would have been classified in the same way by an AI or other algorithmic system if they had a different protected class, such as a different race or gender. This is an intuitive standard, as reflected in the U.S. legal system, but its use is limited because counterfactuals cannot be directly observed in real-world data. On the other hand, group fairness metrics (e.g., demographic parity or equalized odds) are less intuitive but more readily observed. In this paper, we use causal context to bridge the gaps between counterfactual fairness, robust prediction, and group fairness. First, we motivate counterfactual fairness by showing that there is not necessarily a fundamental trade-off between fairness and accuracy because, under plausible conditions, the counterfactually fair predictor is in fact accuracy-optimal in an unbiased target distribution. Second, we develop a correspondence between the causal graph of the data-generating process and which, if any, group fairness metrics are equivalent to counterfactual fairness. Third, we show that in three common fairness contexts—measurement error, selection on label, and selection on predictors—counterfactual fairness is equivalent to demographic parity, equalized odds, and calibration, respectively. Counterfactual fairness can sometimes be tested by measuring relatively simple group fairness metrics.
The increasing use of artificial intelligence and machine learning in high stakes contexts such as healthcare, hiring, and financial lending has driven widespread interest in algorithmic fairness. A canonical example is risk assessment in the U.S. judicial system, which became well-known after a 2016 investigation into the recidivism prediction tool COMPAS revealed significant racial disparities [2]. There are several metrics one can use to operationalize fairness. These typically refer to a protected class or sensitive label, such as race or gender, and define an equality of prediction rates across the protected class. For example, demographic parity, also known as statistical parity or group parity, requires equal classification rates across all levels of the protected class [7].
However, there is an open challenge of deciding which metrics to enforce in a given context [e.g., 24, 44]. Appropriateness can vary based on different ways to measure false positive and false negative rates and the costs of such errors [19] as well as the potential trade-offs between fairness and accuracy [11, 12, 21, 55]. Moreover, there are well-known technical results showing that it is impossible to achieve the different fairness metrics simultaneously, even to an  $\epsilon$ -approximation [10, 30], which suggest that a practitioner's context-specific perspective may be necessary to achieve satisfactory outcomes [4].
A different paradigm, counterfactual fairness, requires that a prediction would have been the same if the person had a different protected class [31]. This matches common intuitions and legal standards
*Corresponding author: anthis@uchicago.edu
Figure 1: A pipeline to detect and enforce counterfactual fairness. This is facilitated by Theorem 2, a correspondence between group fairness metrics and counterfactual fairness, such that tools to detect and enforce group fairness can be applied to counterfactual fairness.
[25]. For example, in McCleskey v. Kemp (1987), the Supreme Court found "disproportionate impact" as shown by group disparity to be insufficient evidence of discrimination and required evidence that "racial considerations played a part." The Court found that there was no merit to the argument that the Georgia legislature had violated the Equal Protection Clause via its use of capital punishment on the grounds that it had not been shown that this was [sic] "because of an anticipated racially discriminatory effect." While judges do not formally specify causal models, their language usually evokes such counterfactuals; see, for example, the test of "but-for causation" in Bostock v. Clayton County (2020).
We take a new perspective on counterfactual fairness by leveraging causal context. First, we provide a novel motivation for counterfactual fairness from the perspective of robust prediction by showing that, with certain causal structures of the data-generating process, counterfactually fair predictors are accuracy-optimal in an unbiased target distribution (Theorem 1). The contexts we consider i have two important features: the faithfulness of the causal structure, in the sense that no causal effects happen to be precisely counterbalanced, which could lead to a coincidental achievement of group fairness metrics, and that the association between the label  $Y$  and the protected class  $A$  is "purely spurious," in the sense that intervening on  $A$  does not affect  $Y$  or vice versa.
Second, we address the fundamental challenge in enforcing counterfactual fairness, which is that, by definition, we never directly observe counterfactual information in the real world. To deal with this, we show that the causal context connects counterfactual fairness to observational group fairness metrics (Theorem 2 and Corollary 2.1), and this correspondence can be used to apply tools from group fairness frameworks to the ideal of achieving counterfactual fairness (Figure 1). For example, the fairness literature has developed techniques to achieve group fairness through augmenting the input data [13, 27, 42], data generation [1], or regularization [45]. With this pipeline, these tools can be applied to enforce counterfactual fairness by achieving the specific group fairness metric that corresponds to counterfactual fairness in the given context. In particular, we show that in each fairness context shown in Figure 2, counterfactual fairness is equivalent to a particular metric (Corollary 2.2). This correspondence can be used to apply tools built for group fairness to the goal of counterfactual fairness or vice versa.
Finally, we conduct brief experiments in a semi-synthetic setting with the Adult income dataset [3] to confirm that a counterfactually fair predictor under these conditions achieves out-of-distribution accuracy and the corresponding group fairness metric. To do this, we develop a novel counterfactually fair predictor that is a weighted average of naive predictors, each under the assumption that the observation is in each protected class (Theorem 3).
(a) Measurement error (corresponding to demographic parity)
(b) Selection on label (corresponding to equalized odds)
(c) Selection on predictors (corresponding to calibration)
Figure 2: DAGs for three causal contexts in which a counterfactually fair predictor is equivalent to a particular group fairness metric. Measurement error is represented with the unobserved true label  $\tilde{Y}$  in a dashed circle, which is a parent of the observed label  $Y$ , and selection is represented with the selection status  $S$  in a rectangle, indicating an observed variable that is conditioned on in the data-generating process, which induces an association between its parents. Bidirectional arrows indicate that either variable could affect the other.
In summary, we make three main contributions:
1. We provide a novel motivation for counterfactual fairness from the perspective of robust prediction by showing that, in certain causal contexts, the counterfactually fair predictor is accuracy-optimal in an unbiased target distribution (Theorem 1).
2. We provide criteria for whether a causal context implies equivalence between counterfactual fairness and each of the three primary group fairness metrics (Theorem 2) as well as seven derivative metrics (Corollary 2.1).
3. We provide three causal contexts, as shown in Figure 2, in which counterfactual fairness is equivalent to a particular group fairness metric: measurement error with demographic parity, selection on label with equalized odds, and selection on predictors with calibration (Corollary 2.2).
For a running example, consider the case of recidivism prediction with measurement error, as shown in Figure 2a, where  $X$  is the set of predictors of recidivism, such as past convictions;  $\tilde{Y}$  is committing a crime;  $Y$  is committing a crime that has been reported and prosecuted; and  $A$  is race, which in this example affects the likelihood crime is reported and prosecuted but not the likelihood of crime itself. In this case, we show that a predictor is counterfactually fair if and only if it achieves demographic parity.
Previous work has developed a number of fairness metrics, particularly demographic parity [7], equalized odds [23], and calibration [19]. Verma and Rubin [48] details 15 group fairness metrics—also known as observational or distributional fairness metrics—including these three, and Makhlouf, Zhioua, and Palamidessi [34] details 19 causal fairness metrics, including counterfactual fairness [31]. Many notions of fairness can be viewed as specific instances of the general goal of invariant representation [18, 53, 55, 56]. Much contemporary work in algorithmic fairness develops ways to better enforce certain fairness metrics, such as through reprogramming a pretrained model [54],
contrastive learning when most data lacks labels of the protected class [8], and differentiating between critical and non-critical features for selective removal [16].
Several papers have criticized the group fairness metrics and enriched them correspondingly. For example, one can achieve demographic parity by arbitrarily classifying a subset of individuals in a protected class, regardless of the other characteristics of those individuals. This is particularly important for subgroup fairness, in which a predictor can achieve the fairness metric for one category within the protected class but fail to achieve it when subgroups are considered across different protected classes, such as people who have a certain race and gender. This is particularly important given the intersectional nature of social disparities [14], and metrics have been modified to include possible subgroups [29].
Well-known impossibility theorems show that fairness metrics such as the three listed above cannot be simultaneously enforced outside of unusual situations such as a perfectly accurate classifier, a random classifier, or a population with equal rates across the protected class, and this is true even to an  $\epsilon$ -approximation [10, 30, 40]. Moreover, insofar as fairness and classification accuracy diverge, there appears to be a trade-off between the two objectives. Two well-known papers in the literature, Corbett-Davies et al. [12] and Corbett-Davies and Goel [11], detail the cost of fairness and argue for the prioritization of classification accuracy. More recent work on this trade-off includes a geometric characterization and Pareto optimal frontier for invariance goals such as fairness [55], the cost of causal fairness in terms of Pareto dominance in classification accuracy and diversity [37], the potential for increases in fairness via data reweighting in some cases without a cost to classification accuracy [32], and showing optimal fairness and accuracy in an ideal distribution given mismatched distributional data [17].
Our project uses the framework of causality to enrich fairness metric selection. Recent work has used structural causal models to formalize and solve a wide range of causal problems in machine learning, such as disentangled representations [41, 50], out-of-distribution generalization [41, 49, 50], and the identification and removal of spurious correlations [47, 50]. While Veitch et al. [47] focuses on the correspondence between counterfactual invariance and domain generalization, Remark 3.3 notes that counterfactual fairness as an instance of counterfactual invariance can imply either demographic parity or equalized odds in the two graphs they analyze, respectively, and Makar and D'Amour [33] draw the correspondence between risk invariance and equalized odds under a certain graph. The present work can be viewed as a generalization of these results to correspondence between counterfactual fairness, which is equivalent to risk invariance when the association is "purely spurious," and each of the group fairness metrics.
3 Preliminaries and Examples
For simplicity of exposition, we focus on binary classification with an input dataset  $X \in \mathcal{X}$  and a binary label  $Y \in \mathcal{Y} = \{0,1\}$ , and in our data we may have ground-truth unobserved labels  $\tilde{Y} \in \{0,1\}$  that do not always match the potentially noisy observed labels  $Y$ . The goal is to estimate a prediction function  $f: \mathcal{X} \mapsto \mathcal{Y}$ , producing estimated labels  $f(X) \in \{0,1\}$ , that minimizes  $\mathbb{E}[\ell(f(X), Y)]$  for some loss function  $\ell: \mathcal{Y} \times \mathcal{Y} \mapsto \mathbb{R}$ . There are a number of popular fairness metrics in the literature that can be formulated as conditional probabilities for some protected class  $A \in \{0,1\}$ , or equivalently, as independence of random variables, and we can define a notion of counterfactual fairness.
Definition 1. (Demographic parity). A predictor  $f(X)$  achieves demographic parity if and only if:
$$
\mathbb {P} (f (X) = 1 \mid A = 0) = \mathbb {P} (f (X) = 1 \mid A = 1) \iff f (X) \perp A
$$
Definition 2. (Equalized odds). A predictor  $f(X)$  achieves equalized odds if and only if:
$$
\mathbb {P} (f (X) = 1 \mid A = 0, Y = y) = \mathbb {P} (f (X) = 1 \mid A = 1, Y = y) \iff f (X) \perp A \mid Y
$$
Definition 3. (Calibration). A predictor  $f(X)$  achieves calibration if and only if:
$$
\mathbb {P} (Y = 1 \mid A = 0, f (X) = 1) = \mathbb {P} (Y = 1 \mid A = 1, f (X) = 1) \iff Y \perp A \mid f (X)
$$
Definition 4. (Counterfactual fairness). A predictor  $f(X)$  achieves counterfactual fairness if and only if for any  $a, a' \in A$ :
$$
f (X (a)) = f (X \left(a ^ {\prime}\right))
$$
In this case,  $x(0)$  and  $x(1)$  are the potential outcomes. That is, for an individual associated with data  $x \in X$  with protected class  $a \in A$ , their counterfactual is  $x(a')$  where  $a' \in A$  and  $a \neq a'$ . The counterfactual observation is what would obtain if the individual had a different protected class. We represent causal structures as directed acyclic graphs (DAGs) in which nodes are variables and directed arrows indicate the direction of causal effect. Nodes in a solid circle are observed variables that may or may not be included in a predictive model. A dashed circle indicates an unobserved variable, such as in the case of measurement error in which the true label  $\tilde{Y}$  is a parent of the observed label  $Y$ . A rectangle indicates an observed variable that is conditioned on in the data-generating process, typically denoted by  $S$  for a selection effect, which induces an association between its parents. For clarity, we decompose  $X$  into  $X_{A}^{\perp}$ , the component that is not causally affected by  $A$ , and  $X_{Y}^{\perp}$ , the component that does not causally affect  $Y$  (in a causal-direction graph, i.e.,  $X$  affects  $Y$ ) or is not causally affected by  $Y$  (in an anti-causal graph, i.e.,  $Y$  affects  $X$ ). In general, there could be a third component that is causally connected to both  $A$  and  $Y$  (i.e., a non-spurious interaction effect). This third component is excluded through the assumption that the association between  $Y$  and  $A$  in the training distribution is purely spurious [47].
Definition 5. (Purely spurious). We say the association between  $Y$  and  $A$  is purely spurious if  $Y \perp X \mid X_A^\perp, A$ .
In other words, if we condition on  $A$ , then the only information about  $Y$  is in  $X_A^\perp$  (i.e., the component of the input that is not causally affected by the protected class). We also restrict ourselves to cases in which  $A$  is exogenous to the causal structure (i.e., there is no confounder of  $A$  and  $X$  or of  $A$  and  $Y$ ), which seems reasonable in the case of fairness because the protected class is typically not caused by other variables in the specified model.
Despite the intuitive appeal and legal precedent for counterfactual fairness, the determination of counterfactuals is fundamentally contentious because, by definition, we do not observe counterfactual worlds, and we cannot run scientific experiments to test what would have happened in a different world history. Some counterfactuals are relatively clear, but counterfactuals in contexts of protected classes such as race, gender, and disability can be ambiguous. Consider Bob, a hypothetical Black criminal defendant being assessed for recidivism who is determined to be high-risk by a human judge or a machine algorithm. The assessment involves extensive information about his life: where he has lived, how he has been employed, what crimes he has been arrested and convicted of before, and so on. What is the counterfactual in which Bob is White? Is it the world in which Bob was born to White parents, the one in which Bob's parents were themselves born to White parents, or another change further back? Would Bob still have the same educational and economic circumstances? Would those White parents have raised Bob in the same majority-Black neighborhood he grew up in or in a majority-White neighborhood? Questions of counterfactual ambiguity do not have established answers in the fairness literature, either in philosophy [35], epidemiology [46], or the nascent machine learning literature [28], and we do not attempt to resolve them in the present work.
Additionally, defining counterfactual fairness in a given context requires some identification of the causal structure of the data generating process. Fortunately, mitigating this challenge is more technically tractable than mitigating ambiguity, as there is an extensive literature on how we can learn about the causal structure, known as as causal inference [15] or causal discovery when we have minimal prior knowledge or assumptions of which parent-child relationships do and do not obtain [43]. The literature also contains a number of methods for assessing counterfactual fairness given a partial or complete causal graph [9, 20, 51, 52, 57]. Again, we do not attempt to resolve debates about appropriate causal inference strategies in the present work, but merely to provide a conceptual tool for those researchers and practitioners who are comfortable staking a claim of some partial or complete causal structure of the context at hand.
To ground our technical results, we briefly sketch three fairness contexts that match those in Figure 2 and will be the basis of Corollary 2.2. First, measurement error [6] is a well-known source of fairness issues as developed in the "measurement error models" of Jacobs and Wallach [26]. Suppose that COMPAS is assessing the risk of recidivism, but they do not have perfect knowledge of who has committed crimes because not all crimes are reported and prosecuted. Thus,  $X$  causes  $\tilde{Y}$ , the actual committing of a crime, which is one cause of  $Y$ , the imperfect labels. Also suppose that the protected class  $A$  affects whether the crime is reported and prosecuted, such as through police bias, but does not affect the actual committing of a crime.  $A$  also affects  $X$ , other data used for the recidivism
prediction. This is represented by the causal DAG in Figure 2a, which connects counterfactual fairness to demographic parity.
Second, a protected class can affect whether individuals are selected into the dataset. For example, drugs may be prescribed on the basis of whether the individual's lab work  $X$  indicates the presence of disease  $Y$ . People with the disease are presumably more likely to have lab work done, and the socioeconomic status  $A$  of the person (or of the hospital where their lab work is done) may make them more likely to have lab work done. We represent this with a selection variable  $S$  that we condition on by only using the lab work data that is available. This collider induces an association between  $A$  and  $Y$ . This is represented by the causal DAG in Figure 2b, which connects counterfactual fairness to equalized odds.
Third, individuals may be selected into the dataset based on predictors,  $X$ , which cause the label  $Y$ . For example, loans may be given out with a prediction of loan repayment  $Y$  based on demographic and financial records  $X$ . There may be data only for people who choose to work with a certain bank based on financial records and a protected class  $A$ . This is represented by the causal DAG in Figure 2c, which connects counterfactual fairness to calibration.
4 Counterfactual fairness and robust prediction
Characterizations of the trade-off between prediction accuracy and fairness treat them as two competing goals [e.g., 11, 12, 21, 55]. That view assumes the task is to find an optimal model  $f^{*}(X)$  that minimizes risk (i.e., expected loss) in the training distribution  $X, Y, A \sim P$ :
$$
f ^ {*} (X) = \operatorname * {a r g m i n} _ {f} \mathbb {E} _ {P} [ \ell (f (X), Y) ]
$$
However, the discrepancy between levels of the protected class in the training data may itself be due to biases in the dataset, such as measurement error, selection on label, and selection on predictors. As such, the practical interest may not be risk minimization in the training distribution, but out-of-distribution (OOD) generalization from the training distribution to an unbiased target distribution where these effects are not present.
Whether the counterfactually fair empirical risk minimizer also minimizes risk in the target distribution depends on how the distributions differ. Because a counterfactually fair predictor does not depend on the protected class, it minimizes risk if the protected class in the target distribution contains no information about the corresponding labels (i.e., if protected class and label are uncorrelated). If the protected class and label are correlated in the target distribution, then risk depends on whether the counterfactually fair predictor learns all the information about the label contained in the protected class. If so, then its prediction has no reason to vary in the counterfactual scenario. Theorem 1 motivates counterfactual fairness by stating that, for a distribution with bias due to selection on label and equal marginal label distributions or due to selection on predictors, the counterfactually fair predictor is accuracy-optimal in the unbiased target distribution.
Theorem 1. Let  $\mathcal{F}^{CF}$  be the set of all counterfactually fair predictors. Let  $\ell$  be a proper scoring rule (e.g., square error, cross entropy loss). Let the counterfactually fair predictor that minimizes risk on the training distribution  $X,Y,A\sim P$  be:
$$
f^{*}(X):= \operatorname *{argmin}_{f\in \mathcal{F}^{CF}}\mathbb{E}_{P}\bigl[\ell (f(X),Y)\bigr]
$$
Then,  $f^*$  also minimizes risk on the target distribution  $X, Y, A \sim Q$  with no selection effects, i.e.,
$$
f ^ {*} (X) = \operatorname * {a r g m i n} _ {f} \mathbb {E} _ {Q} [ \ell (f (X), Y) ]
$$
if either of the following conditions hold:
1. The association between  $Y$  and  $A$  is due to selection on label and the marginal distribution of the label  $Y$  is the same in each distribution, i.e.,  $P(Y) = Q(Y)$ .
2. The association between  $Y$  and  $A$  is due to selection on predictors.
In the context of measurement error, there are not straightforward conditions for the risk minimization of the counterfactually fair predictor because the training dataset contains  $Y$ , observed noisy labels,
and not  $\tilde{Y}$ , the true unobserved labels. Thus any risk minimization (including in the training distribution) depends on the causal process that generates  $Y$  from  $\tilde{Y}$ .
5 Counterfactual fairness and group fairness