SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As society transitions towards an AI-based decision-making infrastructure, an ever-increasing number of decisions once under control of humans are now delegated to automated systems. Even though such developments make various parts of society more efficient, a large body of evidence suggests that a great deal of care needs to be taken to make such automated decision-making systems fair and equitable, namely, taking into account sensitive attributes such as gender, race, and religion. In this paper, we study a specific decision-making task called outcome control in which an automated system aims to optimize an outcome variable  $Y$  while being fair and equitable. The interest in such a setting ranges from interventions related to criminal justice and welfare, all the way to clinical decision-making and public health. In this paper, we first analyze through causal lenses the notion of benefit, which captures how much a specific individual would benefit from a positive decision, counterfactually speaking, when contrasted with an alternative, negative one. We introduce the notion of benefit fairness, which can be seen as the minimal fairness requirement in decision-making, and develop an algorithm for satisfying it. We then note that the benefit itself may be influenced by the protected attribute, and propose causal tools which can be used to analyze this. Finally, if some of the variations of the protected attribute in the benefit are considered as discriminatory, the notion of benefit fairness may need to be strengthened, which leads us to articulating a notion of causal benefit fairness. Using this notion, we develop a new optimization procedure capable of maximizing  $Y$  while ascertaining causal fairness in the decision process.
Decision-making systems based on artificial intelligence and machine learning are being increasingly deployed in real-world settings where they have life-changing consequences on individuals and on society more broadly, including hiring decisions, university admissions, law enforcement, credit lending, health care access, and finance [19, 26, 6]. Issues of unfairness and discrimination are pervasive in those settings when decisions are being made by humans, and remain (or are potentially amplified) when decisions are made using machines with little transparency or accountability. Examples include reports on such biases in decision support systems for sentencing [1], face-detection [7], online advertising [37, 10], and authentication [35]. A large part of the underlying issue is that AI systems designed to make decisions are trained with data that contains various historical biases and past discriminatory decisions against certain protected groups, and such systems may potentially lead to an even more discriminatory process, unless they have a degree of fairness and transparency.
In this paper, we focus on the specific task of outcome control, characterized by a decision  $D$  which precedes the outcome of interest  $Y$ . The setting of outcome control appears across a broad range of applications, from clinical decision-making [13] and public health [15], to criminal justice [23] and various welfare interventions [9]. In general, outcome control will cover settings in which an
institution (such as a hospital or a social service) may attempt to maximize an outcome (such as survival or well-being) using a known control (such as surgery or a welfare intervention)<sup>1</sup>. Importantly, due to historical biases, certain demographic groups may differ in their distribution of covariates and therefore their benefit from treatment (e.g., from surgery or intervention) may be lower than expected, leading to a lower allocation of overall resources. We next discuss two lines of related literature.
Firstly, a large body of literature in reinforcement learning [36, 38] and policy learning [11, 33, 21, 18, 2] analyzes the task of optimal decision-making. Often, these works consider the conditional average treatment effect (CATE) that measures how much probabilistic gain there is from a positive versus a negative decision for a specific group of individuals when experimental data is available. Subsequent policy decisions are then based on the CATE, a quantity that will be important in our approach as well. The focus of this literature is often on developing efficient procedures with desirable statistical properties, and issues of fairness have not traditionally been explored in this context.
On the other hand, there is also a growing literature in fair machine learning, that includes various different settings. One can distinguish three specific and different tasks, namely (1) bias detection and quantification for currently deployed policies; (2) construction of fair predictions of an outcome; (3) construction of fair decision-making policies. Most of the work in fair ML falls under tasks (1) and (2), whereas our setting of outcome control falls under (3). Our work also falls under the growing literature that explores fairness through a causal lens [22, 20, 28, 44, 43, 41, 8, 32].
For concreteness, consider the causal diagram in Fig. 1 that represents the setting of outcome control, with  $X$  the protected attribute,  $Z$  a possibly multidimensional set of confounders,  $W$  a set of mediators. Decision  $D$  is based on the variables  $X, Z$ , and  $W$ , and the outcome  $Y$  depends on all other variables in the model. In this setting, we also assume that the decision-maker is operating under budget constraints.
Previous work introduces a fairness definition that conditions on the potential outcomes of the decision  $D$ , written  $Y_{d_0}, Y_{d_1}$ , and ensures that the decision  $D$  is independent of the protected attribute  $X$  for any fixed value of  $Y_{d_0}, Y_{d_1}$  [14]. Another related work, in a slightly
different setting of risk assessment [9], proposes conditioning on the potential outcome under a negative decision,  $Y_{d_0}$ , and focuses on equalizing counterfactual error rates. Finally, the work [25] is also related to our setting, and proposes that fairness should be assessed with respect to three causal effects of interest: (i) total effect  $X \rightarrow Y$ ; (ii) total effect  $D \rightarrow Y$ ; and (iii) total effect  $X \rightarrow D$ . Our framework, however, integrates considerations of the above effects more coherently, and brings forward a new definition of fairness based on first principles.
Figure 1: Standard Fairness Model (SFM) [31] extended for outcome control.
However, the causal approaches mentioned above take a different perspective from the policy learning literature, in which policies are built based on the CATE of the decision  $D$ , written  $\mathbb{E}[Y_{d_1} - Y_{d_0} \mid x, z, w]$ , which we will refer to as benefit $^2$ . Focusing exclusively on the benefit, though, will provide no fairness guarantees apriori. In particular, as can be seen from Fig. 1, the protected attribute  $X$  may influence the effect of  $D$  on  $Y$  in three very different ways: (i) along the direct pathway  $X \rightarrow Y$ ; (ii) along the indirect pathway  $X \rightarrow W \rightarrow Y$ ; (iii) along the spurious pathway  $X \leftarrow Z \rightarrow Y$ . Often, the decision-maker may view these causal effects differently, and may consider only some of them as discriminatory. Currently, no approach in the literature allows for a principled way of detecting and removing discrimination based on the notion of benefit, while accounting for different underlying causal mechanisms that may lead to disparities.
In light of the above, the goal of this paper is to analyze the foundations of outcome control from a causal perspective of the decision-maker. Specifically, we develop a causal-based decision-making framework for modeling fairness with the following contributions:
(i) We introduce Benefit Fairness (BF, Def. 2) to ensure that at equal levels of the benefit, the protected attribute  $X$  is independent of the decision  $D$ . We then develop an algorithm for achieving BF (Alg. 1) and prove optimality guarantees (Thm. 2),
(ii) We develop Alg. 2 that determines which causal mechanisms from  $X$  to the benefit (direct, indirect, spurious) explain the difference in the benefit between groups. The decision-maker can then decide which causal pathways are considered as discriminatory.
<sup>1</sup>Our setting is not related to the concept of outcome control in the procedural justice literature [24].
2We remark that the notion of benefit discussed in this work is different from that in [34].
(iii) We define the notion of causal benefit fairness (CBF, Def. 3), which models discriminatory pathways in a fine-grained way. We further develop an algorithm (Alg. 3) that ensures the removal of such discriminatory effects, and show some theoretical guarantees (Thm. 4).
1.1 Preliminaries
We use the language of structural causal models (SCMs) as our basic semantical framework [30]. A structural causal model (SCM) is a tuple  $\mathcal{M} \coloneqq \langle V, U, \mathcal{F}, P(u) \rangle$ , where  $V, U$  are sets of endogenous (observable) and exogenous (latent) variables respectively,  $\mathcal{F}$  is a set of functions  $f_{V_i}$ , one for each  $V_i \in V$ , where  $V_i \gets f_{V_i}(\mathrm{pa}(V_i), U_{V_i})$  for some  $\mathrm{pa}(V_i) \subseteq V$  and  $U_{V_i} \subseteq U$ .  $P(u)$  is a strictly positive probability measure over  $U$ . Each SCM  $\mathcal{M}$  is associated to a causal diagram  $\mathcal{G}$  over the node set  $V$ , where  $V_i \to V_j$  if  $V_i$  is an argument of  $f_{V_j}$ , and  $V_i \gets V_j$  if the corresponding  $U_{V_i}, U_{V_j}$  are not independent. Throughout, our discussion will be based on the specific causal diagram known as the standard fairness model (SFM, see Fig. 1). Further, an instantiation of the exogenous variables  $U = u$  is called a unit. By  $Y_x(u)$  we denote the potential response of  $Y$  when setting  $X = x$  for the unit  $u$ , which is the solution for  $Y(u)$  to the set of equations obtained by evaluating the unit  $u$  in the submodel  $\mathcal{M}_x$ , in which all equations in  $\mathcal{F}$  associated with  $X$  are replaced by  $X = x$ .
2 Foundations of Outcome Control
In the setting of outcome control, we are interested in the following decision-making task:
Definition 1 (Decision-Making Optimization). Let  $\mathcal{M}$  be an SCM compatible with the SFM. We define the optimal decision problem as finding the (possibly stochastic) solution to the following optimization problem given a fixed budget  $b$ :
$$
D ^ {*} = \underset {D (x, z, w)} {\arg \max } \quad \mathbb {E} \left[ Y _ {D} \right] \tag {1}
$$
$$
\text {s u b j e c t} \quad P (d) \leq b. \tag {2}
$$
The budget  $b$  constraint is relevant for scenarios when resources are scarce, in which not all patients possibly requiring treatment can be given treatment. In such a setting, the goal is to treat patients who are most likely to benefit from the treatment, as formalized later in the text. We next discuss two different perspectives on solving the above problem. First, we discuss the problem solution under perfect knowledge, assuming that the underlying SCM and the unobserved variables are available to us (we call this the oracle's perspective). Then, we move on to solving the problem from the point of view of the decision-maker, who only has access to the observed variables in the model and a dataset generated from the true SCM.
2.1 Oracle's Perspective
The following example, which will be used throughout the paper, is accompanied by a vignette that performs inference using finite sample data for the different computations described in the sequel. We introduce the example by illustrating the intuition of outcome control through the perspective of an all-knowing oracle:
Example (Cancer Surgery - continued). A clinical team has access to information about the sex of cancer patients ( $X = x_0$  male,  $X = x_1$  female) and their degree of illness severity determined from tissue biopsy ( $W \in [0,1]$ ). They wish to optimize the 2-year survival of each patient ( $Y$ ), and the decision  $D = 1$  indicates whether to perform surgery. The following SCM  $\mathcal{M}^*$  describes the data-generating mechanisms (unknown to the team):
$$
\mathcal {F} ^ {*}, P ^ {*} (U): \left\{ \begin{array}{l} X \leftarrow U _ {X} \\ W \leftarrow \left\{ \begin{array}{l} \sqrt {U _ {W}} i f X = x _ {0}, \\ 1 - \sqrt {1 - U _ {W}} i f X = x _ {1} \end{array} \right. \\ D \leftarrow f _ {D} (X, W) \\ Y \leftarrow \mathbb {1} \left(U _ {Y} + \frac {1}{3} W D - \frac {1}{5} W > 0. 5\right). \\ U _ {X} \in \{0, 1 \}, P \left(U _ {X} = 1\right) = 0. 5, \\ U _ {W}, U _ {Y} \sim \operatorname {U n i f} [ 0, 1 ], \end{array} \right. \tag {3}
$$
(a) Units in the oracle example.
(b) Causal diagram for the running example.
Figure 2: (a) Units in the Oracle example; (b) Standard Fairness Model (SFM) for the cancer surgery example, corresponding to  $W = \{W\}$  and  $Z = \emptyset$  in the SFM in Fig. 1.
where the  $f_{D}$  mechanism is constructed by the team.
The clinical team has access to an oracle that is capable of predicting the future perfectly. In particular, the oracle tells the team how each individual would respond to surgery. That is, for each unit  $U = u$  (of the 500 units), the oracle returns the values of
$$
Y _ {d _ {0}} (u), Y _ {d _ {1}} (u). \tag {9}
$$
Having access to this information, the clinicians quickly realize how to use their resources. In particular, they notice that for units for whom  $(Y_{d_0}(u), Y_{d_1}(u))$  equals  $(0, 0)$  or  $(1, 1)$ , there is no effect of surgery, since they will (or will not) survive regardless of the decision. They also notice that surgery is harmful for individuals for whom  $(Y_{d_0}(u), Y_{d_1}(u)) = (1, 0)$ . These individuals would not survive if given surgery, but would survive otherwise. Therefore, they ultimately decide to treat 100 individuals who satisfy
$$
\left(Y _ {d _ {0}} (u), Y _ {d _ {1}} (u)\right) = (0, 1), \tag {10}
$$
since these individuals are precisely those whose death can be prevented by surgery. They learn there are 100 males and 100 females in the  $(0,1)$ -group, and thus, to be fair with respect to sex, they decide to treat 50 males and 50 females.
The space of units corresponding to the above example is represented in Fig. 2a. The groups described by different values of  $Y_{d_0}(u)$ ,  $Y_{d_1}(u)$  in the example are known as canonical types [3] or principal strata [12]. Two groups cannot be influenced by the treatment decision (which will be called "Safe" and "Doomed", see Fig. 2a). The third group represents those who are harmed by treatment (called "Harmed"). In fact, the decision to perform surgery for this subset of individuals is harmful. Finally, the last group represents exactly those for whom the surgery is life-saving, which is the main goal of the clinicians (this group is called "Helped").
This example illustrates how, in presence of perfect knowledge, the team can allocate resources efficiently. In particular, the consideration of fairness comes into play when deciding which of the individuals corresponding to the  $(0,1)$  principal stratum will be treated. Since the number of males and females in this group is equal, the team decides that half of those treated should be female. The approach described above can be seen as appealing in many applications unrelated to the medical setting, and motivated the definition of principal fairness [14]. As we show next, however, this viewpoint is often incompatible with the decision-maker's perspective.
2.2 Decision-Maker's Perspective
We next discuss the policy construction from the perspective of the decision-maker:
Example (Cancer Surgery - continued). The team of clinicians constructs the causal diagram associated with the decision-making process, shown in Fig. 2b. Using data from their electronic health records (EHR), they estimate the benefit of the treatment based on  $x$ ,  $w$ :
$$
\mathbb {E} \left[ Y _ {d _ {1}} - Y _ {d _ {0}} \mid w, x _ {1} \right] = \mathbb {E} \left[ Y _ {d _ {1}} - Y _ {d _ {0}} \mid w, x _ {0} \right] = \frac {w}{3}. \tag {11}
$$
In words, at each level of illness severity  $W = w$ , the proportion of patients who benefit from the surgery is the same, regardless of sex. In light of this information, the clinicians decide to construct
Figure 3: Difference in perspective between perfect knowledge of an oracle (left) and imperfect knowledge of a decision-maker in a practical application (right).
the decision policy  $f_{D}$  such that  $f_{D}(W) = \mathbb{1}(W > \frac{1}{2})$ . In words, if a patient's illness severity  $W$  is above  $\frac{1}{2}$ , the patient will receive treatment.
After implementing the policy and waiting for the 2-year follow-up period, clinicians estimate the probabilities of treatment within the stratum of those helped by surgery, and compute that
$$
P (d \mid y _ {d _ {0}} = 0, y _ {d _ {1}} = 1, x _ {0}) = \frac {7}{8}, \tag {12}
$$
$$
P \left(d \mid y _ {d _ {0}} = 0, y _ {d _ {1}} = 1, x _ {1}\right) = \frac {1}{2}, \tag {13}
$$
indicating that the allocation of the decision is not independent of sex. That is, within the group of those who are helped by surgery, males are more likely to be selected for treatment than females.
The decision-making policy introduced by the clinicians, somewhat counter-intuitively, does not allocate the treatment equally within the principal stratum of those who are helped, even though at each level of illness severity, the proportion of patients who benefit from the treatment is equal between the sexes. What is the issue at hand here?
To answer this question, the perspective under perfect knowledge is shown in Fig. 3a. In particular, on the horizontal axis the noise variable  $u_y$  is available, which summarizes the patients' unobserved resilience. Together with the value of illness severity (on the vertical axis) and the knowledge of the structural causal model, we can perfectly pick apart the different groups (i.e., principal strata) according to their potential outcomes (groups are indicated by color). In this case, it is clear that our policy should treat patients in the green area since those are the ones who benefit from treatment.
In Fig. 3b, however, we see the perspective of the decision-makers under imperfect knowledge. Firstly, the decision-makers have no knowledge about the values on the horizontal axis, since this represents variables that are outside their model. From their point of view, the key quantity of interest is the conditional average treatment effect (CATE), which we call benefit in this context<sup>3</sup>, defined as
$$
\Delta (x, z, w) = P \left(y _ {d _ {1}} \mid x, z, w\right) - P \left(y _ {d _ {0}} \mid x, z, w\right). \tag {14}
$$
Importantly, in our setting, the benefit  $\Delta$  can be uniquely computed (identified) from observational data, implying that we may use the notion of benefit for practical data analyses:
Proposition 1 (Benefit Identifiability). Let  $\mathcal{M}$  be an SCM compatible with the SFM in Fig. 1. Then the benefit  $\Delta(x, z, w)$  is identifiable from observational data and equals:
$$
\Delta (x, z, w) = P (y \mid d _ {1}, x, z, w) - P (y \mid d _ {0}, x, z, w) \tag {15}
$$
for any choice of  $x, z, w$ .
In practice, however, clinical decision-making may not be based on the actual value of the benefit  $\Delta$  but a biased, imperfect version of it, corresponding to the intuition of the clinician.
Algorithm 1 Decision-Making with Benefit Fairness
- Inputs: Distribution  $P(V)$ , Budget  $b$
1: Compute  $\Delta (x,z,w) = \mathbb{E}[Y_{d_1} - Y_{d_0}\mid x,z,w]$  for all  $(x,z,w)$ .
2: If  $P(\Delta > 0) \leq b$ , set  $D^{*} = \mathbb{1}(\Delta(x, z, w) > 0)$  and  $\text{RETURN}(D^{*})$ .
3: Find  $\delta_b > 0$  such that
$$
P (\Delta > \delta_ {b}) \leq b, P (\Delta \geq \delta_ {b}) > b. \tag {17}
$$
4: Otherwise, define
$$
\mathcal {I} := \{(x, z, w): \Delta (x, z, w) > \delta_ {b} \}, \tag {18}
$$
$$
\mathcal {B} := \{(x, z, w): \Delta (x, z, w) = \delta_ {b} \} \tag {19}
$$
5: Construct the policy  $D^{*}$  such that:
$$
D ^ {*} := \left\{ \begin{array}{l} 1 \text {f o r} (x, z, w) \in \mathcal {I}, \\ 1 \text {w i t h p r o b .} \frac {b - P (\mathcal {I})}{P (\mathcal {B})} \text {f o r} (x _ {0}, z, w) \in \mathcal {B}, \\ 1 \text {w i t h p r o b .} \frac {b - P (\mathcal {I})}{P (\mathcal {B})} \text {f o r} (x _ {1}, z, w) \in \mathcal {B}. \end{array} \right. \tag {20}
$$
The benefit is simply the increase in survival associated with treatment. After computing the benefit (shown in Eq. 11), the decision-makers visualize the male and female groups (lighter color indicates a larger increase in survival associated with surgery) according to it. It is visible from the figure that the estimated benefit from surgery is higher for the  $x_0$  group than for  $x_1$ . Therefore, to the best of their knowledge, the decision-makers decide to treat more patients from the  $x_0$  group.
The example illustrates why the oracle's perspective may be misleading for the decision-makers. The clinicians can never determine exactly which patients belong to the group
$$
Y _ {d _ {0}} (u) = 0, Y _ {d _ {1}} (u) = 1,
$$
that is, who benefits from the treatment. Instead, they have to rely on illness severity  $(W)$  as a proxy for treatment benefit  $(\Delta)$ . In other words, our understanding of treatment benefit will always be probabilistic, and we need to account for this when considering the decision-maker's task. A further discussion on the relation to principal fairness [14] is given in Appendix A.
2.3 Benefit Fairness
To remedy the above issue, we propose an alternative definition, which takes the viewpoint of the decision-maker:
Definition 2 (Benefit Fairness). Let  $\Delta$  denote the benefit of treatment in Eq. 14 and let  $\delta$  be a fixed value of  $\Delta$ . We say that the pair  $(Y, D)$  satisfies benefit fairness (BF, for short) if
$$
P (d \mid \Delta = \delta , x _ {0}) = P (d \mid \Delta = \delta , x _ {1}) \forall \delta . \tag {16}
$$
The above definition pertains to the population-level setting, but without oracle knowledge. In this setting, the objective function in Eq. 1 is expected to be lower than in the oracle case. Further, in finite samples, the constraint from Def. 2 is expected to hold approximately. Notice that BF takes the perspective of the decision-maker who only has access to the unit's attributes  $(x,z,w)$ , as opposed to the exogenous  $U = u^4$ . In particular, the benefit  $\Delta (x,z,w)$  is estimable from the data, i.e., attainable by the decision-maker. BF then requires that at each level of the benefit,  $\Delta (x,z,w) = \delta$ , the rate of the decision does not depend on the protected attribute. The benefit  $\Delta$  is closely related to the canonical types discussed earlier, namely
$$
\Delta (x, z, w) = c (x, z, w) - d (x, z, w). \tag {21}
$$
The values of  $c, d$  in Eq. 21 are covariate-specific, and indicate the proportions of patients helped and harmed by the treatment, respectively, among all patients coinciding with covariate values  $x, z, w$  (i.e.,
Formally, having access to the exogenous instantiation of  $U = u$  implies knowing which principal stratum from Fig. 2a the unit belongs to, since  $U = u$  determines all of the variations of the model.
Algorithm 2 Benefit Fairness Causal Explanation
- Inputs: Distribution  $P\left( V\right)$  ,Benefit  $\Delta \left( {x,z,w}\right)$  ,Decision policy  $D$
1: Compute the causal decomposition of the resource allocation disparity into its direct, indirect, and spurious contributions [44, 31]
$$
P (d \mid x _ {1}) - P (d \mid x _ {0}) = \mathrm {D E} + \mathrm {I E} + \mathrm {S E}. \tag {22}
$$
2: Compare the distributions  $P(\Delta \mid x_1)$  and  $P(\Delta \mid x_0)$ .
3: Compute the causal decomposition of the benefit disparity
$$
\mathbb {E} (\Delta \mid x _ {1}) - \mathbb {E} (\Delta \mid x _ {0}) = \mathbf {D E} + \mathbf {I E} + \mathbf {S E}. \tag {23}
$$
4: Compute the counterfactual distribution  $P(\Delta_C \mid x)$  for specific interventions  $C$  that remove the direct, indirect, or total effect of  $X$  on the benefit  $\Delta$ .