SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Loukas Kavouras
IMSI / Athena RC
kavouras@athenarc.gr
Konstantinos Tsopelas
k.tsopelas@athenarc.gr
Giorgos Giannopoulos
giann@athenarc.gr
Dimitris Sacharidis
ULB
dimitris.sacharidis@ulb.be
Eleni Psaroudaki
NTUA & IMSI / Athena RC
epsaroudaki@mail.ntua.gr
Nikolaos Theologitis
n.theologitis@athenarc.gr
Dimitrios Rontogiannis
dronto@gmail.com
Dimitris Fotakis
NTUA & Archimedes / Athena RC
fotakis@cs.ntua.gr
Ioannis Emiris
NKUA & IMSI / Athena RC
emiris@athenarc.gr
In this work, we present Fairness Aware Counterfactuals for Subgroups (FACTS), a framework for auditing subgroup fairness through counterfactual explanations. We start with revisiting (and generalizing) existing notions and introducing new, more refined notions of subgroup fairness. We aim to (a) formulate different aspects of the difficulty of individuals in certain subgroups to achieve recourse, i.e., receive the desired outcome, either at the micro level, considering members of the subgroup individually, or at the macro level, considering the subgroup as a whole, and (b) introduce notions of subgroup fairness that are robust, if not totally oblivious, to the cost of achieving recourse. We accompany these notions with an efficient, model-agnostic, highly parameterizable, and explainable framework for evaluating subgroup fairness. We demonstrate the advantages, the wide applicability, and the efficiency of our approach through a thorough experimental evaluation of different benchmark datasets.
Machine learning is now an integral part of decision-making processes across various domains, e.g., medical applications [22], employment [5, 7], recommender systems [29], education [26], credit assessment [4]. Its decisions affect our everyday life directly, and, if unjust or discriminative, could potentially harm our society [28]. Multiple examples of discrimination or bias towards specific population subgroups in such applications [33] create the need not only for explainable and interpretable machine learning that is more trustworthy [6], but also for auditing models in order to detect hidden bias for subgroups [30].
Bias towards protected subgroups is most often detected by various notions of fairness of prediction, e.g., statistical parity, where all subgroups defined by a protected attribute should have the same probability of being assigned the positive (favorable) predicted class. These definitions capture the explicit bias reflected in the model's predictions. Nevertheless, an implicit form of bias is the
difficulty for, or the burden [32, 20] of, an individual (or a group thereof) to achieve recourse, i.e., perform the necessary actions to change their features so as to obtain the favorable outcome [10, 35]. Recourse provides explainability (i.e., a counterfactual explanation [38]) and actionability to an affected individual, and is a legal necessity in various domains, e.g., the Equal Credit Opportunity Act mandates that an individual can demand to learn the reasons for a loan denial. *Fairness of recourse* captures the notion that the protected subgroups should bear equal burden [10, 14, 37, 20].
To illustrate these notions, consider a company that supports its promotion decisions with an AI system that classifies employees as good candidates for promotion, the favorable positive class, based on various performance metrics, including their cycle time efficiency (CTE) and the annual contract value (ACV) for the projects they lead. Figure 1a draws ten employees from the negative predicted class as points in the CTE-ACV plane and also depicts the decision boundary of the classifier. Race is the protected attribute, and there are two protected subgroups with five employees each, depicted as circles and triangles. For each employee, the arrow depicts the best action to achieve recourse, i.e., to cross the decision boundary, and the number indicates the cost of the action, here simply computed as the distance to the boundary [10]. For example,  $x_{1}$  may increase their chances for promotion mostly by acquiring more high-valued projects, while  $x_{2}$  mostly by increasing their efficiency. Burden is defined as the mean cost for a protected subgroup [32]. For the protected race 0, the burden is 2, while it is 2.2 for race 1, indicating thus unfairness of recourse against race 1. In contrast, assuming there is an equal number of employees of each race in the company, the classifier satisfies fairness of prediction in terms of statistical parity (equal positive rate in the subgroups).
(b)
Figure 1: (a) An example of affected individuals, the decision boundary, actions, and a subpopulation (in the shaded region), depicted in the feature space; (b) Cumulative distribution of cost of recourse for the individuals in (a); (c) Comparison of two actions to achieve recourse for two individuals.
While fairness of recourse is an important concept that captures a distinct notion of algorithmic bias, as also explained in [14, 37], we argue that it is much more nuanced than the mean cost of recourse (aka burden) considered in all prior work [10, 32, 37, 20], and raise three issues.
First, the mean cost, which comprises a micro viewpoint of the problem, does not provide the complete picture of how the cost of recourse varies among individuals and may lead to contrasting conclusions. Consider again Figure 1a, and observe that for race 1 all but one employee can achieve recourse with cost at most 2, while an outlier achieves recourse with cost 6. It is this outlier that raises the mean cost for race 1 above that of race 0. For the two race subgroups, Figure 1b shows the cumulative distribution of cost, termed the effectiveness-cost distribution (ecd) in Section 2. These distributions allow the fairness auditor to inspect the tradeoff between cost and recourse, and define the appropriate fairness of recourse notion. For example, they may consider that actions with a cost of more than 2 are unrealistic (e.g., because they cannot be realized within some timeframe), and thus investigate how many employees can achieve recourse under this constraint; we refer to this as equal effectiveness within budget in Section 2.3. Under this notion, there is unfairness against race 0, as only  $60\%$  of race 0 employees (compared to  $80\%$  of race 1) can realistically achieve recourse.
There are several options to go beyond the mean cost. One is to consider fairness of recourse at the individual level, and compare an individual with their counterfactual counterpart had their protected attribute changed value [37]. However, this approach is impractical as, similar to other causal-based definitions of fairness, e.g., [21], it requires strong assumptions about the causal structure in the
domain [15]. In contrast, we argue that it's preferable to investigate fairness in subpopulations and inspect the trade-off between cost and recourse.
Second, there are many cases where the aforementioned micro-level aggregation of individuals' costs is not meaningful in auditing real-world systems. To account for this, we introduce a macro viewpoint where a group of individuals is considered as a whole, and an action is applied to and assessed collectively for all individuals in the group. An action represents an external horizontal intervention, such as an affirmative action in society, or focused measures in an organization that would change the attributes of some subpopulation (e.g., decrease tax or loan interest rates, increase productivity skills). In the macro viewpoint, the cost of recourse does not burden the individuals, but the external third party, e.g., the society or an organization. Moreover, the macro viewpoint offers a more intuitive way to audit a system for fairness of recourse, as it seeks to uncover systemic biases that apply to a large number of individuals.
To illustrate the macro viewpoint, consider the group within the shaded region in Figure 1a. In the micro viewpoint, each employee seeks recourse individually, and both race subgroups have the same distribution of costs. However, we can observe that race 0 employees, like  $x_{1}$  achieve recourse by actions in the ACV direction, while race 1 employees, like  $x_{2}$  in the orthogonal CTE direction. This becomes apparent when we take the macro viewpoint, and investigate the effect of action  $a_{1}$ , depicted on the border of the shaded region, discovering that it is disproportionally effective on the race subgroups (leads to recourse for two-thirds of one subgroup but for none in the other). In this example, action  $a_{1}$  might represent the effect of a training program to enhance productivity skills, and the macro viewpoint finds that it would perpetuate the existing burden of race 0 employees.
Third, existing notions of fairness of recourse have an important practical limitation: they require a cost function that captures one's ability to modify one's attributes, whose definition may involve a learning process [31], or an adaptation of off-the-shelf functions by practitioners [35]. Even the idea of which attributes are actionable, in the sense that one can change (e.g., one cannot get younger), or "ethical" to suggest as actionable (e.g., change of marital status from married to divorced) hides many complications [36].
Conclusions drawn about fairness of recourse crucially depend on the cost definition. Consider individuals  $x_{1}, x_{2}$ , and actions  $a_{1}, a_{2}$ , shown in Figure 1c. Observe that is hard to say which action is cheaper, as one needs to compare changes within and across very dissimilar attributes. Suppose the cost function indicates action  $a_{1}$  is cheaper; both individuals achieve recourse with the same cost. However, if action  $a_{2}$  is cheaper, only  $x_{2}$  achieves recourse. Is the classifier fair?
To address this limitation, we propose definitions that are oblivious to the cost function. The idea is to compare the effectiveness of actions to the protected subgroups, rather than the cost of recourse for the subgroups. One way to define a cost-oblivious notion of fairness of recourse is the equal choice for recourse (see Section 2.3), which we illustrate using the example in Figure 1c. According to it, the classifier is unfair against  $x_{1}$ , as  $x_{1}$  has only one option, while  $x_{2}$  has two options to achieve recourse among the set of actions  $\{a_{1}, a_{2}\}$ .
Contribution Our aim is to showcase that fairness of recourse is an important and distinct notion of algorithmic fairness with several facets not previously explored. We make a series of conceptual and technical contributions.
Conceptually, we distinguish between two different viewpoints. The micro viewpoint follows literature on recourse fairness [10, 14, 37, 20] in that each individual chooses the action that is cheaper for them, but revisits existing notions. It considers the trade-off between cost and recourse and defines several novel notions that capture different aspects of it. The macro viewpoint considers how an action collectively affects a group of individuals, and quantifies its effectiveness. It allows the formulation of cost-oblivious notions of recourse fairness. It also leads to an alternative trade-off between cost and recourse that may reveal systemic forms of bias.
Technically, we propose an efficient, interpretable, model-agnostic, highly parametrizable framework termed FACTS (Fairness-Aware Countefactuals for Subgroups) to audit for fairness of recourse. FACTS is efficient in computing the effectiveness-cost distribution, which captures the trade-off between cost and recourse, in both the micro or macro viewpoint. The key idea is that instead of determining the best action for each individual independently (i.e., finding their nearest counterfactual explanation [38]), it enumerates the space of actions and determines how many and which individuals
achieve recourse through each action. Furthermore, FACTS employs a systematic way to explore the feature space and discover any subspace such that recourse bias exists among the protected subgroups within. FACTS ranks the subspaces in decreasing order of recourse bias it detects, and for each provides an interpretable summary of its findings.
Related Work We distinguish between fairness of predictions and fairness of recourse. The former aims to capture and quantify unfairness by comparing directly the model's predictions [27, 2] at: the individual level, e.g., individual fairness [9], counterfactual/causal-based fairness [21, 19, 26]; and the group level, e.g., demographic parity [39], equal odd/opportunity [12].
Fairness of recourse is a more recent notion, related to counterfactual explanations [38], which explains a prediction for an individual (the factual) by presenting the "best" counterfactual that would result in the opposite prediction, offering thus recourse to the individual. Best, typically means the nearest counterfactual in terms of a distance metric in the feature space. Another perspective, which we adopt here, is to consider the action that transforms a factual into a counterfactual, and specify a cost function to quantify the effort required by an individual to perform an action. In the simplest case, the cost function can be the distance between factual and counterfactual, but it can also encode the feasibility of an action (e.g., it is impossible to decrease age) and the plausibility of a counterfactual (e.g., it is out-of-distribution). It is also possible to view actions as interventions that act on a structural causal model capturing cause-effect relationships among attributes [15]. Hereafter, we adopt the most general definition, where a cost function is available, and assume that the best counterfactual explanation is the one that comes from the minimum cost action. Counterfactual explanations have been suggested as a mechanism to detect possible bias against protected subgroups, e.g., when they require changes in protected attributes [13].
Fairness of recourse, first introduced in [35] and formalized in [10], is defined at the group level as the disparity of the mean cost to achieve recourse (called burden in subsequent works) among the protected subgroups. Fairness of recourse for an individual is when they require the same cost to achieve recourse in the actual world and in an imaginary world where they would have a different value in the protected attribute [37]. This definition however only applies when a structural causal model of the world is available. Our work expands on these ideas and proposes alternate definitions that capture a macro and a micro viewpoint of fairness of recourse.
There is a line of work on auditing models for fairness of predictions at the subpopulation level [17, 18]. For example, [34] identifies subpopulations that show dependence between a performance measure and the protected attribute. [3] determines whether people are harmed due to their membership in a specific group by examining a ranking of features that are most associated with the model's behavior. There is no equivalent work for fairness of recourse, although the need to consider the subpopulation is recognized in [16] due to uncertainty in assumptions or to intentionally study fairness. Our work is the first that audits for fairness of recourse at the subpopulation level.
A final related line of work is global explainability. For example, recourse summaries [31, 24, 25] summarizes individual counterfactual explanations globally, and as the authors in [31, 25] suggest can be used to manually audit for unfairness in subgroups of interest. [23] aims to explain how a model behaves in subspaces characterized by certain features of interest. [8] uses counterfactuals to unveil whether a black-box model, that already complies with the regulations that demand the omission of sensitive attributes, is still biased or not, by trying to find a relation between proxy features and bias. Our work is related to these methods in that we also compute counterfactual explanations for all instances, albeit in a more efficient manner, and with the goal to audit fairness of recourse on subpopulation level.
2 Fairness of Recourse for Subgroups
We consider a feature space  $X = X_{1} \times \dots \times X_{n}$ , where  $X_{n}$  denotes the protected feature, which, for ease of presentation, takes two protected values  $\{0,1\}$ . For an instance  $x \in X$ , we use the notation  $x.X_{i}$  to refer to its value in feature  $X_{i}$ .
We consider a binary classifier  $h: X \to \{-1, 1\}$  where the positive outcome is the favorable one. For a given  $h$ , we are concerned with a dataset  $D$  of adversely affected individuals, i.e., those who
receive the unfavorable outcome. We prefer the term instance to refer to any point in the feature space  $X$ , and the term individual to refer to an instance from the dataset  $D$ .
We define an action  $a$  as a set of changes to feature values, e.g.,  $a = \{ \text{country} \to US, \text{education-num} \to 12 \}$ . We denote as  $A$  the set of possible actions. An action  $a$  when applied to an individual (a factual instance)  $x$  results in a counterfactual instance  $x' = a(x)$ . If the individual  $x$  was adversely affected  $(h(x) = -1)$  and the action results in a counterfactual that receives the desired outcome  $(h(a(x)) = 1)$ , we say that action  $a$  offers recourse to the individual  $x$  and is thus effective. In line with the literature, we also refer to an effective action as a counterfactual explanation for individual  $x$  [38].
An action  $a$  incurs a cost to an individual  $x$ , which we denote as  $\mathrm{cost}(a, x)$ . The cost function captures both how feasible the action  $a$  is for the individual  $x$ , and how plausible the counterfactual  $a(x)$  is [14].
Given a set of actions  $A$ , we define the recourse cost  $\operatorname{rc}(A, x)$  of an individual  $x$  as the minimum cost among effective actions if there is one, or otherwise some maximum cost represented as  $c_{\infty}$ :
$$
\operatorname {r c} (A, x) = \left\{ \begin{array}{l l} \min  \{\operatorname {c o s t} (a, x) | a \in A: h (a (x)) = 1 \}, & \text {i f} \exists a \in A: h (a (x)) = 1; \\ c _ {\infty}, & \text {o t h e r w i s e}. \end{array} \right.
$$
An effective action of minimum cost is also called a nearest counterfactual explanation [14].
We define a subspace  $X_{p} \subseteq X$  using a predicate  $p$ , which is a conjunction of feature-level predicates of the form "feature-operator-value", e.g., the predicate  $p = (country = US) \land (education-num \geq 9)$  defines instances from the US that have more than 9 years of education.
Given a predicate  $p$ , we define the subpopulation group  $G_{p} \subseteq D$  as the set of affected individuals that satisfy  $p$ , i.e.,  $G_{p} = \{x \in D | p(x)\}$ . We further distinguish between the protected subgroups  $G_{p,1} = \{x \in D | p(x) \wedge x. X_{n} = 1\}$  and  $G_{p,0} = \{x \in D | p(x) \wedge x. X_{n} = 0\}$ . When the predicate  $p$  is understood, we may omit it in the designation of a group to simplify notation.
2.2 Effectiveness-Cost Trade-Off
For a specific action  $a$ , we naturally define its effectiveness (eff) for a group  $G$ , as the proportion of individuals from  $G$  that achieve recourse through  $a$ :
$$
\operatorname {e f f} (a, G) = \frac {1}{| G |} | \{x \in G | h (a (x)) = 1 \} |.
$$
Note that effectiveness is termed correctness in [31] and coverage in [25]. We want to examine how recourse is achieved for the group  $G$  through a set of possible actions  $A$ . We define the aggregate effectiveness (aeff) of  $A$  for  $G$  in two distinct ways.
In the micro viewpoint, the individuals in the group are considered independently, and each may choose the action that benefits itself the most. Concretely, we define the micro-effectiveness of set of actions  $A$  for group  $G$  as the proportion of individuals in  $G$  that can achieve recourse through some action in  $A$ , i.e.,
$$
\operatorname {a e f f} _ {\mu} (A, G) = \frac {1}{| G |} | \{x \in G | \exists a \in A, \operatorname {e f f} (a, x) = 1 \}.
$$
In the macro viewpoint, the group is considered as a whole, and an action is applied collectively to all individuals in the group. Concretely, we define the macro-effectiveness of set of actions  $A$  for group  $G$  as the largest proportion of individuals in  $G$  that can achieve recourse through the same action in  $A$ , i.e.,
$$
\operatorname {a e f f} _ {\mathbb {M}} (A, G) = \max  _ {a \in A} \frac {1}{| G |} | \{x \in G | \operatorname {e f f} (a, x) = 1 \} |.
$$
For a group  $G$ , actions  $A$ , and a cost budget  $c$ , we define the in-budget actions as the set of actions that cost at most  $c$  for any individual in  $G$ :
$$
A _ {c} = \{a \in A | \forall x \in G, \operatorname {c o s t} (a, x) \leq c \}.
$$
We define the effectiveness-cost distribution (ecd) as the function that for a cost budget  $c$  returns the aggregate effectiveness possible with in-budget actions:
$$
\operatorname {e c d} (c; A, G) = \operatorname {a e f f} \left(A _ {c}, G\right).
$$
We use  $\mathrm{ecd}_{\mu}$ ,  $\mathrm{ecd}_{\mathbb{M}}$  to refer to the micro, macro viewpoints of aggregate effectiveness. A similar concept, termed the coverage-cost profile, appears in [25].
The value  $\operatorname{ecd}(c; A, G)$  is the proportion of individuals in  $G$  that can achieve recourse through actions  $A$  with cost at most  $c$ . Therefore, the ecd function has an intuitive probabilistic interpretation. Consider the subspace  $X_p$  determined by predicate  $p$ , and define the random variable  $C$  as the cost required by an instance  $x \in X_p$  to achieve recourse. The function  $\operatorname{ecd}(c; A, G_p)$  is the empirical cumulative distribution function of  $C$  using sample  $G_p$ .
The inverse effectiveness-cost distribution function  $\operatorname{ecd}^{-1}(\phi; A, G)$  takes as input an effectiveness level  $\phi \in [0, 1]$  and returns the minimum cost required so that  $\phi|G|$  individuals achieve recourse.
2.3 Definitions of Subgroup Recourse Fairness
We define recourse fairness of classifier  $h$  for a group  $G$  by comparing the ecd functions of the protected subgroups  $G_0, G_1$  in different ways.
The first two definitions are cost-oblivious, and apply whenever we can ignore the cost function. Specifically, given a set of actions  $A$  and a group  $G$ , we assume that all actions in  $A$  are considered equivalent, and that the cost of any action is the same for all individuals in the group; i.e.,  $\mathrm{cost}(a,x) = \mathrm{cost}(a',x')$ ,  $\forall a \neq a' \in A, x \neq x' \in G$ . The definitions simply compare the aggregate effectiveness of a set of actions on the protected subgroups.
Equal Effectiveness This definition has a micro and a macro interpretation, and says that the classifier is fair if the same proportion of individuals in the protected subgroups can achieve recourse:  $\mathsf{aeff}(A,G_0) = \mathsf{aeff}(A,G_1)$ .
Equal Choice for Recourse This definition has only a macro interpretation and claims that the classifier is fair if the protected subgroups can choose among the same number of sufficiently effective actions to achieve recourse, where sufficiently effective means the actions should work for at least  $100\phi \%$  (for  $\phi \in [0,1]$ ) of the subgroup:
$$
\left| \left\{a \in A \mid \operatorname {e f f} \left(a, G _ {0}\right) \geq \phi \right\} \right| = \left| \left\{a \in A \mid \operatorname {e f f} \left(a, G _ {1}\right) \geq \phi \right\} \right|.
$$
The next three definitions assume the cost function is specified, and have both a micro and a macro interpretation.
Equal Effectiveness within Budget The classifier is fair if the same proportion of individuals in the protected subgroups can achieve recourse with a cost at most  $c$ :
$$
\operatorname {e c d} (c; A, G _ {0}) = \operatorname {e c d} (c; A, G _ {1}).
$$
Equal Cost of Effectiveness The classifier is fair if the minimum cost to achieve aggregate effectiveness of  $\phi \in [0,1]$  in the protected subgroups is equal:
$$
\operatorname {e c d} ^ {- 1} (\phi ; A, G _ {0}) = \operatorname {e c d} ^ {- 1} (\phi ; A, G _ {1}).
$$