SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Beepul Bharti
Johns Hopkins University bbharti1@jhu.edu
Paul Yi
University of Maryland pyi@som.umaryland.edu
Jeremias Sulam
Johns Hopkins University jsulam1@jhu.edu
As the use of machine learning models in real world high-stakes decision settings continues to grow, it is highly important that we are able to audit and control for any potential fairness violations these models may exhibit towards certain groups. To do so, one naturally requires access to sensitive attributes, such as demographics, biological sex, or other potentially sensitive features that determine group membership. Unfortunately, in many settings, this information is often unavailable. In this work we study the well known equalized odds (EOD) definition of fairness. In a setting without sensitive attributes, we first provide tight and computable upper bounds for the EOD violation of a predictor. These bounds precisely reflect the worst possible EOD violation. Second, we demonstrate how one can provably control the worst-case EOD by a new post-processing correction method. Our results characterize when directly controlling for EOD with respect to the predicted sensitive attributes is – and when is not – optimal when it comes to controlling worst-case EOD. Our results hold under assumptions that are milder than previous works, and we illustrate these results with experiments on synthetic and real datasets.
Machine learning (ML) algorithms are increasingly used in high stakes prediction applications that can significantly impact society. For example, ML models have been used to detect breast cancer in mammograms [37], inform parole and sentencing decisions [14], and aid in loan approval decisions [41]. While these algorithms often demonstrate excellent overall performance, they can be dangerously unfair and negatively impact under-represented groups [38]. Some of these unforeseen negative consequences can even be fatal, e.g. deep learning models for chest x-ray disease classification exhibit under diagnosis bias towards certain sensitive groups [36]. Recommendations to ensure that ML systems do not exacerbate societal biases have been raised by several groups, including the White House, which in 2016, released a report on big data, algorithms, and civil rights [31]. It is thus critical to understand how to rigorously evaluate the fairness of ML algorithms and control for unfairness during model development.
These needs have prompted considerable research in the area of Fair ML. While there exist many definitions of algorithmic fairness [7], common notions of group fairness consider different error rates of a predictor across different groups: males and females, white and non-white, etc. For example, the equal opportunity criterion requires the true positive rate (TPR) be equal across both groups, while equalized odds requires both TPR and false positive rate (FPR) to be the same across groups [21]. Obtaining predictors that are fair therefore requires enforcing these constraints on error rates across groups during model development, which can be posed as a constrained (or regularized) optimization problem [35, 42, 2, 13, 16]. Alternatively, one can devise post-processing strategies to modify a certain predictor to correct for differences in TPR and FPR [21, 18, 12, 1], or even include data
pre-processing steps that ensure that unfair models could not be obtained from such data to begin with [39, 8].
Naturally, all these techniques for estimating or enforcing fairness require access to a dataset with features,  $X$ , responses,  $Y$ , and sensitive attributes,  $A$ . However, in many settings this is difficult or impossible, as datasets often do not include samples that have all these variables. This could be because the sensitive attribute data was withheld due to privacy concerns, which is very common with medical data due to HIPAA federal law requirements, or simply because it was deemed unnecessary to record [40, 43]. A real-world example of this is in the recent Kaggle-hosted RSNA Chest X-Ray Pneumonia Detection Challenge [33]. Even though this dataset of chest x-rays was painstakingly annotated for pneumonia disease by dozens of radiologists, it did not include sensitive attributes (e.g., age, sex, and race), precluding the evaluation of fairness of the models developed as part of the challenge. In settings like this, where there is limited or no information on the sensitive attribute of interest, it is still important to be able to accurately estimate the violation of fairness constraints by a ML classifier and to be able to alleviate these violations before deploying it in a sensitive application
This leads to the natural question:
How can one assess and control the fairness of a classifier without having access to sensitive attribute data?
In other words, how can we measure and potentially control the fairness violations of a classifier for  $Y$  with respect to a sensitive attribute  $A$ , when we have no data that jointly observes  $A$  and  $Y$ ?
Estimating unfairness and, more importantly, developing fair predictors where there is no – or only partial – information about the sensitive attribute has only recently received increasing attention. The recent work by Zhao et al. [44] explores the perspective of employing features that are correlated with the sensitive attribute, and shows that enforcing low correlation with such "fairness related features" can lead to models with lower bias. Although these techniques are promising, they require domain expertise to determine which features are highly correlated with the sensitive attribute. An appealing alternative that has been studied is the use proxy sensitive attributes that are created by a second predictor trained on a different data set that contains only sensitive attribute information [20, 25, 10, 5]. This strategy has been widely adopted in many domains such as healthcare [17], finance [6], and politics [23]. While using sensitive attribute predictors has proven to be an effective and practical solution, it must be done with care, as this opens new problems for estimating and controlling for fairness. The work by Prost et al. [34] considers the estimation of fairness in a setting where one develops a predictor for an unobserved covariate, but it does not contemplate predicting the sensitive attribute itself. On the other hand Chen et al. [10] study the sources of error in the estimation of fairness via predicted proxies computed using threshold functions, which are prone to over-estimation.
The closest to our work are the recent results by Kallus et al. [25], and Awasthi et al. [5, 4]. Kallus et al. [25] study the identifiability of fairness violations under general assumptions on the distribution and classifiers. They show that, in the absence of the sensitive attribute, the fairness violation of predictions,  $\widehat{Y}$ , is unidentifiable unless strict assumptions<sup>1</sup> are made or if there is some common observed data over  $A$  and  $Y$ . Nonetheless, they show not all hope is lost and provide closed form upper and lower bounds of the fairness violations of  $\widehat{Y}$  under the assumption that one has two datasets: one that is drawn from the marginal over  $(X, A)$  and the other drawn from the marginal over  $(X, Y, \widehat{Y})$ . Their analysis, however, does not consider predictors  $\widehat{Y} = f(X)$  or  $\widehat{A} = h(X)$ , and instead their bounds depend explicitly on the conditional probabilities,  $\mathbb{P}(A \mid X)$  and  $\mathbb{P}(\widehat{Y}, Y \mid X)$ , along with the distribution over the features,  $\mathbb{P}(X)$ . Unfortunately, with this formulation, it is unclear how the bounds would change if the estimation of the conditional probabilities was inaccurate (as is the case when developing predictors  $\widehat{A}$  in the real world). Furthermore, in settings where  $X$  is high dimensional (as for image data), calculating such bounds would become intractable. Since the fairness violation cannot be directly modified, they then study when these bounds can be reduced and improved. However, they do so in settings that impose smoothness assumptions over  $(X, A, Y, \widehat{Y})$ ,
<sup>1</sup>Kallus et al. [25] show that if,  $\widehat{Y}, Y \perp A \mid X$ , then the fairness violations are identifiable.
which clearly are not-verifiable without data over the complete joint distribution. As a result, their results do not provide any actionable method that could improve the bounds.
Awasthi et al. [5], on the other hand, make progress in understanding properties of the sensitive attribute predictor,  $\widehat{A}$ , that are desirable for easy and accurate fairness violation estimation and control of a classifier  $\widehat{Y}$ . Assuming that  $\widehat{Y} \perp \widehat{A} \mid (A, Y)$ , they demonstrate that the true fairness violation is in fact proportional to the estimated fairness violation (the fairness violation using  $\widehat{A}$  in lieu of A). This relationships yield the counter-intuitive result that given a fixed error budget for the sensitive attribute predictor, the optimal attribute predictor for the estimation of the true fairness violation is one with the most unequal distribution of errors across the subgroups of the sensitive attribute. However, one is still unable to actually calculate the true fairness violation – as it is unidentifiable. Nonetheless, the relationship does demonstrate that if one can maintain the assumption above while controlling for fairness with respect to  $\widehat{A}$ , then doing so will provably reduce the true fairness violation with respect to  $A$ . Unfortunately, while these rather strict assumptions can be met in some limited scenarios (as in [4]), these are not applicable in general – and cannot even be tested without access to data over  $(A, Y)$ .
Overall, while progress has been made in understanding how to estimate and control fairness violations in the presence of incomplete sensitive attribute information, these previous results highlight that this can only be done in simple settings (e.g., having access to some data from the entire distribution, or by making strong assumptions of conditional independence). Moreover, it remains unclear whether tight bounds can be obtained that explicitly depend on the properties of the predictor  $\widehat{Y}$ , allowing for actionable bounds that can provably mitigate for its fairness violation without having an observable sensitive attribute or making stringent assumptions.
The contributions of our work can be summarized as follows:
- We study the well-known equalized odds (EOD) definition of fairness in a setting where the sensitive attributes,  $A$ , are not observed with the features  $X$  and labels  $Y$ . We provide tight and computable bounds on the EOD violation of a classifier,  $\widehat{Y} = f(X)$ . These bounds represent the worst-case EOD violation of  $f$  and employ a predictor for the sensitive attributes,  $\widehat{A} = h(X)$ , obtained from a sample over the distribution  $(X, A)$ .
- We provide a precise characterization of the classifiers that achieve minimal worst-case EOD violations with respect to unobserved sensitive attributes. Through this characterization, we demonstrate when simply correcting for fairness with respect to the proxy sensitive attributes will yield minimal worst-case EOD violations, and when instead it proves to be sub-optimal.
- We provide a simple and practical post-processing technique that provably yields classifiers that maximize prediction power while achieving minimal worst-case EOD violations with respect to unobserved sensitive attributes.
- We illustrate our results on a series of simulated and real data of increasing complexity.
We work within a binary classification setting and consider a distribution  $\mathcal{Q}$  over  $(\mathcal{X} \times \mathcal{A} \times \mathcal{Y})$  where  $\mathcal{X} \subseteq \mathbb{R}^n$  is the feature space,  $\mathcal{Y} = \{0,1\}$  the label space, and  $\mathcal{A} = \{0,1\}$  the sensitive attribute space. Furthermore, and adopting the setting of [5], we consider 2 datasets,  $\mathcal{D}_1$  and  $\mathcal{D}_2$ . The former is drawn from the marginal over  $(\mathcal{X},\mathcal{A})$  of  $\mathcal{Q}$  while  $\mathcal{D}_2$  is drawn from the marginal  $(\mathcal{X},\mathcal{Y})$  of  $\mathcal{Q}$ . In this way,  $\mathcal{D}_1$  and  $\mathcal{D}_2$  contain the same set of features,  $\mathcal{D}_1$  contains sensitive attribute information and  $\mathcal{D}_2$  contains label information. The drawn samples in  $\mathcal{D}_1$  and  $\mathcal{D}_2$  are i.i.d over their respective marginals and different from one another.
Similar to previous work [10, 34, 5], we place ourselves in a demographically scarce regime where there is designer who has access to  $\mathcal{D}_1$  to train a sensitive attribute predictor  $h: \mathcal{X} \to \mathcal{A}$  and a developer, who has access to  $\mathcal{D}_2$ , the sensitive attribute classifier  $h$ , and all computable probabilities  $\mathbb{P}(h(X), A)$  that the designer of  $h$  can extract from  $\mathcal{D}_1$ . In this setting, the goal of the developer is to learn a classifier  $f: \mathcal{X} \to \mathcal{Y}$  (from  $\mathcal{D}_2$ ) that is fair with respect to  $A$  utilizing  $\widehat{A}$ . The central idea is
to augment every sample in  $\mathcal{D}_2$ ,  $(x_i, y_i)$ , by  $(x_i, y_i, \hat{a}_i)$ , where  $\hat{a}_i = h(x_i)$ . Intuitively, if the error of the sensitive attribute predictor, denoted herein by  $U = \mathbb{P}(h(X) \neq A)$ , is low, we could hope that fairness with respect to the real (albeit unobserved) sensitive attribute can be faithfully estimated. Our goal is to thus estimate the error incurred in measuring and enforcing fairness constraints by means of  $\hat{A} = h(X)$ , and potentially alleviate or control for it.
Throughout the remainder of this work we focus on equalized odds (EOD) as our fairness metric [21] of interest as it is one of the most popular notions of fairness. Thus moving forward, the term fairness refers specifically to EOD. We denote  $\hat{Y} = f(X)$  for simplicity, and for  $i,j\in \{0,1\}$  define the group conditional probabilities
$$
\alpha_ {i, j} = \mathbb {P} (\hat {Y} = 1 \mid A = i, Y = j). \tag {1}
$$
These probabilities quantify the TPR (when  $j = 1$ ) and FPR (when  $j = 0$ ), for either protected group ( $i = 0$  or  $i = 1$ ). We assume that the base rates,  $r_{i,j} = \mathbb{P}(A = i, Y = j) > 0$  so that these quantities are not undefined. With these conditionals probabilities, we define the true fairness violation of  $f$ , with the two quantities
$$
\Delta_ {\mathrm {T P R}} (f) = \alpha_ {1, 1} - \alpha_ {0, 1} \quad \text {a n d} \quad \Delta_ {\mathrm {F P R}} (f) = \alpha_ {1, 0} - \alpha_ {0, 0} \tag {2}
$$
which respectively quantify the absolute difference in TPR and FPRs among the two protected groups.
We also need to characterize the performance of the sensitive attribute classifier,  $h$ . The missclassification error of  $h$  can be decomposed as,  $U = U_{0} + U_{1}$  where  $U_{i} = \mathbb{P}(\widehat{A} = i, A \neq i)$ , for  $i \in \{0, 1\}$ . We define the difference in errors to be
$$
\Delta U = U _ {0} - U _ {1}. \tag {3}
$$
In a demographically scarce regime, the rates  $r_{i,j}$ , and more importantly the quantities of interest,  $\Delta_{\mathrm{TPR}}(f)$  and  $\Delta_{\mathrm{FPR}}(f)$ , cannot be computed because samples from  $A$  and  $Y$  are not jointly observed. However, using the sensitive attribute classifier  $h$ , we can predict  $\hat{A}$  on  $\mathcal{D}_2$  and compute
$$
\hat {r} _ {i, j} = \mathbb {P} (\widehat {A} = i, Y = j) \quad \text {a n d} \quad \widehat {\alpha} _ {i, j} = \mathbb {P} (\widehat {Y} = 1 \mid \widehat {A} = i, Y = j),
$$
which serve as the estimates for the true base rates and group TPRs and FPRs.
With the setting defined, we will now present our results. The first result provides computable bounds on the true fairness violation of  $f$  with respect to the true, but unobserved, sensitive attribute  $A$ . The bounds precisely characterize the worst-case fairness violation of  $f$ . Importantly, as we will explain later, this first result will provide insight into what properties  $f$  must satisfy so that its worst-case fairness violation is minimal. In turn, these results will lead to a simple post-processing method that can correct a pretrained classifier  $f$  into another one,  $\bar{f}$ , that has minimal worst-case fairness violations. Before presenting our findings, we first describe the key underlying assumption we make about the pair of classifiers,  $h$  and  $f$ , so that the subsequent results are true.
Assumption 1. For  $i,j\in \{0,1\}$ , the classifiers  $\widehat{Y} = f(X)$  and  $\widehat{A} = h(X)$  satisfy
$$
\frac {U _ {i}}{\hat {r} _ {i , j}} \leq \widehat {\alpha} _ {i, j} \leq 1 - \frac {U _ {i}}{\hat {r} _ {i , j}}. \tag {4}
$$
To parse this assumption, it is easy to show that this is met when a)  $h$  is accurate enough for the setting, namely that  $\mathbb{P}(\widehat{A} = i, A \neq i) \leq \frac{1}{2} \mathbb{P}(\widehat{A} = i, Y = j)$ , and b) the predictive power of  $h$  is better than the ability of  $f$  to predict the labels,  $Y$  – or more precisely,  $\mathbb{P}(\widehat{A} = i, A \neq i) \leq \mathbb{P}(\widehat{Y} = j, \widehat{A} = i, Y \neq j)$ . We refer the reader to Appendix A for a thorough explanation for why this is true. While this assumption may seem limiting, this is milder than those in existing results: First, accurate predictors  $h$  can be developed [6, 17, 23, 19], thus satisfying the assumption (our numerical results will highlight this fact as well). Second, other works [5, 9], require assumptions on  $\widehat{A}$  and  $\widehat{Y}$  that are unverifiable in a demographically scarce regime. Our assumption, on the other hand, can always be easily verified because all the quantities are computable. Furthermore, this assumption can be also relaxed if one desires partial guarantees. More specifically, if the above is true for  $i \in \{0, 1\}$  and only for  $j = 1$ , then all subsequent results for  $\Delta_{\mathrm{TPR}}$  hold. Similarly, if it only holds for  $j = 0$ , the results for  $\Delta_{\mathrm{FPR}}$  hold. Therefore, if one only cares about the equal opportunity definition of fairness, as an example, then this only needs to hold for  $i \in \{0, 1\}$  and  $j = 1$ .
3.1 Bounding Fairness Violations with Proxy Sensitive Attributes
With Assumption 1 in place, we present our main result.
Theorem 1 (Bounds on  $\Delta_{\mathrm{TPR}}(f)$  and  $\Delta_{\mathrm{FPR}}(f)$ ). Under Assumption 1, we have that
$$
\left| \Delta_ {T P R} (f) \right| \leq B _ {T P R} (f) \triangleq \max  \left\{\left| B _ {1} + C _ {0, 1} \right|, \left| B _ {1} - C _ {1, 1} \right| \right\} \tag {5}
$$
$$
\left| \Delta_ {F P R} (f) \right| \leq B _ {F P R} (f) \triangleq \max  \left\{\left| B _ {0} + C _ {0, 0} \right|, \left| B _ {0} - C _ {1, 0} \right| \right\}
$$
$$
B _ {j} = \frac {\hat {r} _ {1 , j}}{\hat {r} _ {1 , j} + \Delta U} \widehat {\alpha} _ {1, j} - \frac {\hat {r} _ {0 , j}}{\hat {r} _ {0 , j} - \Delta U} \widehat {\alpha} _ {0, j} \quad a n d \quad C _ {i, j} \quad = U _ {i} \left(\frac {1}{\hat {r} _ {1 , j} + \Delta U} + \frac {1}{\hat {r} _ {0 , j} - \Delta U}\right).
$$
Furthermore, the upper bounds for  $|\Delta_{TPR}(f)|$  and  $|\Delta_{FPR}(f)|$  are tight.
The proof, along with all others in this work, are included in Appendix A. We now make a few remarks on this result. First, the bound is tight in that there exists settings (albeit unlikely) with particular marginal distributions such that the bounds hold with equality. Second, even though  $|\Delta_{\mathrm{TPR}}(f)|$  and  $|\Delta_{\mathrm{FPR}}(f)|$  cannot be calculated, a developer can still calculate the worst-case fairness violations,  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$ , because these depend on quantities that are all computable in practice. Thus, if  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$  are low, then the developer can proceed having a guarantee on the maximal fairness violation of  $f$ , even while not observing  $|\Delta_{\mathrm{TPR}}(f)|$  and  $|\Delta_{\mathrm{FPR}}(f)|$ . On the other hand, if these bounds are large, this implies a potentially large fairness violation over the protected group  $A$  by  $f$  and the developer should not proceed in deploying  $f$  and instead seek to learn another classifier with smaller bounds. Third, the obtained bounds are linear in the parameters  $\hat{\alpha}_{i,j}$ , which the developer can adjust as they are properties of  $f$ : this will become useful shortly.
3.2 Optimal Worst-Case Fairness Violations
Given the result above, what properties should classifiers  $f$  satisfy such that  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$  are minimal? Moreover, are the classifiers  $f$  that are fair with respect to  $\hat{A}$ , the ones that have smallest  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$ ? We now answer these questions in the following theorem.
Theorem 2 (Minimizers of  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$ ). Let  $\widehat{A} = h(X)$  be a fixed sensitive attribute classifier with errors  $U_0$  and  $U_1$  that produces rates  $\hat{r}_{i,j} = \mathbb{P}(\widehat{A} = i, Y = j)$ . Let  $\mathcal{F}$  be the set of all predictors of  $Y$ , parameterized by rates  $\widehat{\alpha}_{i,j}$ , that, paired with  $h$ , satisfy Assumption 1. Then,  $\exists \overline{Y} \in \mathcal{F}$  with group conditional probabilities,  $\widehat{\underline{\alpha}}_{i,j} = \mathbb{P}(\overline{Y} = 1 \mid \widehat{A} = i, Y = j)$  that satisfies the following condition,
$$
\frac {\hat {r} _ {0 , j}}{\hat {r} _ {0 , j} - \Delta U} \underline {{\alpha}} _ {0, j} - \frac {\hat {r} _ {1 , j}}{\hat {r} _ {1 , j} + \Delta U} \underline {{\alpha}} _ {1, j} = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , j} + \Delta U} + \frac {1}{\hat {r} _ {0 , j} - \Delta U}\right). \tag {6}
$$
Furthermore, any such  $\overline{Y}$  has minimal maximal fairness violation, i.e.  $\forall \widehat{Y} \in \mathcal{F}$ ,
$$
\left| \Delta_ {T P R} (\bar {Y}) \right| \leq B _ {T P R} (\bar {Y}) \leq B _ {T P R} (\hat {Y}) \quad a n d \quad \left| \Delta_ {F P R} (\bar {Y}) \right| \leq B _ {F P R} (\bar {Y}) \leq B _ {F P R} (\hat {Y}). \tag {7}
$$
This result provides a precise characterization of the conditions that lead to minimal worst-case fairness violations. Observe that if  $\Delta U \neq 0$ , the classifier with minimal  $B_{\mathrm{TPR}}(f)$  and  $B_{\mathrm{FPR}}(f)$  involves  $\widehat{\alpha}_{i,j}$  such that  $\widehat{\alpha}_{1,j} \neq \widehat{\alpha}_{0,j}$ , i.e. it is not fair with respect to  $\widehat{A}$ . On the other hand, if the errors of  $h$  are balanced ( $\Delta U = 0$ ), then minimal bounds are achieved by being fair with respect to  $\widehat{A}$ .
3.3 Controlling Fairness Violations with Proxy Sensitive Attributes
Now that we understand what conditions  $f$  must satisfy so that it's worst case fairness violations are minimal, what remains is a method to obtain such a classifier. We take inspiration from the post-processing method proposed by Hardt et al. [21], which derives a classifier  $\overline{Y} = \bar{f}(X)$  from  $\widehat{Y} = f(X)$  that satisfies equalized odds with respect to a sensitive attribute  $A$  while minimizing an expected missclassification loss - only applicable if one has access to  $A$ , which is not true in our setting. Nonetheless, since our method will generalize this idea, we first briefly comment on this