SlowGuess/NeurIPS_2023_OCR huggingface dataset:
approach. The method they propose works as follows: given a sample with initial prediction  $\widehat{Y} = \hat{y}$  and sensitive attribute  $A = a$ , the derived predictor  $\bar{f}$ , with group conditional probabilities  $\underline{\alpha}_{i,j} = \mathbb{P}(\overline{Y} = 1 \mid A = i, Y = j)$ , predicts  $\overline{Y} = 1$  with probability  $p_{a,\hat{y}} = \mathbb{P}(\overline{Y} = 1 \mid A = a, \widehat{Y} = \hat{y})$ . The four probabilities  $p_{0,0}, p_{0,1}, p_{1,0}, p_{1,1}$  can be then calculated so that  $\overline{Y}$  satisfies equalized odds and the expected loss between  $\overline{Y}$  and labels  $Y$ , i.e.  $\mathbb{E}[L(\overline{Y}, Y)]$ , is minimized. The fairness constraint, along with the objective to minimize the expected loss, give rise to the linear program:
Equalized Odds Post-Processing (Hardt et al. [21])
$$
\min  _ {p _ {a, \hat {y}} \in [ 0, 1 ]} \mathbb {E} [ L (\bar {Y}, Y) ] \quad \text {s u b j e c t t o} \quad \underline {{\alpha}} _ {0, j} = \underline {{\alpha}} _ {1, j} \quad \text {f o r} \quad j \in \{0, 1 \}. \tag {8}
$$
Returning to our setting where we do not have access to  $A$  but only proxy variables  $\widehat{A} = h(X)$ , we seek classifiers  $f$  that are fair with respect to the sensitive attribute  $A$ . Since these attributes are not available, (thus rendering the fairness violation to be unidentifiable [25]), a natural alternative is to minimize the worst-case fair violation with respect to  $A$ , which can be computed as shown in Theorem 1. Of course, such an approach will only minimize the worst case fairness and one cannot certify that the true fairness violation will decrease because – as explained above – it is unidentifiable. Nonetheless, since we know what properties optimal classifiers must satisfy (as per Theorem 2), we can now modify the above problem to construct a corrected classifier,  $\bar{f}$ , as follows. First, we must employ  $\widehat{A}$  in place of  $A$ , which amounts to employing  $\widehat{\alpha}_{i,j}$  in lieu of  $\alpha_{i,j}$ . To this end, denote the (corrected) group conditional probabilities of  $\overline{Y}$  to be  $\underline{\alpha}_{i,j}$ . Second, the equalized odds constraint is replaced with the constraint in Theorem 2. Lastly, we also enforce the additional constraints detailed in Assumption 1 on the  $\underline{\alpha}_{i,j}$ . With these modifications in place, we present the following generalized linear program:
Worst-case Fairness Violation Reduction
$$
\min  _ {p _ {\hat {a}, \hat {y}} \in [ 0, 1 ]} \quad \mathbb {E} [ L (\overline {{Y}}, Y) ]
$$
$$
\frac {\hat {r} _ {0 , j}}{\hat {r} _ {0 , j} + \Delta U} \underline {{\hat {\alpha}}} _ {0, j} - \frac {\hat {r} _ {1 , j}}{\hat {r} _ {1 , j} - \Delta U} \underline {{\hat {\alpha}}} _ {1, j} = \frac {\Delta U}{2} \left(\frac {1}{\hat {r} _ {1 , j} + \Delta U} + \frac {1}{\hat {r} _ {0 , j} - \Delta U}\right) \tag {9}
$$
$$
\frac {U _ {i}}{\hat {r} _ {i , j}} \leq \underline {{\alpha}} _ {i, j} \leq 1 - \frac {U _ {i}}{\hat {r} _ {i , j}} \quad \text {f o r} \quad i, j \in \{0, 1 \}.
$$
The solution to this linear program will yield a classifier  $\bar{f}$  that satisfies Assumption 1, has minimal  $B_{\mathrm{TPR}}(\bar{f})$  and  $B_{\mathrm{FPR}}(\bar{f})$ , and has minimal expected loss. Note, that if  $\Delta U = 0$ , then the coefficients of  $\underline{\hat{\alpha}}_{0,j}$  and  $\underline{\hat{\alpha}}_{1,j}$  will equal one, and so the first set of constraints simply reduces to  $\underline{\hat{\alpha}}_{0,j} = \underline{\hat{\alpha}}_{1,j}$  for  $j \in \{0,1\}$  and the linear program above is precisely the post-processing method of [21] with  $\hat{A}$  in place of  $A$  (while enforcing Assumption 1).
Let us briefly recap the findings of this section: We have shown that in a demographically scarce regime, one can provide an upper bound on the true fairness violation of a classifier (Theorem 1). Second, we have presented a precise characterization of the classifiers with minimal worst-case fairness violations. Lastly, we have provided a simple and practical post-processing method (a linear program) that utilizes a sensitive attribute predictor to construct classifiers with minimal worst-case fairness violations with respect to the true, unknown, sensitive attribute.
We now illustrate our theoretical results on synthetic and real world datasets. The code and data necessary to reproduce these experiments are available at https://github.com/Sulam-Group/EOD-with-Proxyes.
We begin with a synthetic example that allows us to showcase different aspects of our results. The data consists of 3 features,  $X_{1}, X_{2}, X_{3} \in \mathbb{R}$ , sensitive attribute  $A \in \{0,1\}$ , and response  $Y \in \{0,1\}$ .
(a)  $\Delta_{\mathrm{TPR}}$  and  $B_{\mathrm{TPR}}$
(b)  $\Delta_{\mathrm{FPR}}$  and  $B_{\mathrm{FPR}}$
Figure 1: Synthetic data  $(\Delta U = 0)$ : true fairness violations, worst-case fairness violations, and expected loss for  $f$ ,  $f_{\widehat{\text{fair}}}$ , and  $f_{\text{opt}}$
(c) Expected loss
The features are sampled from  $(X_{1},X_{2},X_{3})\sim \mathcal{N}(\mu ,\Sigma)$ , with
$$
\mu = \left[ \begin{array}{c} 1 \\ - 1 \\ 0 \end{array} \right] \quad \text {a n d} \quad \Sigma = \left[ \begin{array}{c c c} 1 & 0. 0 5 & 0 \\ 0. 0 5 & 1 & 0 \\ 0 & 0 & 0. 0 5 \end{array} \right].
$$
The sensitive attribute,  $A$ , response  $Y$ , and classifier  $f$ , are modeled as
$$
A = \mathbb {I} [ (X _ {3} + 0. 1) \geq 0 ]
$$
$$
Y \mid X, A \sim \operatorname {B e r n o u l l i} \left(S \left(X _ {1} + X _ {2} + X _ {3} + \epsilon_ {0} (1 - A) + \epsilon_ {1} A\right)\right)
$$
$$
f (X; c _ {1}, c _ {2}) = \mathbb {I} \left(S \left(c _ {1} X _ {1} + c _ {2} X _ {2} + c _ {3} X _ {3}\right) \geq 0. 5\right)
$$
where  $\mathbb{I}(\cdot)$  is the indicator function,  $S(\cdot)$  is the sigmoid function and  $c_{1}, c_{2}, c_{3} \sim \mathcal{N}(1, 0.01)$ . Furthermore,  $\epsilon_0 \sim \mathcal{N}(0, 1), \epsilon_1 \sim \mathcal{N}(0, 0.5)$  are independent noise variables placed on the samples belonging to the groups  $A = 0$  and  $A = 1$  to ensure there is a non trivial fairness violation. Specifically,  $\mathrm{Var}(\epsilon_0) > \mathrm{Var}(\epsilon_1)$  makes  $f(X; c_1, c_2, c_3)$  unfair with respect to  $A = 0$ . Lastly, to measure the predictive capabilities of  $f$ , we use the loss function,  $L(\hat{Y} \neq y, Y = y) = \mathbb{P}(Y \neq y)$ , as this maximizes the well known Youden's Index<sup>2</sup>.
4.1.1 Equal Errors  $(\Delta \mathbf{U} = 0)$
We model the sensitive attribute predictor as
$$
h (X; \delta) = \mathbb {I} ((X _ {3} + 0. 1 + \delta) \geq 0)
$$
where  $\delta \sim \mathcal{N}(0,\sigma^2)$ . We choose  $\sigma^2$  so that  $h(X)$  has a total error  $U\approx 0.04$  distributed so that  $\Delta U\approx 0$  with  $U_{0}\approx U_{1}\approx 0.02$ . We generate 1000 classifiers  $f$  and for each one, calculate  $\Delta_{\mathrm{TPR}},\Delta_{\mathrm{FPR}},B_{\mathrm{TPR}},B_{\mathrm{FPR}}$ , and  $\mathbb{E}[L(f,Y)]$ . Then, on each  $f$ , we run the (naive) equalized odds post processing algorithm to correct for fairness with respect  $\hat{A}$  to yield a classifier  $f_{\widehat{\text{fair}}}$ , and we also run our post-processing algorithm to yield an optimal classifier  $f_{\mathrm{opt}}$ . For both sets of classifiers we again calculate the same quantities.
The results in Fig. 1 present the true fairness violations, worst-case fairness violations, and expected loss for the 3 sets of different classifiers. Observe that both  $B_{\mathrm{TPR}}$  and  $B_{\mathrm{FPR}}$  are significantly lower for  $f_{\widehat{\mathrm{fair}}}$  and  $f_{\mathrm{opt}}$  and that these values for both sets of classifiers are approximately the same. This is expected as  $U_0 \approx U_1$  and so performing the fairness correction algorithm and our proposed algorithm amount to solving nearly identical linear programs. We also show the true fairness violations,  $\Delta_{\mathrm{TPR}}$  and  $\Delta_{\mathrm{FPR}}$  for all the classifiers to portray the gap between the bounds and the true values. As mentioned before, these true values cannot be calculated in a real demographically scarce regime. Nonetheless, the developer of  $f$  now knows, post correction, that  $|\Delta_{\mathrm{TPR}}|, |\Delta_{\mathrm{FPR}}| \lesssim 0.15$ . Lastly, observe that the expected loss for  $f_{\widehat{\mathrm{fair}}}$  and  $f_{\mathrm{opt}}$  are naturally higher compared to that of  $f$ , however the increase in loss is minimal.
4.1.2 Unequal Errors  $(\Delta \mathbf{U}\neq 0)$
We model the sensitive attribute predictor in the same way as in the previous experiment except with  $\delta = c$ , for a constant  $c$  so that the sensitive attribute classifier still has the same total error of
2One can show that  $\mathbb{E}[L(f,Y)] = \mathbb{P}(Y = 1)\mathbb{P}(Y = 0)(1 - Y_f)$  where  $Y_{f}$  is the Youden's Index of  $f$
(b)  $\Delta_{\mathrm{FPR}}$  and  $B_{\mathrm{FPR}}$
Figure 2: Synthetic data  $(\Delta U \neq 0)$ : true fairness violations, worst-case fairness violations, and expected loss for  $f$ ,  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$
$U \approx 0.04$  but distributed unevenly so that  $\Delta U = -0.04$  with  $U_0 = 0$  and  $U_1 \approx 0.04$ . As in the previous experiment we generate classifiers  $f$ , and perform the same correction algorithms to yield  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$  and present the same metrics as before.
The results in Fig. 2 depict the worst-case fairness violations and expected loss for the 3 sets of different classifiers. Observe that our correction algorithm yields classifiers,  $f_{\mathrm{opt}}$ , that have significantly lower  $B_{\mathrm{TPR}}$  and  $B_{\mathrm{FPR}}$ . Furthermore, observe that the  $f_{\widehat{\mathrm{fair}}}$  that results from performing the naive fairness correction algorithm in fact have higher  $B_{\mathrm{FPR}}$  the original classifier  $f!$ . Even though the total error  $U$  has remained the same, its imbalance showcases the optimality of our correction method. Lastly, observe that there is a trade-off, albeit slight, in performing our correction algorithm. The expected loss for  $f_{\mathrm{opt}}$  is higher than that of  $f_{\widehat{\mathrm{fair}}}$  and  $f$ .
4.2 Real World Data
We will now showcase our results on various real world datasets and prediction tasks. We provide a brief description of each task below and further experimental details are included in Appendix B.
FIFA 2020 (Awasthi et al. [5]): The task is to learn a classifier  $f$ , using FIFA 2020 player data [29], that determines if a soccer players wage is above  $(Y = 1)$  or below  $(Y = 0)$  the median wage based on the player's age and their overall attribute. The sensitive attribute  $A$  is player nationality and the player's name is used to learn the sensitive attribute predictor  $h$ . We consider two scenarios, when  $A \in \{\text{English}, \text{Argentine}\}$  (English = 0) and  $A \in \{\text{French}, \text{Spanish}\}$  (French = 0).
ACSPublicCoverage (Ding et al. [15]): The task is to learn a classifier  $f$ , using the 2018 state census data, that determines if a low-income individual, not eligible for Medicare, has coverage from public health insurance  $(Y = 1)$  or does not  $(Y = 0)$ . The sensitive attribute  $A$  is sex (Female = 0) and with a separate dataset (containing the same features used to learn  $f$  (disregarding sex)), we learn the sensitive attribute predictor  $h$ . We work with the 2018 California census data.
CheXpert (Irvin et al. [24]): CheXpert is a large public dataset for chest radiograph interpretation, consisting of 224,316 chest radio graphs of 65,240 patients, with labeled annotations for 14 observations (positive, negative, or unlabeled) including cardiomegaly, atelectasis, consolidation, and several others. The task is to learn a classifier  $f$  to determine if an X-ray contains an annotation for any abnormal condition  $(Y = 1)$  or does not  $(Y = 0)$ . The sensitive attribute  $A$  is sex (Female  $= 0$ ) and with a separate set of X-rays (different from those used to learn  $f$ ) we learn the sensitive attribute predictor  $h$ .
4.2.1 Verification of Assumption 1
In Appendix C we provide Table 1, which demonstrates that Assumption 1 holds for the settings described above. The Actual Value column of Table 1 lists the rates  $\widehat{\alpha}_{i,j}$  and the left and right columns list  $\frac{U_i}{\hat{r}_{i,j}}$  and  $1 - \frac{U_i}{\hat{r}_{i,j}}$  respectively. From Table 1, it is clear that the  $\widehat{\alpha}_{i,j}$  lie in between  $\frac{U_i}{\hat{r}_{i,j}}$  and  $1 - \frac{U_i}{\hat{r}_{i,j}}$  as required by Assumption 1. Notice, we only list the estimated group TPRs,  $\widehat{\alpha}_{i,1}$  for the ACSPublicCoverage 2018 California dataset. This is because Assumption 1, for  $j = 0$ , does not hold for this state: even though  $\hat{A}$  is a very accurate predictor for  $A$ , the label classifier  $f$  has very low estimated group FPRs,  $\widehat{\alpha}_{i,0}$ . As a result, our theoretical results for  $\Delta_{\mathrm{FPR}}$  cannot be used.
(b)  $\Delta_{\mathrm{FPR}}$  and  $B_{\mathrm{FPR}}$
Figure 3: CheXpert data: true fairness violations, worst-case fairness violations, and expected loss for  $f$ ,  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$
4.2.2 Results on CheXpert
The task is to measure and correct for any fairness violations that  $f$  may exhibit towards the sex attribute assuming we do not have the true sex attribute and instead have a sensitive attribute predictor,  $h$ . On a test dataset, we generate our predictions  $\hat{Y} = f$  and  $\hat{A} = h$  to yield a dataset over  $(\hat{A}, Y, \hat{Y})$ , for which the sex predictor,  $h$ , achieves an error of  $U = 0.023$  with  $U_0 \approx 0.008$  and  $U_1 \approx 0.015$ . We utilize the bootstrap method to generate 1000 samples of this dataset and for each sample, perform the same correction algorithms as before to yield  $f_{\widehat{\text{fair}}}$  and  $f_{\text{opt}}$  and calculate the same metrics as done in the previous experiments.
The results in Fig. 3 show that our proposed correction method performs the best in reducing  $B_{\mathrm{TPR}}$  and  $B_{\mathrm{FPR}}$ . Even though  $U$  is very small, since  $U_1 \approx 2U_0$ , simply correcting for fairness with respect to  $\hat{A}$  is suboptimal in reducing the worst-case fairness violations. In particular, the results in Fig. 3 are noteworthy as they depict how our proposed correction method and bounds allow the user to certify that the obtained classifier has a fairness violation in TPRs of no more than 0.06 without having access to the true sensitive attributes. Moreover, the improvement is significant, since before the correction one had  $|\Delta_{\mathrm{TPR}}| \lesssim 0.10$ . In a high-stakes decision setting, such as this one where the model  $f$  could be used to aid in diagnosis, this knowledge could be vital. Naturally, the expected loss is highest for  $f_{\mathrm{opt}}$  but that the increase is minimal. We make no claim as to whether this (small) increase in loss is reasonable for this particular problem setting, and the precise trade-offs must be defined in the context of a broader discussion involving policy makers, domain experts and other stakeholders.
5 Limitations and Broader Impacts
While our results are novel and informative, they come with limitations. First, our results are limited to EOD (and its relaxations) as definitions of fairness. Fairness is highly context-specific and in many scenarios one may be interested in utilizing other definitions of fairness. One can easily extend our results to other associative definitions of fairness, such as demographic parity, predictive parity, and others. However, extending our results to counter-factual notions of fairness [28, 11, 32] is nontrivial and matter of future work. We recommend thoroughly assessing the problem and context in question prior to selecting a definition. It is crucial to ensure that the rationale behind choosing a definition is based on reasoning from both philosophical and political theory, as each definition implicitly makes a distinct set of moral assumptions. For example, with EOD, we implicitly assert that all individuals with the same true label have the same effort-based utility [22]. More generally, other statistical definitions of fairness such as demographic parity and equality of accuracy can be thought of as special instances of Rawlsian equality of opportunity and predictive parity, the other hand, can be thought of as an instance of egalitarian equality of opportunity [22, 30, 3]. We refer the reader to [22] to understand the relationship between definitions of fairness in machine learning and models of Equality of opportunity (EOP) – an extensively studied ideal of fairness in political philosophy.
A second limitation of our results is Assumption 1. This assumption is relatively mild, as it is met for accurate proxy sensitive attributes (as illustrated in the chest X-rays study). Yet, we conjecture that one can do away with this assumption and consider less accurate proxy sensitive attributes with the caveat that the worst case fairness violations will no longer be linear in the TPRs and FPRs. Thus, the
characterization of the classifiers with minimal worst-case bounds would be more involved and the method to minimize these violations will likely be more difficult. Furthermore, while we proposed a simple post-processing correction method, it would be of interest to understand how one could train a classifier – from scratch – to have minimal violations. Lastly, in our setting we assume the sensitive attribute predictor and label classifier are trained on marginal distributions from the same joint distribution. As a next step, it would be important to understand how these results extend to settings where these marginal distributions come from (slightly) different joint distributions. All of this constitutes matter of future work.
Finally, we would like to remark the positive and potentially negative societal impacts of this work. Our contribution is focused on a solution to a technical problem - estimating and correcting for fairness violations when the sensitive attribute and responses are not jointly observed. However, we understand that fairness is a complex and multifaceted issue that extends beyond technical solutions and, more importantly, that there can be disconnect between algorithmic fairness and fairness in a broader socio-technical context. Nonetheless, we believe that technical contributions such as ours can contribute to the fair deployment of machine learning tools. In regards to the technical contribution itself, our results rely on predicting missing sensitive attributes. While such a strategy could be seen as controversial - e.g. because it could involve potential negative consequences such as the perpetuation of discrimination or violation of privacy - this is necessary to build classifiers with minimal worst-case fairness violations in a demographically scarce regime. On the one hand, not allowing for such predictions could be seen as one form of "fairness through unawareness", which has been proven to be an incorrect and misleading strategy in fairness [21, 10]. Moreover, our post-processing algorithm, similar to that of Hardt et al. [21], admits implementations in a differentially private manner as well, since it only requires aggregate information about the data. As a result, our method, which uses an analogous formulation with different constraints, can also be carried out in a manner that preserves privacy. Lastly, note that if one does not follow our approach of correcting for the worst-case fairness by predicting the sensitive attributes, other models trained on this data can inadvertently learn this sensitive attribute indirectly and base decisions of it with negative and potentially grave consequences. Our methodology prevents this from happening by appropriately correcting models to have minimal worst-case fairness violations.
In this paper we address the problem of estimating and controlling potential EOD violations towards an unobserved sensitive attribute by means of predicted proxies. We have shown that under mild assumptions (easily satisfied in practice, as demonstrated) the worst-case fairness violations,  $B_{\mathrm{TPR}}$  and  $B_{\mathrm{FPR}}$ , have simple closed form solutions that are linear in the estimated group conditional probabilities  $\widehat{\alpha}_{i,j}$ . Furthermore, we give an exact characterization of the properties that a classifier must satisfy so that  $B_{\mathrm{TPR}}$  and  $B_{\mathrm{FPR}}$  are indeed minimal. Our results demonstrate that, even when the proxy sensitive attributes are highly accurate, simply correcting for fairness with respect to these proxy attributes might be suboptimal in regards to minimizing the worst-case fairness violations. To this end, we present a simple post-processing method that can correct a pre-trained classifier  $f$  to yield an optimally corrected classifier,  $\bar{f}$ , i.e. one with minimal worst-case fairness violations.
Our experiments on both synthetic and real data illustrate our theoretical findings. We show how, even if the proxy sensitive attributes are highly accurate, the smallest imbalance in  $U_{0}$  and  $U_{1}$  renders the naive correction for fairness with respect to the proxy attributes suboptimal. More importantly, our experiments highlight our method's ability to effectively control the worst-case fairness violation of a classifier with minimal decrease in the classifier's overall predictive power. On a final observation on our empirical results, the reader might be tempted to believe that the classifier  $f_{\widehat{\text{fair}}}$  (referring to, e.g., Fig. 3) is better because it provides a lower "true" fairness than that of  $f_{\text{opt}}$ . Unfortunately, these true fairness violations are not identifiable in practice, and all one can compute are the provided upper bounds, which  $f_{\text{opt}}$  minimizes. In conclusion, our contribution aims to provide better and more rigorous control over potential negative societal impacts that arise from unfair machine learning algorithms in settings of unobserved data.
This work was supported in part by NIH award R01CA287422.
[35] Yaniv Romano, Stephen Bates, and Emmanuel Candes. Achieving equalized odds by resampling sensitive attributes. Advances in neural information processing systems, 33:361-371, 2020.
[36] Laleh Seyyed-Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine, 27(12):2176-2182, 2021.
[37] Li Shen, Laurie R Margolies, Joseph H Rothstein, Eugene Fluder, Russell McBride, and Weiva Sieh. Deep learning to improve breast cancer detection on screening mammography. Scientific reports, 9(1):12495, 2019.
[38] Kush R Varshney and Homa AleMZadeh. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. *Big data*, 5(3):246–255, 2017.
[39] Dennis Wei, Karthikeyan Natesan Ramamurthy, and Flavio P Calmon. Optimized score transformation for fair classification. Proceedings of Machine Learning Research, 108, 2020.
[40] Joel S. Weissman and Romana Hasnain-Wynia. Advancing health care equity through improved data collection. New England Journal of Medicine, 364, 2011.
[41] Xiaojiao Yu. Machine learning application in online lending risk prediction. arXiv preprint arXiv:1707.04831, 2017.
[42] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962-970. PMLR, 2017.
[43] Yan Zhang. Assessing fair lending risks using race/ethnicity proxies. Management Science, 64 (1):178-197, 2018.
[44] Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, pages 1433-1442, 2022.
A.1 Explanation of Assumption 1