Title: Estimating and Controlling for Equalized Odds via Sensitive Attribute Predictors
Venue: NeurIPS 2023
Accepted: yes
Keywords: fairness

sensitive attributes

equalized odds

missing data

proxies
Abstract: As the use of machine learning models in real world high-stakes decision settings continues to grow, it is highly important that we are able to audit and control for any potential fairness violations these models may exhibit towards certain groups. To do so, one naturally requires access to sensitive attributes, such as demographics, biological sex, or other potentially sensitive features that determine group membership. Unfortunately, in many settings, this information is often unavailable. In this work we study the well known equalized odds (EOD) definition of fairness. In a setting without sensitive attributes, we first provide tight and computable upper bounds for the EOD violation of a predictor. These bounds precisely reflect the worst possible EOD violation. Second, we demonstrate how one can provably control the worst-case EOD by a new post-processing correction method. Our results characterize when directly controlling for EOD with respect to the predicted sensitive attributes is -- and when is not -- optimal when it comes to controlling worst-case EOD. Our results hold under assumptions that are milder than previous works, and we illustrate these results with experiments on synthetic and real datasets.

Estimating and Controlling for Equalized Odds via Sensitive Attribute Predictors

Abstract

As the use of machine learning models in real world high-stakes decision settings continues to grow, it is highly important that we are able to audit and control for any potential fairness violations these models may exhibit towards certain groups. To do so, one naturally requires access to sensitive attributes, such as demographics, biological sex, or other potentially sensitive features that determine group membership. Unfortunately, in many settings, this information is often unavailable. In this work we study the well known equalized odds (EOD) definition of fairness. In a setting without sensitive attributes, we first provide tight and computable upper bounds for the EOD violation of a predictor. These bounds precisely reflect the worst possible EOD violation. Second, we demonstrate how one can provably control the worst-case EOD by a new post-processing correction method. Our results characterize when directly controlling for EOD with respect to the predicted sensitive attributes is - and when is not - optimal when it comes to controlling worst-case EOD. Our results hold under assumptions that are milder than previous works, and we illustrate these results with experiments on synthetic and real datasets.

1 Introduction

Machine learning (ML) algorithms are increasingly used in high stakes prediction applications that can significantly impact society. For example, ML models have been used to detect breast cancer in mammograms [37], inform parole and sentencing decisions [14], and aid in loan approval decisions [41]. While these algorithms often demonstrate excellent overall performance, they can be dangerously unfair and negatively impact under-represented groups [38]. Some of these unforeseen negative consequences can even be fatal, e.g. deep learning models for chest x-ray disease classification exhibit under diagnosis bias towards certain sensitive groups [36]. Recommendations to ensure that ML systems do not exacerbate societal biases have been raised by several groups, including the White House, which in 2016, released a report on big data, algorithms, and civil rights [31]. It is thus critical to understand how to rigorously evaluate the fairness of ML algorithms and control for unfairness during model development.

These needs have prompted considerable research in the area of Fair ML . While there exist many definitions of algorithmic fairness [7], common notions of group fairness consider different error rates of a predictor across different groups: males and females, white and non-white, etc. For example, the equal opportunity criterion requires the true positive rate (TPR) be equal across both groups, while equalized odds requires both TPR and false positive rate (FPR) to be the same across groups [21]. Obtaining predictors that are fair therefore requires enforcing these constraints on error rates across groups during model development, which can be posed as a constrained (or regularized) optimization problem [35, 42, 2, 13, 16]. Alternatively, one can devise post-processing strategies to modify a certain predictor to correct for differences in TPR and FPR [21, 18, 12, 1], or even include data

pre-processing steps that ensure that unfair models could not be obtained from such data to begin with [39, 8].

Naturally, all these techniques for estimating or enforcing fairness require access to a dataset with features, X , responses, Y , and sensitive attributes, A . However, in many settings this is difficult or impossible, as datasets often do not include samples that have all these variables. This could be because the sensitive attribute data was withheld due to privacy concerns, which is very common with medical data due to HIPAA federal law requirements, or simply because it was deemed unnecessary to record [40, 43]. A real-world example of this is in the recent Kaggle-hosted RSNA Chest X-Ray Pneumonia Detection Challenge [33]. Even though this dataset of chest x-rays was painstakingly annotated for pneumonia disease by dozens of radiologists, it did not include sensitive attributes (e.g., age, sex, and race), precluding the evaluation of fairness of the models developed as part of the challenge. In settings like this, where there is limited or no information on the sensitive attribute of interest, it is still important to be able to accurately estimate the violation of fairness constraints by a ML classifier and to be able to alleviate these violations before deploying it in a sensitive application

This leads to the natural question:

How can one assess and control the fairness of a classifier without having access to sensitive attribute data?

In other words, how can we measure and potentially control the fairness violations of a classifier for Y with respect to a sensitive attribute A , when we have no data that jointly observes A and Y ?

1.1 Related Work

Estimating unfairness and, more importantly, developing fair predictors where there is no - or only partial - information about the sensitive attribute has only recently received increasing attention. The recent work by Zhao et al. [44] explores the perspective of employing features that are correlated with the sensitive attribute, and shows that enforcing low correlation with such 'fairness related features' can lead to models with lower bias. Although these techniques are promising, they require domain expertise to determine which features are highly correlated with the sensitive attribute. An appealing alternative that has been studied is the use proxy sensitive attributes that are created by a second predictor trained on a different data set that contains only sensitive attribute information [20, 25, 10, 5]. This strategy has been widely adopted in many domains such as healthcare [17], finance [6], and politics [23]. While using sensitive attribute predictors has proven to be an effective and practical solution, it must be done with care, as this opens new problems for estimating and controlling for fairness. The work by Prost et al. [34] considers the estimation of fairness in a setting where one develops a predictor for an unobserved covariate, but it does not contemplate predicting the sensitive attribute itself. On the other hand Chen et al. [10] study the sources of error in the estimation of fairness via predicted proxies computed using threshold functions, which are prone to over-estimation.

The closest to our work are the recent results by Kallus et al. [25], and Awasthi et al. [5, 4]. Kallus et al. [25] study the identifiability of fairness violations under general assumptions on the distribution and classifiers. They show that, in the absence of the sensitive attribute, the fairness violation of predictions, ̂ Y , is unidentifiable unless strict assumptions 1 are made or if there is some common observed data over A and Y . Nonetheless, they show not all hope is lost and provide closed form upper and lower bounds of the fairness violations of ̂ Y under the assumption that one has two datasets: one that is drawn from the marginal over ( X,A ) and the other drawn from the marginal over ( X,Y, ̂ Y ) . Their analysis, however, does not consider predictors ̂ Y = f ( X ) or ̂ A = h ( X ) , and instead their bounds depend explicitly on the conditional probabilities, P ( A | X ) and P ( ̂ Y , Y | X ) , along with the distribution over the features, P ( X ) . Unfortunately, with this formulation, it is unclear how the bounds would change if the estimation of the conditional probabilities was inaccurate (as is the case when developing predictors ̂ A in the real world). Furthermore, in settings where X is high dimensional (as for image data), calculating such bounds would become intractable. Since the fairness violation cannot be directly modified, they then study when these bounds can be reduced and improved. However, they do so in settings that impose smoothness assumptions over ( X,A,Y, ̂ Y ) ,

which clearly are not-verifiable without data over the complete joint distribution. As a result, their results do not provide any actionable method that could improve the bounds.

Awasthi et al. [5], on the other hand, make progress in understanding properties of the sensitive attribute predictor, ̂ A , that are desirable for easy and accurate fairness violation estimation and control of a classifier ̂ Y . Assuming that ̂ Y ⊥ ⊥ ̂ A | ( A,Y ) , they demonstrate that the true fairness violation is in fact proportional to the estimated fairness violation (the fairness violation using ̂ A in lieu of A). This relationships yields the counter-intuitive result that given a fixed error budget for the sensitive attribute predictor, the optimal attribute predictor for the estimation of the true fairness violation is one with the most unequal distribution of errors across the subgroups of the sensitive attribute. However, one is still unable to actually calculate the true fairness violation - as it is unidentifiable. Nonetheless, the relationship does demonstrates that if one can maintain the assumption above while controlling for fairness with respect to ̂ A , then doing so will provably reduce the true fairness violation with respect to A . Unfortunately, while these rather strict assumptions can be met in some limited scenarios (as in [4]), these are not applicable in general - and cannot even be tested without access to data over ( A,Y ) .

Overall, while progress has been made in understanding how to estimate and control fairness violations in the presence of incomplete sensitive attribute information, these previous results highlight that this can only be done in simple settings (e.g., having access to some data from the entire distribution, or by making strong assumptions of conditional independence). Moreover, it remains unclear whether tight bounds can be obtained that explicitly depend on the properties of the predictor ̂ Y , allowing for actionable bounds that can provably mitigate for its fairness violation without having an observable sensitive attribute or making stringent assumptions.

1.2 Contributions

The contributions of our work can be summarized as follows:

We study the well known equalized odds (EOD) definition of fairness in a setting where the sensitive attributes, A , are not observed with the features X and labels Y . We provide tight and computable bounds on the EOD violation of a classifier, ̂ Y = f ( X ) . These bounds represent the worst-case EOD violation of f and employ a predictor for the sensitive attributes, ̂ A = h ( X ) , obtained from a sample over the distribution ( X,A ) .

We provide a precise characterization of the classifiers that achieve minimal worst-case EOD violations with respect to unobserved sensitive attributes. Through this characterization, we demonstrate when simply correcting for fairness with respect to the proxy sensitive attributes will yield minimal worst-case EOD violations, and when instead it proves to be sub-optimal.

We provide a simple and practical post-processing technique that provably yields classifiers that maximize prediction power while achieving minimal worst-case EOD violations with respect to unobserved sensitive attributes.

We illustrate our results on a series of simulated and real data of increasing complexity.

2 Problem Setting

We work within a binary classification setting and consider a distribution Q over ( X ×A×Y ) where X ⊆ R n is the feature space, Y = { 0 , 1 } the label space, and A = { 0 , 1 } the sensitive attribute space. Furthermore, and adopting the setting of [5], we consider 2 datasets, D 1 and D 2 . The former is drawn from the marginal over ( X , A ) of Q while D 2 is drawn from the marginal ( X , Y ) of Q . In this way, D 1 and D 2 contain the same set of features, D 1 contains sensitive attribute information and D 2 contains label information. The drawn samples in D 1 and D 2 are i.i.d over their respective marginals and different from one another.

Similar to previous work [10, 34, 5], we place ourselves in a demographically scarce regime where there is designer who has access to D 1 to train a sensitive attribute predictor h : X → A and a developer, who has access to D 2 , the sensitive attribute classifier h , and all computable probabilites P ( h ( X ) , A ) that the designer of h can extract from D 1 . In this setting, the goal of the developer is to learn a classifier f : X → Y (from D 2 ) that is fair with respect to A utilizing ̂ A . The central idea is

̸

̸

to augment every sample in D 2 , ( x i , y i ) , by ( x i , y i , ˆ a i ) , where ˆ a i = h ( x i ) . Intuitively, if the error of the sensitive attribute predictor, denoted herein by U = P ( h ( X ) = A ) , is low, we could hope that fairness with respect to the real (albeit unobserved) sensitive attribute can be faithfully estimated. Our goal is to thus estimate the error incurred in measuring and enforcing fairness constraints by means of ˆ A = h ( X ) , and potentially alleviate or control for it.

Throughout the remainder of this work we focus on equalized odds (EOD) as our fairness metric [21] of interest as it is one of the most popular notions of fairness. Thus moving forward, the term fairness refers specifically to EOD. We denote ˆ Y = f ( X ) for simplicity, and for i, j ∈ { 0 , 1 } define the group conditional probabilities

These probabilities quantify the TPR (when j = 1 ) and FPR (when j = 0 ), for either protected group ( i = 0 or i = 1 ). We assume that the base rates, r i,j = P ( A = i, Y = j ) > 0 so that these quantities are not undefined. With these conditionals probabilities, we define the true fairness violation of f , with the two quantities

which respectively quantify the absolute difference in TPR and FPRs among the two protected groups.

̸

We also need to characterize the performance of the sensitive attribute classifier, h . The missclassification error of h can be decomposed as, U = U 0 + U 1 where U i = P ( ̂ A = i, A = i ) , for i ∈ { 0 , 1 } . We define the difference in errors to be

In a demographically scarce regime, the rates r i,j , and more importantly the quantities of interest, ∆ TPR ( f ) and ∆ FPR ( f ) , cannot be computed because samples from A and Y are not jointly observed. However, using the sensitive attribute classifier h , we can predict ̂ A on D 2 and compute

which serve as the estimates for the true base rates and group TPRs and FPRs.

3 Theoretical Results

With the setting defined, we will now present our results. The first result provides computable bounds on the true fairness violation of f with respect to the true, but unobserved, sensitive attribute A . The bounds precisely characterize the worst-case fairness violation of f . Importantly, as we will explain later, this first result will provide insight into what properties f must satisfy so that its worst-case fairness violation is minimal . In turn, these results will lead to a simple post-processing method that can correct a pretrained classifier f into another one, ¯ f , that has minimal worst-case fairness violations. Before presenting our findings, we first describe the key underlying assumption we make about the pair of classifiers, h and f , so that the subsequent results are true.

̸

̸

To parse this assumption, it is easy to show that this is met when a) h is accurate enough for the setting, namely that P ( ̂ A = i, A = i ) ≤ 1 2 P ( ̂ A = i, Y = j ) , and b) the predictive power of h is better than the ability of f to predict the labels, Y - or more precisely, P ( ̂ A = i, A = i ) ≤ P ( ̂ Y = j, ̂ A = i, Y = j ) . We refer the reader to Appendix A for a thorough explanation for why this is true. While this assumption may seem limiting, this is milder than those in existing results: First, accurate predictors h can be developed [6, 17, 23, 19], thus satisfying the assumption (our numerical results will highlight this fact as well). Second, other works [5, 9], require assumptions on ̂ A and ̂ Y that are unverifiable in a demographically scarce regime. Our assumption, on the other hand, can always be easily verified because all the quantities are computable. Furthermore, this assumption can be also be relaxed if one desires partial guarantees. More specifically, if the above is true for i ∈ { 0 , 1 } and only for j = 1 , then all subsequent results for ∆ TPR hold. Similarly, if it only holds for j = 0 , the results for ∆ FPR hold. Therefore, if one only cares about the equal opportunity definition of fairness, as an example, then this only needs to hold for i ∈ { 0 , 1 } and j = 1 .

3.1 Bounding Fairness Violations with Proxy Sensitive Attributes

With Assumption 1 in place, we present our main result.

Theorem 1 (Bounds on ∆ TPR ( f ) and ∆ FPR ( f ) ) . Under Assumption 1, we have that

where

Furthermore, the upper bounds for | ∆ TPR ( f ) | and | ∆ FPR ( f ) | are tight.

The proof, along with all others in this work, are included in Appendix A. We now make a few remarks on this result. First, the bound is tight in that there exists settings (albeit unlikely) with particular marginal distributions such that the bounds hold with equality. Second, even though | ∆ TPR ( f ) | and | ∆ FPR ( f ) | cannot be calculated, a developer can still calculate the worst-case fairness violations, B TPR ( f ) and B FPR ( f ) , because these depend on quantities that are all computable in practice. Thus, if B TPR ( f ) and B FPR ( f ) are low, then the developer can proceed having a guarantee on the maximal fairness violation of f , even while not observing | ∆ TPR ( f ) | and | ∆ FPR ( f ) | . On the other hand, if these bounds are large, this implies a potentially large fairness violation over the protected group A by f and the developer should not proceed in deploying f and instead seek to learn another classifier with smaller bounds. Third, the obtained bounds are linear in the parameters ̂ α i,j , which the developer can adjust as they are properties of f : this will become useful shortly.

3.2 Optimal Worst-Case Fairness Violations

Given the result above, what properties should classifiers f satisfy such that B TPR ( f ) and B FPR ( f ) are minimal? Moreover, are the classifiers f that are fair with respect to ̂ A , the ones that have smallest B TPR ( f ) and B FPR ( f ) ? We now answer these questions in the following theorem.

Theorem 2 (Minimizers of B TPR ( f ) and B FPR ( f ) ) . Let ̂ A = h ( X ) be a fixed sensitive attribute classifier with errors U 0 and U 1 that produces rates ˆ r i,j = P ( ̂ A = i, Y = j ) . Let F be the set of all predictors of Y , parameterized by rates ̂ α i,j , that, paired with h , satisfy Assumption 1. Then, ∃ Y ∈ F with group conditional probabilities, ̂ α i,j = P ( Y = 1 | ̂ A = i, Y = j ) that satisfies the following condition,

Furthermore, any such

Y

has minimal maximal fairness violation, i.e.

∀

̂

Y

∈ F

,

̸

This result provides a precise characterization of the conditions that lead to minimal worst-case fairness violations. Observe that if ∆ U = 0 , the classifier with minimal B TPR ( f ) and B FPR ( f ) involves ̂ α i,j such that ̂ α 1 ,j = ̂ α 0 ,j , i.e. it is not fair with respect to ̂ A . On the other hand, if the errors of h are balanced ( ∆ U = 0 ), then minimal bounds are achieved by being fair with respect to ̂ A .

̸

3.3 Controlling Fairness Violations with Proxy Sensitive Attributes

Now that we understand what conditions f must satisfy so that it's worst case fairness violations are minimal, what remains is a method to obtain such a classifier. We take inspiration from the post-processing method proposed by Hardt et al. [21], which derives a classifier Y = ¯ f ( X ) from ̂ Y = f ( X ) that satisfies equalized odds with respect to a sensitive attribute A while minimizing an expected missclassification loss only applicable if one has access to A , which is not true in our setting. Nonetheless, since our method will generalize this idea, we first briefly comment on this

approach. The method they propose works as follows: given a sample with initial prediction ̂ Y = ˆ y and sensitive attribute A = a , the derived predictor ¯ f , with group conditional probabilities α i,j = P ( Y = 1 | A = i, Y = j ) , predicts Y = 1 with probability p a, ˆ y = P ( Y = 1 | A = a, ̂ Y = ˆ y ) . The four probabilities p 0 , 0 , p 0 , 1 , p 1 , 0 , p 1 , 1 can be then calculated so that Y satisfies equalized odds and the expected loss between Y and labels Y , i.e. E [ L ( Y , Y )] , is minimized. The fairness constraint, along with the objective to minimize the expected loss, give rise to the linear program:

Equalized Odds Post-Processing (Hardt et al. [21])

Returning to our setting where we do not have access to A but only proxy variables ̂ A = h ( X ) , we seek classifiers f that are fair with respect to the sensitive attribute A . Since these attributes are not available, (thus rendering the fairness violation to be unidentifiable [25]), a natural alternative is to minimize the worst-case fair violation with respect to A , which can be computed as shown in Theorem 1. Of course, such an approach will only minimize the worst case fairness and one cannot certify that the true fairness violation will decrease because - as explained above - it is unidentifiable. Nonetheless, since we know what properties optimal classifiers must satisfy (as per Theorem 2), we can now modify the above problem to construct a corrected classifier, ¯ f , as follows. First, we must employ ̂ A in place of A , which amounts to employing ̂ α i,j in lieu of α i,j . To this end, denote the (corrected) group conditional probabilities of Y to be ̂ α i,j . Second, the equalized odds constraint is replaced with the constraint in Theorem 2. Lastly, we also enforce the additional constraints detailed in Assumption 1 on the ̂ α i,j . With these modifications in place, we present the following generalized linear program:

Worst-case Fairness Violation Reduction

The solution to this linear program will yield a classifier ¯ f that satisfies Assumption 1, has minimal B TPR ( ¯ f ) and B FPR ( ¯ f ) , and has minimal expected loss. Note, that if ∆ U = 0 , then the coefficients of ̂ α 0 ,j and ̂ α 1 ,j will equal one, and so the first set of constraints simply reduces to ̂ α 0 ,j = ̂ α 1 ,j for j ∈ { 0 , 1 } and the linear program above is precisely the post-processing method of [21] with ̂ A in place of A (while enforcing Assumption 1).

Let us briefly recap the findings of this section: We have shown that in a demographically scarce regime, one can provide an upper bound on the true fairness violation of a classifier (Theorem 1). Second, we have presented a precise characterization of the classifiers with minimal worst-case fairness violations. Lastly, we have provided a simple and practical post-processing method (a linear program) that utilizes a sensitive attribute predictor to construct classifiers with minimal worst-case fairness violations with respect to the true, unknown, sensitive attribute.

4 Experimental Results

We now illustrate our theoretical results on synthetic and real world datasets. The code and data necessary to reproduce these experiments are available at https://github.com/Sulam-Group/ EOD-with-Proxies .

4.1 Synthetic Data

We begin with a synthetic example that allows us to showcase different aspects of our results. The data consists of 3 features, X 1 , X 2 , X 3 ∈ R , sensitive attribute A ∈ { 0 , 1 } , and response Y ∈ { 0 , 1 } .

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

0.25

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

f

f

fair

f

opt

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Fairness Violation

BFPR

FPR

(b)

∆

FPR and

B

FPR

f

f

fair

f

opt

0.145

0.150

0.155

0.160

0.165

Expected Loss

(c) Expected loss

The features are sampled from ( X 1 , X 2 , X 3 ) ∼ N ( µ, Σ) , with

The sensitive attribute, A , response Y , and classifier f , are modeled as

̸

̸

where I ( · ) is the indicator function, S ( · ) is the sigmoid function and c 1 , c 2 , c 3 ∼ N (1 , 0 . 01) . Furthermore, ϵ 0 ∼ N (0 , 1) , ϵ 1 ∼ N (0 , 0 . 5) are independent noise variables placed on the samples belonging to the groups A = 0 and A = 1 to ensure there is a non trivial fairness violation. Specifically, Var ( ϵ 0 ) > Var ( ϵ 1 ) makes f ( X ; c 1 , c 2 , c 3 ) unfair with respect to A = 0 . Lastly, to measure the predictive capabilities of f , we use the loss function, L ( ˆ Y = y, Y = y ) = P ( Y = y ) , as this maximizes the well known Youden's Index 2 .

4.1.1 Equal Errors ( ∆U = 0 )

We model the sensitive attribute predictor as

where δ ∼ N (0 , σ 2 ) . We choose σ 2 so that h ( X ) has a total error U ≈ 0 . 04 distributed so that ∆ U ≈ 0 with U 0 ≈ U 1 ≈ 0 . 02 . We generate 1000 classifiers f and for each one, calculate ∆ TPR, ∆ FPR, B TPR, B FPR, and E [ L ( f, Y )] . Then, on each f , we run the (naïve) equalized odds post processing algorithm to correct for fairness with respect ˆ A to yield a classifier f ̂ fair , and we also run our post-processing algorithm to yield an optimal classifier f opt. For both sets of classifiers we again calculate the same quantities.

The results in Fig. 1 present the true fairness violations, worst-case fairness violations, and expected loss for the 3 sets of different classifiers. Observe that both B TPR and B FPR are significantly lower for f ̂ fair and f opt and that these values for both sets of classifiers are approximately the same. This is expected as U 0 ≈ U 1 and so performing the fairness correction algorithm and our proposed algorithm amount to solving nearly identical linear programs. We also show the true fairness violations, ∆ TPR and ∆ FPR for all the classifiers to portray the gap between the bounds and the true values. As mentioned before, these true values cannot be calculated in a real demographically scarce regime. Nonetheless, the developer of f now knows, post correction, that | ∆ TPR | , | ∆ FPR | ≲ 0 . 15 . Lastly, observe that the expected loss for f ̂ fair and f opt are naturally higher compared to that of f , however the increase in loss is minimal.

̸

4.1.2 Unequal Errors ( ∆U = 0 )

We model the sensitive attribute predictor in the same way as in the previous experiment except with δ = c , for a constant c so that the sensitive attribute classifier still has the same total error of

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

Fairness Violation

BFPR

FPR

(b)

∆

FPR and

B

FPR

f

f

fair

f

opt

0.140

0.145

0.150

0.155

0.160

0.165

0.170

Expected Loss

(c) Expected loss

̸

U ≈ 0 . 04 but distributed unevenly so that ∆ U = -0 . 04 with U 0 = 0 and U 1 ≈ 0 . 04 . As in the previous experiment we generate classifiers f , and perform the same correction algorithms to yield f ̂ fair and f opt and present the same metrics as before.

The results in Fig. 2 depict the worst-case fairness violations and expected loss for the 3 sets of different classifiers. Observe that our correction algorithm yields classifiers, f opt, that have significantly lower B TPR and B FPR. Furthermore, observe that the f ̂ fair that results from performing the naïve fairness correction algorithm in fact have higher B FPR the original classifier f ! Even though the total error U has remained the same, its imbalance showcases the optimality of our correction method. Lastly, observe that there is a trade-off, albeit slight, in performing our correction algorithm. The expected loss for f opt is higher than that of f ̂ fair and f .

4.2 Real World Data

We will now showcase our results on various real world datasets and prediction tasks. We provide a brief description of each task below and further experimental details are included in Appendix B.

FIFA 2020 (Awasthi et al. [5]) : The task is to learn a classifier f , using FIFA 2020 player data [29], that determines if a soccer players wage is above ( Y = 1 ) or below ( Y = 0) the median wage based on the player's age and their overall attribute. The sensitive attribute A is player nationality and the player's name is used to learn the sensitive attribute predictor h . We consider two scenarios, when A ∈ { English , Argentine } ( English = 0) and A ∈ { French , Spanish } ( French = 0).

ACSPublicCoverage (Ding et al. [15]) : The task is to learn a classifier f , using the 2018 state census data, that determines if a low-income individual, not eligible for Medicare, has coverage from public health insurance ( Y = 1) or does not ( Y = 0) . The sensitive attribute A is sex ( Female = 0) and with a separate dataset (containing the same features used to learn f (disregarding sex )), we learn the sensitive attribute predictor h . We work with the 2018 California census data.

CheXpert (Irvin et al. [24]) : CheXpert is a large public dataset for chest radiograph interpretation, consisting of 224,316 chest radio graphs of 65,240 patients, with labeled annotations for 14 observations (positive, negative, or unlabeled) including cardiomegaly, atelectasis,consolidation , and several others. The task is to learn a classifier f to determine if an X-ray contains an annotation for any abnormal condition ( Y = 1) or does not ( Y = 0) . The sensitive attribute A is sex ( Female = 0) and with a separate set of X-rays (different from those used to learn f ) we learn the sensitive attribute predictor h .

4.2.1 Verification of Assumption 1

In Appendix C we provide Table 1, which demonstrates that Assumption 1 holds for the settings described above. The Actual Value column of Table 1 lists the rates ̂ α i,j and the left and right columns list U i ˆ r i,j and 1 -U i ˆ r i,j respectively. From Table 1, it is clear that the ̂ α i,j lie in between U i ˆ r i,j and 1 -U i ˆ r i,j as required by Assumption 1. Notice, we only list the estimated group TPRs, ̂ α i, 1 for the ACSPublicCoverage 2018 California dataset. This is because Assumption 1, for j = 0 , does not hold for this state: even though ˆ A is a very accurate predictor for A , the label classifier f has very low estimated group FPRs, ̂ α i, 0 . As a result, our theoretical results for ∆ FPR cannot be used.

f

f

fair

f

opt

0.00

0.02

0.04

0.06

0.08

0.10

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Fairness Violation

BFPR

FPR

(b)

∆

FPR and

B

FPR

f

f

fair

f

opt

0.054

0.056

0.058

0.060

0.062

Expected Loss

(c) Expected loss

4.2.2 Results on CheXpert

The task is to measure and correct for any fairness violations that f may exhibit towards the sex attribute assuming we do not have the true sex attribute and instead have a sensitive attribute predictor, h . On a test dataset, we generate our predictions ˆ Y = f and ˆ A = h to yield a dataset over ( ˆ A,Y, ˆ Y ) , for which the sex predictor, h , achieves an error of U = 0 . 023 with U 0 ≈ 0 . 008 and U 1 ≈ 0 . 015 . We utilize the bootstrap method to generate 1000 samples of this dataset and for each sample, perform the same correction algorithms as before to yield f ̂ fair and f opt and calculate the same metrics as done in the previous experiments.

The results in Fig. 3 show that our proposed correction method performs the best in reducing B TPR and B FPR. Even though U is very small, since U 1 ≈ 2 U 0 , simply correcting for fairness with respect to ˆ A is suboptimal in reducing the worst-case fairness violations. In particular, the results in Fig. 3 a are noteworthy as they depict how our proposed correction method and bounds allow the user to certify that the obtained classifier has a fairness violation in TPRs of no more than 0.06 without having access to the true sensitive attributes. Moreover, the improvement is significant, since before the correction one had | ∆ TPR | ≲ 0 . 10 . In a high-stakes decision setting, such as this one where the model f could be used to aid in diagnosis, this knowledge could be vital. Naturally, the expected loss is highest for f opt but that the increase is minimal. We make no claim as to whether this (small) increase in loss is reasonable for this particular problem setting, and the precise trade-offs must be defined in the context of a broader discussion involving policy makers, domain experts and other stakeholders.

5 Limitations and Broader Impacts

While our results are novel and informative, they come with limitations. First, our results are limited to EOD (and its relaxations) as definitions of fairness. Fairness is highly context-specific and in many scenarios one may be interested in utilizing other definitions of fairness. One can easily extend our results to other associative definitions of fairness, such as demographic parity, predictive parity, and others. However, extending our results to counter-factual notions of fairness [28, 11, 32] is non trivial and matter of future work. We recommend thoroughly assessing the problem and context in question prior to selecting a definition. It is crucial to ensure that the rationale behind choosing a definition is based on reasoning from both philosophical and political theory, as each definition implicitly make a distinct set of moral assumptions. For example, with EOD, we implicitly assert that all individuals with the same true label have the same effort-based utility [22]. More generally, other statistical definitions of fairness such as demographic parity and equality of accuracy can be thought of as special instances of Rawlsian equality of opportunity and predictive parity, the other hand, can be thought of as an instance of egalitarian equality of opportunity [22, 30, 3]. We refer the reader to [22] to understand the relationship between definitions of fairness in machine learning and models of Equality of opportunity (EOP) - an extensively studied ideal of fairness in political philosophy.

A second limitation of our results is Assumption 1. This assumption is relatively mild, as it is met for accurate proxy sensitive attributes (as illustrated in the chest X-rays study). Yet, we conjecture that one can do away with this assumption and consider less accurate proxy sensitive attributes with the caveat that the worst case fairness violations will no longer be linear in the TPRs and FPRs. Thus, the

characterization of the classifiers with minimal worst-case bounds would be more involved and the method to minimize these violations will likely be more difficult. Furthermore, while we proposed a simple post-processing correction method, it would be of interest to understand how one could train a classifier - from scratch - to have minimal violations. Lastly, in our setting we assume the sensitive attribute predictor and label classifier are trained on marginal distributions from the same joint distribution. As a next step, it would be important to understand how these results extend to settings where these marginal distributions come from (slightly) different joint distributions. All of this constitutes matter of future work.

Finally, we would like to remark the positive and potentially negative societal impacts of this work. Our contribution is focused on a solution to a technical problem - estimating and correcting for fairness violations when the sensitive attribute and responses are not jointly observed. However, we understand that fairness is a complex and multifaceted issue that extends beyond technical solutions and, more importantly, that there can be disconnect between algorithmic fairness and fairness in a broader socio-technical context. Nonetheless, we believe that technical contributions such as ours can contribute to the fair deployment of machine learning tools. In regards to the technical contribution itself, our results rely on predicting missing sensitive attributes. While such a strategy could be seen as controversial - e.g. because it could involve potential negative consequences such as the perpetuation of discrimination or violation of privacy - this is necessary to build classifiers with minimal worst-case fairness violations in a demographically scarce regime. On the one hand, not allowing for such predictions could be seen as one form of 'fairness through unawareness', which has been proven to be an incorrect and misleading strategy in fairness [21, 10]. Moreover, our postprocessing algorithm, similar to that of Hardt et al. [21], admits implementations in a differentially private manner as well, since it only requires aggregate information about the data. As a result, our method, which uses an analogous formulation with different constraints, can also be carried out in a manner that preserves privacy. Lastly, note that if one does not follow our approach of correcting for the worst-case fairness by predicting the sensitive attributes, other models trained on this data can inadvertently learn this sensitive attribute indirectly and base decisions of it with negative and potentially grave consequences. Our methodology prevents this from happening by appropriately correcting models to have minimal worst-case fairness violations.

6 Conclusion

In this paper we address the problem of estimating and controlling potential EOD violations towards an unobserved sensitive attribute by means of predicted proxies. We have shown that under mild assumptions (easily satisfied in practice, as demonstrated) the worst-case fairness violations, B TPR and B FPR, have simple closed form solutions that are linear in the estimated group conditional probabilities ̂ α i,j . Furthermore, we give an exact characterization of the properties that a classifier must satisfy so that B TPR and B FPR are indeed minimal. Our results demonstrate that, even when the proxy sensitive attributes are highly accurate, simply correcting for fairness with respect to these proxy attributes might be suboptimal in regards to minimizing the worst-case fairness violations. To this end, we present a simple post-processing method that can correct a pre-trained classifier f to yield an optimally corrected classifier, ¯ f , i.e. one with with minimal worst-case fairness violations.

Our experiments on both synthetic and real data illustrate our theoretical findings. We show how, even if the proxy sensitive attributes are highly accurate, the smallest imbalance in U 0 and U 1 renders the naïve correction for fairness with respect to the proxy attributes suboptimal. More importantly, our experiments highlight our method's ability to effectively control the worst-case fairness violation of a classifier with minimal decrease in the classifier's overall predictive power. On a final observation on our empirical results, the reader might be tempted to believe that the classifier f ̂ fair (referring to, e.g., Fig. 3) is better because it provides a lower 'true' fairness than that of f opt. Unfortunately, these true fairness violations are not identifiable in practice, and all one can compute are the provided upper bounds, which f opt minimizes. In conclusion, our contribution aims to provide better and more rigorous control over potential negative societal impacts that arise from unfair machine learning algorithms in settings of unobserved data.

Acknowledgments and Disclosure of Funding

This work was supported in part by NIH award R01CA287422.

References

Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. Auditing black-box models for indirect influence. Knowledge and Information Systems , 54:95-122, 2018.

Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference on machine learning , pages 60-69. PMLR, 2018.

Larry A Alexander. Fair equality of opportunity: John rawls'(best) forgotten principle. Philosophy research archives , 11:197-208, 1985.

Pranjal Awasthi, Matthäus Kleindessner, and Jamie Morgenstern. Equalized odds postprocessing under imperfect group information. In International conference on artificial intelligence and statistics , pages 1770-1780. PMLR, 2020.

Pranjal Awasthi, Alex Beutel, Matthäus Kleindessner, Jamie Morgenstern, and Xuezhi Wang. Evaluating fairness of machine learning models under uncertain and incomplete information. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency , pages 206-214, 2021.

Arthur P Baines and Marsha J Courchane. Fair lending: Implications for the indirect auto finance market. Study Prepared for the American Financial Services Association , 2014.

Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning: Limitations and Opportunities . fairmlbook.org, 2019. http://www.fairmlbook.org .

Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. Optimized pre-processing for discrimination prevention. Advances in neural information processing systems , 30, 2017.

L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Fair classification with noisy protected attributes: A framework with provable guarantees. In International Conference on Machine Learning , pages 1349-1361. PMLR, 2021.

Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. Fairness under unawareness: Assessing disparity when protected class is unobserved. In Proceedings of the conference on fairness, accountability, and transparency , pages 339-348, 2019.

Silvia Chiappa. Path-specific counterfactual fairness. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 7801-7808, 2019.

Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pontil. Leveraging labeled and unlabeled data for consistent fair binary classification. Advances in Neural Information Processing Systems , 32, 2019.

Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You. Training well-generalizing classifiers for fairness metrics and other data-dependent constraints. In International Conference on Machine Learning , pages 1397-1405. PMLR, 2019.

Swarup Dhar, Vanessa Massaro, Darakhshan Mir, and Nathan C Ryan. Uncertainty in criminal justice algorithms: simulation studies of the pennsylvania additive classification tool. arXiv preprint arXiv:2112.00301 , 2021.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring adult: New datasets for fair machine learning. Advances in neural information processing systems , 34:6478-6490, 2021.

Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. Advances in neural information processing systems , 31, 2018.

Marc N Elliott, Peter A Morrison, Allen Fremont, Daniel F McCaffrey, Philip Pantoja, and Nicole Lurie. Using the census bureau's surname list to improve estimates of race/ethnicity and associated disparities. Health Services and Outcomes Research Methodology , 9:69-83, 2009.

Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 259-268, 2015.

Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy, John L Burns, Leo Anthony Celi, Li-Ching Chen, Ramon Correa, Natalie Dullerud, Marzyeh Ghassemi, Shih-Cheng Huang, et al. Ai recognition of patient race in medical imaging: a modelling study. The Lancet Digital Health , 4(6):e406-e414, 2022.

Maya Gupta, Andrew Cotter, Mahdi Milani Fard, and Serena Wang. Proxy fairness, 2018.

Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems , 29, 2016.

Hoda Heidari, Michele Loi, Krishna P Gummadi, and Andreas Krause. A moral framework for understanding fair ml through economic models of equality of opportunity. In Proceedings of the conference on fairness, accountability, and transparency , pages 181-190, 2019.

Kosuke Imai and Kabir Khanna. Improving ecological inference by predicting individual ethnicity from voter registration records. Political Analysis , 24(2):263-272, 2016.

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 590-597, 2019.

Nathan Kallus, Xiaojie Mao, and Angela Zhou. Assessing algorithmic fairness with unobserved protected class using data combination. Management Science , 68(3):1959-1981, 2022.

Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT , volume 1, page 2, 2019.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA, 2015.

Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems , 30, 2017.

Stefano Leone. Fifa 21 complete player dataset, Oct 2020. URL https://www.kaggle.com/ datasets/stefanoleone992/fifa-21-complete-player-dataset .

Andrew Mason. Levelling the playing field: The idea of equal opportunity and its place in egalitarian thought . OUP Oxford, 2006.

Cecilia Munoz, DJ Patil, and Megan Smith. Big data: A report on algorithmic systems, opportunity, and civil rights . Executive Office of the President, 2016.

Razieh Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Radiological Society of North America. Rsna pneumonia detection challenge, kaggle competition, 2018.

Flavien Prost, Pranjal Awasthi, Nick Blumm, Aditee Kumthekar, Trevor Potter, Li Wei, Xuezhi Wang, Ed H Chi, Jilin Chen, and Alex Beutel. Measuring model fairness under noisy covariates: A theoretical perspective. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages 873-883, 2021.

Yaniv Romano, Stephen Bates, and Emmanuel Candes. Achieving equalized odds by resampling sensitive attributes. Advances in neural information processing systems , 33:361-371, 2020.

Laleh Seyyed-Kalantari, Haoran Zhang, Matthew BA McDermott, Irene Y Chen, and Marzyeh Ghassemi. Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine , 27(12):2176-2182, 2021.

Li Shen, Laurie R Margolies, Joseph H Rothstein, Eugene Fluder, Russell McBride, and Weiva Sieh. Deep learning to improve breast cancer detection on screening mammography. Scientific reports , 9(1):12495, 2019.

Kush R Varshney and Homa Alemzadeh. On the safety of machine learning: Cyber-physical systems, decision sciences, and data products. Big data , 5(3):246-255, 2017.

Dennis Wei, Karthikeyan Natesan Ramamurthy, and Flavio P Calmon. Optimized score transformation for fair classification. Proceedings of Machine Learning Research , 108, 2020.

Joel S. Weissman and Romana Hasnain-Wynia. Advancing health care equity through improved data collection. New England Journal of Medicine , 364, 2011.

Xiaojiao Yu. Machine learning application in online lending risk prediction. arXiv preprint arXiv:1707.04831 , 2017.

Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics , pages 962-970. PMLR, 2017.

Yan Zhang. Assessing fair lending risks using race/ethnicity proxies. Management Science , 64 (1):178-197, 2018.

Tianxiang Zhao, Enyan Dai, Kai Shu, and Suhang Wang. Towards fair classifiers without sensitive attributes: Exploring biases in related features. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining , pages 1433-1442, 2022.

A Proofs

A.1 Explanation of Assumption 1

Assumption 1: For i, j ∈ { 0 , 1 } , the classifiers ̂ Y = f ( X ) and ̂ A = h ( X ) satisfy

We now expand on the implications of this assumptions. Recall that U i = P ( ̂ A = i, A = i ) , ˆ r i,j = P ( ̂ A = i, Y = j ) , and ̂ α i,j = P ( ̂ Y = 1 | ̂ A = i, Y = j ) . Thus Assumption 1 states,

̸

̸

Immediately, it is clear that Eq. (10) implies

̸

̸

̸

Now, the left inequality of Eq. (10) states

̸

and the right inequality of Eq. (10) states

which implies

̸

̸

̸

If j = 1 , then this implies and if j = 0 ,

̸

̸

̸

̸

Any reasonable classifier ̂ Y would have the properties

Thus, Assumption 1 is met when

̸

̸

A.2 Proof of Theorem 1

We only prove the result for | ∆ TPR ( f ) | as the proof for | ∆ FPR ( f ) | is completely analogous.

Proof. The rules of conditional probability and the law of total probability allow us to decompose α 1 , 1 and α 0 , 1 in the following manner,

and

Therefore, ∆ TPR ( f ) = α 1 , 1 -α 0 , 1 is a function of the four probabilities given by

which are unidentifiable in a demographically scarce regime and therefore not computable.

The Fréchet inequalities tell us that for i, j ∈ { 0 , 1 }

Observe that, ∆ TPR ( f ) is an increasing function with respect to the two probabilities

and a decreasing one with respect to the two probabilities,

As a result, ∆ TPR ( f ) is maximal when P ( ˆ Y = 1 , A = 1 , Y = 1 | ˆ A = i ) achieve their maximum values and P ( ˆ Y = 0 , A = 1 , Y = 1 | ˆ A = i ) achieve their minimum values. On the other hand, ∆ TPR ( f ) is minimal when P ( ˆ Y = 1 , A = 1 , Y = 1 | ˆ A = i ) achieve their minimum values and P ( ˆ Y = 0 , A = 1 , Y = 1 , | ˆ A = i ) achieve their maximum values. With these facts, we now provide the upper bound. Recall from Appendix A.1 that Assumption 1 implies

̸

̸

̸

(38)

With Assumption 1 we first provide the values of min [ P ( ˆ Y = 0 , A = 1 , Y = 1 , | ˆ A = i )] . First, P ( ˆ Y = 0 , Y = 1 , | ˆ A = 1) -P ( A = 0 | ˆ A = 1) = P ( ˆ Y = 0 , ˆ A = 1 , Y = 1) P ( ˆ A = 1) -U 1 P ( ˆ A = 1) (39) ≥ 0 (40) because P ( ˆ Y = 0 , ˆ A = 1 , Y = 1) -U 1 ≥ 0 . Second, P ( ˆ Y = 0 , Y = 1 , | ˆ A = 0) -P ( A = 0 | ˆ A = 0) = P ( ˆ Y = 0 , ˆ A = 0 , Y = 1) P ( ˆ A = 0) -P ( A = 0 , ˆ A = 0) P ( ˆ A = 0) (41) = P ( ˆ Y = 0 , ˆ A = 0 , Y = 1) P ( ˆ A = 0) -P ( ˆ A = 0) -U 0 P ( ˆ A = 0) (42) = P ( ˆ Y = 0 , ˆ A = 0 , Y = 1) P ( ˆ A = 0) -P ( ˆ A = 0) -U 0 P ( ˆ A = 0) (43) = P ( ˆ Y = 0 , ˆ A = 0 , Y = 1) P ( ˆ A = 0) + U 0 P ( ˆ A = 0) -1 (44)

Now note that,

where the second equality is due to Assumption 1. As a result

Therefore,

Now we provide the values of max [ P ( ˆ Y = 1 , A = 1 , Y = 1 , | ˆ A = i )] . First,

Second,

Therefore,

Plugging these 4 values into ∆ TPR will yield the upper bound,

One can similarly use the assumptions to derive the lower bound,

and thus | ∆ TPR | ≤ max {| B 1 + C 0 , 1 | , | B 1 -C 1 , 1 |} . One can use same arguments to derive the upper bound for | ∆ FPR | .

A.3 Proof of Theorem 2

We prove the result for | ∆ TPR ( f ) | . We first start by proving the existence portion of the theorem.

Let ̂ A = h ( X ) be a sensitive attribute classifier with errors U 0 and U 1 that produces rates ˆ r i,j = P ( ̂ A = i, Y = j ) . Let F be the set of classifiers for Y such that ∀ f ∈ F , f and h satisfy Assumption 1. Consider any f ∈ F with group conditional probabilities, ̂ α i,j = P ( ̂ Y = 1 | ̂ A = i, Y = j ) . Since we are only proving the result for | ∆ TPR ( f ) | , set j = 1 . Consider the xy plane, with the x -axis being ̂ α 0 , 1 and the y -axis being ̂ α 1 , 1 . We know,

The two equations above define a rectangular region in the xy plane with a center ( 1 2 , 1 2 ) , meaning any classifier f ∈ F , has ̂ α i,j that are in this region.

Now, denote ¯ F to be a the set of classifiers for Y , with group conditional probabilities ̂ α i, 1 , that satisfy the condition,

This condition defines a line in the xy plane meaning any classifier in ¯ F has ̂ α i, 1 that are on this line. Now observe that the classifier ¯ f ∈ ¯ F with ̂ α i, 1 = 1 2 , satisfy the above condition because,

This implies that the line defined by Eq. (63) intersects the rectangular region that Assumption 1 defines. As a result, F ∩ ¯ F is not empty, meaning there exists a classifier ¯ f ∈ F with group conditional probabilities ̂ α i, 1 that also satisfies the condition,

Now we prove that such a classifier has minimal bounds. Theorem 1 tells us that for f ∈ F

Note that B 1 is linear in ̂ α 1 , 1 and ̂ α 0 , 1 and that C 0 , 1 and C 1 , 1 are constants such that B 1 + C 0 , 1 ≥ B 1 -C 1 , 1 simply because B 1 + C 0 , 1 is the upper bound for ∆ TPR and B 1 -C 0 , 1 is the lower bound. Since these bounds are affine functions shifted by a constant, then minmax {| B 1 + C 0 , 1 | , | B 1 -C 1 , 1 |} necessarily occurs when

have minimal upper bounds on | ∆ TPR | . After rearranging terms, this condition is precisely

B Experimental Details

FIFA 2020 (Awasthi et al. [5]) : The task is to learn a classifier f , using FIFA 2020 player data [29], that determines if a soccer players wage is above ( Y = 1 ) or below ( Y = 0) the median wage based on the player's age and their overall attribute. The sensitive attribute A is player nationality and the player's name is used to learn the sensitive attribute predictor h . We consider two scenarios, when A ∈ { English , Argentine } ( English = 0) and A ∈ { French , Spanish } ( French = 0). To learn the sensitive attribute predictor h , we train a Bidirectional Encoder Representations from Transformers (BERT) model [26] using an Adam optimizer [27] for 5 epochs. To learn the label classifier f , we train a Random Forest classifier. On a test dataset, we generate our predictions ˆ Y = f ( X ) and ˆ A = h ( X ) to yield a dataset over ( ˆ A,Y, ˆ Y ) . We utilize the bootstrap method to generate 1,000 samples of this dataset and, for each sample, perform the same correction algorithms as before to yield f ̂ fair and f opt and calculate the same metrics as done in the synthetic data experiments. For A ∈ { English , English } the nationality predictor, h , achieves an error of U = 0 . 033 with U 0 ≈ 0 . 008 and U 1 ≈ 0 . 015 . For A ∈ { French , Spanish } the nationality predictor, h , achieves an error of U = 0 . 053 with U 0 ≈ 0 . 025 and U 1 ≈ 0 . 028 . The results for both experiments are displayed in Fig. C.1 and Fig. C.2. In addition, we include Fig. C.1d, Fig. C.1e, Fig. C.2d, and Fig. C.2e, which display the difference between B TPR ( f ̂ fair ) and B TPR ( f opt ) and between B FPR ( f ̂ fair ) and B FPR ( f opt ) . These figures depict that the bounds of f opt are smaller than the bounds of f ̂ fair because the histograms of B TPR ( f ̂ fair ) -B TPR ( f opt ) and B FPR ( f ̂ fair ) -B FPR ( f opt ) are strictly to the right of the dashed green vertical line at 0 , indicating that both differences are greater than or equal to 0.

ACSPublicCoverage (Ding et al. [15]) : The task is to learn a classifier f , using the 2018 state census data, that determines if a low-income individual, not eligible for Medicare, has coverage from public health insurance ( Y = 1) or does not ( Y = 0) . The sensitive attribute A is sex ( Female = 0) and with a separate dataset (containing the same features used to learn f (disregarding sex )), we learn the sensitive attribute predictor h . We work with the 2018 California census data. To learn the sensitive attribute predictor h and label classifier f , we train two Random Forest classifiers. On a test dataset, we generate our predictions ˆ Y = f ( X ) and ˆ A = h ( X ) to yield a dataset over ( ˆ A,Y, ˆ Y ) . We utilize the bootstrap method to generate 1,000 samples of this dataset and, for each sample, perform the same correction algorithms as before to yield f ̂ fair and f opt and calculate the same metrics as done in the previous experiments. The sex predictor, h , achieves an error of U = 0 . 07 with U 0 ≈ 0 . 05 and U 1 ≈ 0 . 02 . The results for this experiment are in Fig. C.3. In addition, we include Fig. C.3b, which displays the difference between B TPR ( f ̂ fair ) and B TPR ( f opt ) . This figures depict that the TPR bounds of f opt are smaller than the TPR bounds of f ̂ fair because the histogram of B TPR ( f ̂ fair ) -B TPR ( f opt ) is strictly to the right of the dashed green vertical line at 0 , indicating that this difference is greater than 0.

CheXpert (Irvin et al. [24]) : CheXpert is a large public dataset for chest radiograph interpretation, consisting of 224,316 chest radio graphs of 65,240 patients, with labeled annotations for 14 observations (positive, negative, or unlabeled) including cardiomegaly, atelectasis,consolidation , meaning

and several others. The task is to learn a classifier f to determine if an X-ray contains an annotation for any abnormal condition ( Y = 1) or does not ( Y = 0) . The sensitive attribute A is sex ( Female = 0) and with a separate set of X-rays (different from those used to learn f ) we learn the sensitive attribute predictor h . To learn both f and h we use a DenseNet121 convolutional neural network architecture. Images are fed into the network with size 320 × 320 pixels. We use the Adam optimizer [27] with default β -parameters of β 1 = 0 . 9 , β 2 = 0 . 999 and a fixed learning rate of 1 × 10 -4 . Batches are sampled using a fixed batch size of 16 images and we train for 5 epochs. On a test dataset, we generate our predictions ˆ Y = f and ˆ A = h to yield a dataset over ( ˆ A,Y, ˆ Y ) . We utilize the bootstrap method to generate 1,000 samples of this dataset, and for each sample, perform the same correction algorithms as before to yield f ̂ fair and f opt and calculate the same metrics as done in the previous experiments. The sex predictor, h , achieves an error of U = 0 . 023 with U 0 ≈ 0 . 008 and U 1 ≈ 0 . 015 . The results for this experiment are in Fig. 3.

C Additional Figures & Results

̂

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

f

f

fair

f

opt

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Fairness Violation

BFPR

FPR

(b)

∆

FPR and

B

FPR

f

f

fair

f

opt

0.06

0.08

0.10

0.12

0.14

Expected Loss

(c) Expected loss

0.00

0.02

0.04

0.06

0.08

BTPR

(

f

fair

)

BTPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Probability

(d) Difference between

B

TPR

(

f

̂

fair

)

and

B

TPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

BFPR

(

f

fair

)

BFPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

Probability

(e) Difference between

B

FPR

(

f

̂

fair

)

and

B

FPR

(

f

opt

)

f

f

fair

f

opt

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

f

f

fair

f

opt

0.0

0.1

0.2

0.3

0.4

Fairness Violation

BFPR

FPR

(b)

∆

FPR and

B

FPR

f

f

fair

f

opt

0.04

0.05

0.06

0.07

0.08

0.09

0.10

0.11

0.12

Expected Loss

(c) Expected loss

0.00

0.02

0.04

0.06

0.08

0.10

BTPR

(

f

fair

)

BTPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

Probability

(d) Difference between

B

TPR

(

f

̂

fair

)

and

B

TPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

BFPR

(

f

fair

)

BFPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

Probability

(e) Difference between

B

FPR

(

f

̂

fair

)

and

B

FPR

(

f

opt

)

f

f

fair

f

opt

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Fairness Violation

BTPR

TPR

(a)

∆

TPR and

B

TPR

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

BTPR

(

f

fair

)

BTPR

(

f

opt

)

0.00

0.02

0.04

0.06

0.08

0.10

Probability

(b) Difference between

B

TPR

(

f

̂

fair

)

and

B

TPR

(

f

opt

)

f

f

fair

f

opt

0.166

0.168

0.170

0.172

0.174

Expected Loss

(c) Expected loss

Peer reviews:
comments: SUMMARY:
This work provides tight and computable bounds on the EOD violation of a classifier in a setting where the sensitive attributes are unknown. In addition, a post-processing technique is proposed to provably yield classifiers that maximize prediction power while achieving minimal worst-case EOD violations with respect to unobserved sensitive attributes. Experiments are done on both synthetic and real data.

STRENGTHS:
Significance: The problem under study is interesting and important. It would be useful to be able reduce the upper bound of certain fairness violation in models when sensitive attributes are unknown.

Originality and quality: the proposed solution is novel and technically sound.

Clarity: the presentation is clear with limitations analyzed in details.

WEAKNESSES:
1. One assumption of the proposed approach is that there exists a dataset of (X,A) to learn $\hat{A} = h(X)$ from. This could limit the use of the approach and should be discussed.

2. It would be beneficial to demonstrate the verification of Assumption 1 on the two datasets.

QUESTIONS:
Q1: can you elaborate on the potential bad outcome of applying $f_{opt}$ yields a less accurate and more biased (in terms of $\Delta_{FPR}$) model as in Figure 3?

Q2: Is that possible to use a different dataset of similar features (but not the same features as in the experiments) to learn $\hat{A} = h(X)$ (in some form of transfer learning)?

LIMITATIONS:
Results on the CheXpert data shows that, although the proposed method successfully reduces the worst case scenario TPR and FPR, it may actually increase the true TPR or FPR. Figure 3b shows an increase of $\Delta_{FPR}$ of $f_{opt}$ over $f$. This should be discussed as the potential outcome of applying $f_{opt}$ can be a less accurate and more biased model.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the fairness problem in machine learning when sensitive attributes (e.g., demographics and gender) are unavailable for practitioners. The authors focus on the Equalized-Odds (EOD), a well-known definition of fairness for classifiers. The authors provide bounds for EOD violation in a setting where sensitive attributes are unavailable. Additionally, they propose a post-processing method to control the worst-case EOD. Finally, the authors illustrate their results via synthetic and real-world datasets experiments.\n\nSTRENGTHS:\n* The paper is very well written. Particularly the related work section is very informative.\n\n* The authors explore an interesting problem for the community and propose a different view of the problem, i.e., a worst-case optimization. \n\n* The theoretical guarantees provide a method for practitioners to access the quality of the method (assuming a worst-case scenario).\n\n* The proposed method is simple, intuitive, and computationally inexpensive.\n\nWEAKNESSES:\n* The evaluation in the main paper is limited. The authors only apply their method to one real-world example. I suggest adding at least two more examples and as many examples as possible in the appendix. These examples can convince us that: (i) Assumption 1 is followed in multiple scenarios and (ii) your method helps to improve fairness. \n\n*  The fact that the Naïve approach had a better \"true\" EOD is interesting, and it might indicate that your method is not very efficient, i.e., using the sharp upper bound as a proxy for the true value is too pessimistic. Performing more experiments in real-world datasets can help the authors and readers understand when minimizing the upper bound is better than the naïve approach. \n\n* It needs to be clarified how to extend the approach to a setting where the sensitive attribute is not binary. In the introduction, the authors list examples of sensitive attributes as \"demographics and gender,\" which are not binary. A subsection is recommended to discuss how your results would change when $A$ is not binary.\n\nQUESTIONS:\n* Is it possible to extend this worst-case analysis to other post-processing methods? Combining it with the optimized method proposed in [1] would be very interesting.\n\n* In Theorem 2 the authors assume that $\\mathcal{F}$ is such that $ \\forall f \\in \\mathcal{F}$ the Assumption 1 is followed. After that, they conclude that then, there exists a fair model in $\\mathcal{F}$. This doesn’t seem correct. For example, take $\\mathcal{F} = \\{f_0\\}$ where $f_0$ follows Assumption 1 but is not worst-case optimal. Then, there is no fair model in $\\mathcal{F}$. Perhaps, the authors mean $\\mathcal{F}$ to be the set of *all* models that follow Assumption 1. However, in this case, assuming knowledge of $\\mathcal{F}$ does not sound reasonable. Could the authors clarify this point?\n\n* Could you describe (in the main paper) when the worst-case bound is achieved?\n\nLIMITATIONS:\nThe authors discuss their work limitations. I also suggest discussing the ethics of predicting users' sensitive attributes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a tight upper bounds for the equalized odds violation of a predictor in a setting without sensitive attributes.\nIt also presents a post-processing correction method to control the worst-case equalized odds violation and presents results on a variety of synthetic and real datasets.\n\nSTRENGTHS:\n1. The paper addresses a critical issue in fair ML. \n\n2. The tight and computable upper bounds for equalized odds (EOD) violation is a valuable contribution as it allows for a precise understanding of the worst-case EOD violation of a predictor.\n\n3. The proposed post-processing correction method for controlling worst-case EOD is interesting.\n\n4. The paper is backed by experiments on both synthetic and real datasets, strengthening the validity of the results.\n\nWEAKNESSES:\n1. Assumption 1 may limit its applicability in scenarios where the proxy sensitive attributes are not accurate.\n\n2. The paper's results are limited to EOD and its relaxations as definitions of fairness. Extending the results to other notions of fairness may be non-trivial and deserves further consideration.\n\n3. The paper does not consider how one could train a classifier from scratch to have minimal violations, which could be an important\n\nQUESTIONS:\n1. Could you provide more insight into the limitations of Assumption 1 and how these might be addressed?\n\n2. Could you elaborate on how the proposed post-processing correction method could be implemented in real-world scenarios?\n\n3. How could your results be extended to other definitions of fairness, and what challenges might be encountered in this process?\n\nLIMITATIONS:\nThe paper relies on Assumption 1, which may limit its applicability in scenarios where the proxy sensitive attributes are not accurate. \nDespite this limitation, this work is a nice contribution.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}