There is no trade-off: enforcing fairness can improve accuracy
Authors: Subha Maity
Debarghya Mukherjee
Mikhail Yurochkin
Yuekai Sun
Venue:  (2021)

Abstract: One of the main barriers to the broader adoption of algorithmic fairness in machine learning is the trade-off between fairness and performance of ML models: many practitioners are unwilling to sacrifice the performance of their ML model for fairness. In this paper, we show that this trade-off may not be necessary. If the algorithmic biases in an ML model are due to sampling biases in the training data, then enforcing algorithmic fairness may improve the performance of the ML model on unbiased test data. We study conditions under which enforcing algorithmic fairness helps practitioners learn the Bayes decision rule for (unbiased) test data from biased training data. We also demonstrate the practical implications of our theoretical results in real-world ML tasks.

Under review as a conference paper at ICLR 2021
THERE IS NO TRADE-OFE; ENFORCING FAIRNESS CAN IMPROVE ACCURACY
Anonymous authors Paper under double-blind review
ABSTRACT
One of the main barriers to the broader adoption of algorithmic fairness in ma- chine learning the trade-off between fairness and performance of ML models: many practitioners are unwilling to sacrifice the performance of their ML model for fairness. In this paper, we show that this trade-off may not be necessary. If the algorithmic biases in an MI model are due to sampling biases in the training data, then enforeing algorithmic fairness may improve the performance of the MI. model on unbiased test data. We study conditions under which enforcing algorith- mic fairness helps practitioners learn the Bayes decision rule for (unbiased) test We also demonstrate the practical implications of our theoretical results in real-world ML tasks.
data from biased training data.
INTRODUCTION
Machine learning (ML) models are routinely used to make or support consequential decisions in hiring, lending, sales etc. (Citron and Pasquale| (0144). This proliferation of ML models in decision making and decision support roles has led concerns that ML models may inherit (or even exacer- bate) social biases in the training data. For example, Pro-Publica 's investigation of Northpointe (now Equivant)'s COMPAS recidivism prediction tool revealed racial biases against AifrrinA (Angwin et al. 2016).
In response, the ML community has developed many rigorous definitions of algorithmic fairness, including calibration (Corbett-Davies and Goel 2018). (statistical) parity (Feldman et al., 2014). equalized odds (Hardt et al.) 2016), and individual fairness (Dwork al. (011). Researchers have also designed many algorithms for enforcing the definitions during training (Agarwal et al.) 2018
Cotter et al.) 2019 Yurochkin et al. 2020). Despite this flurry of work, algorithmic fairness practices remain uncommon in production.
We conjecture that the lack of broader adoption of algorithmic faimess practices because there seems to be trade-off between accuracy and fairness. Many algorithms that enforce faimess solve optimization problems that maximize how well the model fits the training data subject to fairness constraints. The trade-off arises because imposing faimess constraints usually leads model that fits the training data less well (compared to model from maximizing goodness-of offlnes--offftt fit without any extra constraints).
In practice, this trade-off may not be relevant because the training data may be biased. For example, resume screening model may reject most female applicants for technical roles because women are historically underrepresented in STEM fields, so women are underrepresented in the training data. This is form of sampling bias, and causes the model perform poorly at test time because women are better represented in STEM fields today. In this example, the trade-off is irrelevant because we are mostly concerned with duttr--distribuution (OOD) performance of the model.
There are many other examples of algorithmic bias arising due to biases in the training data. As another example, the systemic racism in the US criminal justice system disproportionately affects African-Americans, leading to higher rates of arrest, conviction, and incarceration. It is no sur- prise that recidivism prediction instruments trained on such biased data is biased against African- Americans (Angwin et al. 2016). In 2014, then U.S. Attorney General Eric Holder warned that recidivism prediction instruments "may exacerbate unwarranted and unjust disparities that are al- ready far too common in criminal justice system and in society"..

---

Under review as a conference paper at ICLR 2021
In this paper, we study whether the common algorithmic faimess practice of enforcing equal ac- curacy on certain segments of the population improves the OOD performance the model. Such algorithmic fairess practices are common enough that there are methods (Agarwal et al., 2018 2019) and software (e-g. TensorFlow Constrained Optimization (?)) devoted to operationalizing them. This provides an alternative argument for broader adoption of algorithmic fairness practices. Instead of viewing fairness as an intrinsically desirable property of ML models, we show that en- forcing fairness helps ML models overcome biases in the training data. Our main contributions are:
We decompose the bias in the training data into two parts: recoverable part orthogonal to the fair constraint and non-recoverable part. We also derive necessary and sufficient conditions under which enforcing fairness on the training data leads to the Bayes optimal model test time (see Theorem 3.4).
2. We show that it is possible to completely overcome the recoverable part of the bias (hence its name) by enforeing an appropriate risk-based notion of algorithmic fairness. This is possible regardless of the magnitude of this part of the bias (see Corollary (3.5).
3. We specialize our results recidivism prediction task and demonstrate the benefits of enforcing fairness empirically (see section
2 PROBLEM SETUP
To keep things simple, we consider standard classification setup. Our results generalize readily to other supervised learning problems (see Appendix for details).
Let X' C Rd be the feature space. y be the set of possible labels, and A be the set of possible values of the sensitive attribute. In this setup, training and test examples are tuples of the form (X, Y) E X' x A x y. If the MI. task is predicting whether a borrower will default on a loan, then each training/test example corresponds borrower's credit history, income level, and outstanding debts; the label Y E {0, 1 encodes whether the borrower defaulted on the loan; the sensitive attribute may be the borrower's gender or race.
loan. The features in X may include the
Let P and P be probability distributions on X' AxX. We consider l as the unbiased distribution from which samples at test time come from and as the biased distribution from which the training data comes from. Let 7 {h X' + }} be model class (e.g. neural nets with particular architecture) and é be a loss function. Our goal is to learn the unbiased Bayes decision rule
h" E arg min I," (h) A E [e(h(X), Y)],
(2.1)
where EE* denotes expectation with respect to Pa using only the biased training data from P. With- further assumptions on Pa, this goal is impossible. To facilitate our goal, we assume the unbiased Bayes decision rule is algorithmically fair in some sense and hope that enforcing the correct notion of fairness allows us to recover h" from P. We shall elaborate on the allowable differences between pa and P in subsection 2.2
2.1 BISK-BASED NOTIONS OF ALGORITHMIC FAIRNESS
In this paper, we study the efficacy of enforces risk-based notions of algorithmic fairness in over- coming bias in the training data. To fix ideas, we provide two examples of risk-based notions of algorithmic fairness.
The first notion of algorithmic fairness we consider is risk parity (RP). This definition is motivated by the notion of demographic parity (DP) in classification. Recall DP requires the output of the ML model (X)) to be independent of the sensitive attribute A: h(X) 1 A. RP imposes a similar condition on the risk of the ML model.
Definition 2.1 (risk parity). ML model h satisfies risk parity with respect to data distribution P if Ep [e(((((X)))) Y) A = Ep [e(h(X), Y) == a') for all a, a' e A.
RP is widely used in practice to measure algorithmic bias in ML models. For example, the US National Institute of Standards and Technology (NIST) tested facial recognition systems and found
2

---

Under review as a conference paper at ICLR 2021
that the systems misidentify blacks rates 5 to 10 times higher than whites Simonite (2019). By comparing the error rates of the system on blacks and whites, NIST is implicitly adopting RP as its definition of algorithmic fairness.
The second notion of algorithmic fairness that we consider is conditional risk parity (CRP). This definition is similar the notion of equalized odds (EO) (Hardt al., 2016) in classification. Recall EO requires the output of the ML model ((X) to be independent the sensitive attribute A condi- tioned on the label: h(X) 1 A Y. CRP imposes similar condition the risk of the ML model; i.e. the risk of the ML model must be independent of the sensitive attribute conditioned on the label.
Definition 2.2 (conditional risk parity). An ML model h satisfies conditional risk parity with respect to data distribution P and label y if
Ep (e(((((X)))) A= a, = = Ep Ep(p(((()) [((((X),
| for all a, d' E A, A.Y...
We say ML model h satisfies CRP with respect P (without mentioning a label value) iff satisfies CRP with respect to and all label values y E
We observe that EO implies CRP because (h((). y) is function of h(X) after conditioning on A and Y. CRP is also closely related to error rate balance (Chouldechova 2017) and overall accuracy equality (Berk et al., 2017) in classification.
As we shall see, both RP and CRP are instances of risk-based notions of algorithmic faimeess. The general form of such a notion
Ep [e(h(x), Y) A a, V = = = Bp((p((()) | A= = v] for all a' A. v E
V.
(2.2)
where V is known the discriminative attribute (Ritov et al. 2017). To keep things simple, we assume V is finite-valued, but it is possible to generalize our results to risk-based notions of algorithmic fairness with more general V's (see Appendix C For RP, V is a trivial random variable; for CRP, V is Y. It is not hard to see that risk-based notions of algorithmic fairness are equivalent to linear constraints the risk profiles of MI. models:
R(h)
EE [((h(X),)) A a.V = u]].EAAEEV
The general fairness constraint has the form R(h) E F, where F a subspace. Figure provides some examples of risk sets and when it is possible to recover Bayes' classifier. We wrap up this subsection by presenting general structure of risk profiles under RP and CRP constraints:
Example 2.3 (risk parity).. Define RRP (h) E RIAI as the vector whose entries are
RRP (h) Ep (((M(X), A= a].
In terms of R (h). RP with respect P implies that RRT (h) c1 for some constant R. This a linear constraint: the set of risk profiles that satisfy the RP constraint is the subspace
FRP A (R E RIAI| |R= R RI
RR.IRRRCCEE .E RR.
Example 2.4 (conditional risk parity). Define RCRP(M) E |AXx) as the matrix whose entries are
RCRF (h) ([((L())))))) -= a. v]
In terms of (CRP(h) (h), CRP with respect to P implies RCRP (h) lu This is again linear constraint: the set of risk profiles that satisfy the CRP constraint is the subspace
FCRP 2 {R E RLAXXMI
R= lu 1ER u R
2.2 BIAS IN THE TRAINING DATA
In this subsection, we describe the allowable differences between the unbiased distribution P and the distribution of the (biased) training data P. To relate the risk of ML models on the training data and at test time, we assume that the risk profiles of the models with respect to P and the profiles with respect to P are identical:
E" [((A(X), Y) == V = = = [[((((X))) A= a for all A, V. (2.3)
3

---

Under review as a conference paper at ICLR 2021
This assumption is similar to the covariate shift and label shift assumptions in transfer learning. It is (slightly) less restrictive because it only requires the expected value (instead of all moments) of the loss to be identical in the training data and at test time. In fact, both covariate and label shift imply instances of (2.3) (with appropriate discriminative attributes).
We also note that this assumption is implicit in enforcing risk-based notions of algorithmic fairness. If the risk profiles are not identical in the training data and at test time, then enforce risk-based notions of algorithmic fairness during training pointless because constraints on the risk profiles in the training data do not generalize at test time.
Finally, we note that the choice of discriminative attributes is crucial. The general problem of learning the (unbiased) Bayes decision rule from biased training data is impossible because the (biased) training data may be totally uninformative of the risks of ML models test time. A good choice of discriminative attributes keeps the training data informative by ensuring the risk profiles are identical on the training data and at test time. Here are two examples of good discriminative attributes.
Example 2.5 (label bias). In binary classiftcation, training data may suffer from label bias. This kind of bias arises when positive examples from disadvantaged groups are rnper-represented in the training data. Here is an example of a data generating process that suffers from label bias: (i) sam- ple training examples (Xi, Yi, A.) from P*, (ii) discard training examples from the disadvantaged group (A, 0) with positive label (Y; = 1) with probability B. This leads to P(X.Y,A) A) P*(X,,, (1- (1--
0,)
Because there are fewer positive examples from the disadvantaged group in the training data (com- pared to test data), this kind of bias causes the ML model to predict mostly negative outcomes for the disadvantaged group. In practice, this kind of bias may creep into the training data more subtly. For example, if human judgements is crucial part of the data generating process, then implicit biases may lead to vver-eepreentatiinn of negative examples from disadvantaged groups in the training data Yeom and Tschantz 2019).
For training data with label bias, a good choice of discriminative attribute is the label. This is because the training data a filtered version of the data at test time, and the filtering process only depends on the label (and sensitive altribute). Thus the class conditionals at test time are preserved in the training data; i.e. Px|e.1 = (a for all a E A. y E y
Example 2.6. In some applications, positive examples of the disadvantaged group are not missing random; their missingness may depend on some factor(s). In such cases, it is possible to keep risk profiles identical in the training data and at test time including the factor(s) as discriminative attributes. For example, the missingness of female applicants for technical roles may depend on their qualifications. By including (some measures of) qualification in the discriminative attribute, the model allows the missingness of female applicants to vary depending on their qualifications.
2.3 ENFORCING ALGORITHMIC FAIRNESS DURING TRAINING
Recall our goal is to learn the (unbiased) Bayes decision rule 2.1.. Unfortunately. we cannot solve (the empirical version of) [2.1 because the training data is biased. Instead, we consider solving (the
empirical version of)
[[((((),))) subject to R(h) E F
minsEH
minRER subject to R F
(PAV, R))
(2.4)
where R & (R(h) | h E H} is the set of all possible risk profiles and PA.V E [0, 1|44 is
the distribution of (A, (so (PA.v. Rt) = E.AV [R..v]). We hope that the fair constraint in (.4 corrects the bias in the training data. As we shall see, this is possible if (i) the discriminative attribute is chosen such that the risk profiles on the training data and at test time are identical, and (ii) the bias is small in certain "directions".
Before moving on to the main result, we remark that there are efficient algorithms for solving (..4) One popular algorithm is a reductions approach by Agarwal et al. (2018). At a high level, the algorithm solves sequence of weighted classification problems in which the weights so that the resulting classifier satisfies the desired algorithmic fairness constraints. This algorithm outputs randomized classifiers, which justifies one of the subsequent assumptions on (.4).
chosen
4

---

Under review as a conference paper at ICLR 2021
3 BENEFITS AND DRAWBACKS OF FAIR RISK MINIMIZATION
The main result provides necessary and sufficient conditions for recovering the unbiased Bayes' classifier with (2.4). Before stating the main result, we state and justify our assumptions. Assumption 3.1. The unconstrained risk minimizer on unbiased data is algorithmically fair; i.e. arg min (P", (R) F.
This assumption is necessary. If the unbiased Bayes classifier is not algorithmically fair, then there no hope for (2.4) recover the unbiased Bayes classifier; there will always be a bias term. This assumption is also implicit in large swaths of the algorithmic faimess literature. For example, Buo- lamwini and Gebru (2018) and Yang et al. (2020) suggest collecting representative training data to improve the accuracy of computer vision systems on individuals from underrepresented demo- graphic groups. This suggestion implicitly assumes the Bayes classifier on representative training data is algorithmically fair. We refer to section 3 for a brief discussion on relaxing this assumption.
At first blush, tempting to think that because the unbiased Bayes classifier satisfies a fairness constraint, then enforcing this constraint always increases accuracy. Unfortunately, this is not the case: enforcing algorithmic fairness may harm OOD generalization, even if the Bayes classifier at test time is algorithmically fair. Intuitively, the assumption that R is fair is a constraint on P* R, and F; it imposes no constraints P. By picking P adversarially, is possible to have
(P. S (P, RF))
Such examples are not pathological, and we provide a graphical example in Appendix A Assumption 3.2. The risk set R is convex.
This assumption is innocuous because it is possible to convexify the risk set by considering random- ized decision rules. A randomized decision rule is distribution on the hypothesis class. To evaluate a randomized decision rule H, we sample a decision rule from H and evaluate h. It not hard to see that the risk profiles of randomized decision rules are convex combinations of the risk profiles of (non-randomized) decision rules, so including randomized decision rules convexifies the risk set. Assumption 3.3. The risk profiles of the models in are identical with respect to P and P, i.e. E² [e(h(X), Y) a, = o] T. (e()(X), Y) a, V v) for all A.EVV.
This assumption needed to keep the risk profiles on the (biased) training data informative for the (unbiased) test data. We refer to section 2.2 for more comprehensive discussion of this assumption.
Theorem 3.4. Under assumptions such that R(h) = R if and only if
3.1
3.2 and 3.3
the fair risk minimization
2.4 obtains H
Is - PAv) - Ply E (RRRR) F+.
(3.1)
where PA.y (resp. PA.v) is the marginal of P (resp. P) with respect to (A, (R" is the normal cone of R at R. and II is the projection the fair hyperplane.
Theorem 3.4 characterizes the biases in the training data from which is possible to totally recover by enforcing appropriate algorithmic faimess constraints. By totally recover from bias, mean recovering the unbiased Bayes decision rule. To keep things simple, we stated our main result only for finite-valued discriminative attributes. Please Appendix C for more general version of Theorem that applies to more general (including continuous-valuedl discriminative attributes.
The main insight from Theorem 3.4 (and its counterpart for continuous discriminative attributes in Appendix C) is a decomposition of the training bias into two parts: part orthogonal to the fair constraint and the remaining part in Nvx (R*). Enforcing an appropriate risk-based notion of algorithmic fairness overcomes the first part of the bias. This occurs regardless of the magnitude of this part of the bias (see Corollary 3.5). and we see this later in our computational results.
The second part of training bias (the part in (R*)) represents the "natural" robustness of R to changes in p": if P is in (R*) then the unconstrained risk minimizer training data remains R. The magnitude of the bias in this set cannot be too large, and enforcing algorithmic fairness constraints does not help overcome this part of the bias.
5

---

Under review as a conference paper at ICLR 2021
20
Figure 1: Total recovery from training bias by enforcing risk parity. this simple example, the training bias P p* is always orthogonal to the risk parity constraint (blue line) because P and P probability distributions. Thus if the training bias does not affect the risk profiles (i.e. P satisfies Assumption 3.3), then enforcing risk parity allows us to to- show an example in which the risk decomposes into recoverable and non-recoverable parts, we need (at least) two more dimen- sions.
R
Elzinko -1
10-
R
05-
R
tally overcome the training bias. Unfortunately,
00
Ex
P
-35
-05
30
6.3
15
EL(UK)) YIJA =0]
Corollary 3.5. A sufficient condition for 3.1 is PAN PA EFL.
Proof of Corollary 3.5 If PAV Py,v E F+, then IIx(Px PA.) = 0, so we need to check
that -Av E (R FL. For any RET R,
(-PA. R$) = (PA.v R R) 0
as R" is the minimum value of (Pr,y: : R) over R. This shows - -Pa E NRR(R*) desired.
Corollary 3.5 allows large differences between PA.V and its unbiased counterpart Pä.v v. as long as the differences are confined to F Intuitively, (2.4) enables practitioners to recover from large biases in F= because the algorithmic fairness constraint "soaks up" any component of PA,V v in F+ We explore the implications of Corollary 3.5 for risk parity and CRP.
Risk Parity: For RP, V is trivial random variable, hence PA PA E F means that it has mean 0. This true for any PA as (PH, 1) (Px, = 1. Hence, the Bayes' classifier can be recovered under any perturbation. More specifically, recall the example of women historically underrepresented in STEM fields mentioned in the introduction. Such train data is biased in its gender representation which differs at test time where women are better represented. Classifiers trained on biased data with the risk Parity fairness constraint will generalize better at test time.
Conditional risk parity: In this case V = and the condition PA,Y PY, E Fe implies that
sum of each column of PA,y Py,y must be 0. Hence, to recover the Bayes classifier under equalized odds faimeess constraints, we are allowed to perturb P in such way, that they have the same column sums: i.e. for any label, we are allowed to perturb the distribution of protected attributes for that label, but we have to keep the marginal distribution of the label to be same for both PA,Y and Pa,y We investigate this scenario empirically in Section
In practice, it is unlikely that the training bias is exactly orthogonal to the fair constraint, Theorem 3.4 is a more general recovery result that characterizes conditions under which fair risk minimization recovers the Bayes classifier. For this happen, the remaining part of the bias must be small enough. Theorem proviiee a precise characterization "small enough".
3.1 RELATED WORK
Most of the prior works on algorithmic faimess assume fairness an intrinsically desirable property of an ML model, but this assumption is unrealistic in practice (Agarwal et al.) 2018 Cotter et al.) 2019 Yurochkin et al.) 2020). There is small but growing line of work on how enforcing fairness helps ML models recover from bias in the training data. Kleinberg and Raghavan (2018): Celis
et al. (020)) consider strategies for correcting biases in hiring processes. They show that correcting the blases not only increases the fraction of successful applicants from the minority group but also boosts the quality successful applicants. Dutta et al. (2019) study the eecuracy-fairneee trade-off in binary classification in terms of the separation of the classes within the protected groups They

---

Under review as a conference paper at ICLR 2021
1-0, yrd t-l, yel
Figure 2: Decision heatmaps for (left) baseline on train data from P: (center left) fair classifier on train data from P: (center right) baseline test data from P*; (right) fair classifier test data from P Decision boundary of the fair classifier has larger slope better accounting for the group a = underrepresented in the train data. Consequently its performance is better the unbiased test data.
explain the eccuracy-fairnes, trade-off in terms of this separation and propose way of achieving fairness without compromising separation by collecting more features.
Blum and Stangl (2019) study how common group faimess criteria help binary classification models recover from bias in the training data. In particular, they show that the equal opportunity criteria (Hardt et al.] 2016) recovers the Bayes classifier despite under-representation and laheling biases in the training data. Our results complement theirs. Instead of comparing the effects of enforcing various fairness criteria on training data with two types of biases, we characterize the types of biases that the fairness criteria help overcome. Our results also reveal the geometric underpinnings of the constants that arise in Blum and Stangl|s results. Three other differences between our results and theirs are: (i) they only consider binary classification, while we consider all ML tasks that boil down to risk minimization, (ii) they allow some form of posterior drift (so the risk profiles of the models in H. with respect to p and P may differ in some ways), but only permit marginal drift in the label (V Y), (iii) their conditions are sufficient for recovery of the fair Bayes decision rule (in their setting), while our conditions are also necessary (in our setting).
4 COMPUTATIONAL RESULTS
We verify the theoretical findings of the paper empirically. Our goal is to show that an algorithm trained with fairness constraints on the biased train data 1 achieves superior performance on the true data generating P* test time in comparison an algorithm trained without fairness considerations.
There are several algorithms in the literature that offer the functionality of empirical risk minimiza- tion subject various fairness constraints, e.g. Cotter al. (20T9) and [Agawall al. (2018). Any such algorithm will suffice to verify our theory. In our experiments we use Reductions fair clas- sification algorithm (Agarwal et al., 2018) with logistic regression as the base classifier. For the fairness constraint we consider Equalized Odds (Hardt et al. 2016) (EO) one of the major and more nuanced fairness definitions. We refer to Reductions algorithm trained with loose EO viola- tion constraint as baseline and Reductions trained with tight EO violation constraint as fair classifier (please see Appendix D for additional details and supplementary material for the code).
Simulations. We first verify the implications of Corollary 3.5 Jusjinnn simulation studies. We follow the Conditional risk parity scenario from Section 3 Specifically, consider a binary classification problem with two protected groups, i... Y E {0, }} and A E 1). We set p* to have equal representation of protected groups conditioned on the label and biased data P have one of the protected groups underrepresented. Specifically, let Pay = PA=a,Y =y. i.e. the a, indexed element
of PAAY: Pag = 0.25 Va, y for p* and Ply Pminorr Poy Pmajor 0.5 Pminor for P. For both Pa and we fix class marginals p.o = p.1 = 0.5 and generate Gaussian features X|A a, Y y (Haa, E,yy) in 2-dimensions (see additional data generating details in Appendix D
In Figure 2 we show a qualitative example of simulated train data from P with Pminor = 0.1 and test data from P", and the corresponding decision boundaries of baseline classifier and classifier trained with the Equalized Odds fairness constraint (irregularities in the decision heatmaps are due to stochasticity in the Reductions prediction rule). In this example fair classifier is 3 more accurate
7

---

Under review as a conference paper at ICLR 2021
on the test data and 1% less accurate biased test data sampled from P (latter not shown in the figure).
We proceed with quantitative study by vary- ing degree of bias in P via changing Pminor in [0.01, 0.25] and comparing performance of the baseline and fair classifier on test data from P and P. We present results over 100 runs of the experiment in Figure 3 Notice that the sum of each column of PAAY PA is 0 for any value
Fair on
0.95
Baseline P
Fair on F
0.90
Baseline P
accuracy
0.85
0.80
of Pminnor and we observe that the fair classifier
has almost constant accuracy on P (consis-
0.75
tently outperforming the baseline), as predicted by Corollary [35 The largest bias in the train- ing data corresponds Pminor 0.01, where baseline is erroneous on the whole = 1. 0 subgroup (ef. Figure 2 resulting in close to 75% accuracy corresponding to the remaining 3 (out of 4) subgroups. For Piiinrr 0.05 mi- nority group acts as outliers causing additional errors at test time resulting in the worst performance overall. When Pminor 0.25, P= P and all methods perform the same as expected. Results on P correspond to the case where test data follows same distribution as train data, often assumed in the literature: here baseline can outperform fair classifier under the more extreme sampling bias conditions, i.e. Pminor I 0.1. We note that as the society moves towards eliminating injustice, we expect test data in practice to be closer to P rather then replicating biases of the historical train data P.
0.70
0.01
0.05
0.10
0.15
0.20
0.25
Pminor
Figure 3: Test accuracy on P and P when trained on the (biased) data from P.
Recidivism prediction on COMPAS data. We verify that our theoretical findings continue to apply on real data. We train baseline and fair classifier on COMPAS dataset (Angwin jet al. 2016). There are two binary protected attributes, Gender (male and female) and Race (white and non-white), resulting in 4 protected groups A E 2,, 4]. The task is to predict if a defendant will re-offend, i.e. Y {0, 1]. We repeat the experiment 100 times, each time splitting
Table 1: Accuracy on COMPAS data
Test on P
Test on P
Fair Bascline
0.652+0.013 0.63410.011
0. 0.660=10.009 0.668+0.010
the data into identically distributed 70-30 train-test split, i.c. for train and test, and obtaining test set from P* by subsampling test data to preserve Y marginals and enforcing equal representation at each of the 4 levels of the protected attrribute A. We present results in Table [
We see that our theory holds in practice: accuracy of the fair classifier is 1.8% higher on P Baseline is expectedly more accurate on the biased test data from P. but only by 0.8%.
We present results for the same experimental setup on the Adult dataset (Bache and Lichman 2013) in Table 2 in Appendix D We observe same pattern: in comparison to the baseline, fair classifier increases accuracy on P*, but is slightly worse on the biased test data from P.
5 SUMMARY AND DISCUSSION
We showed that enforcing algorithmic fairness allows practitioners to recover from certain biases in the training data. The main insight from our theoretical results is enforcing risk-based fairness constraints mitigates bias in the training data that is orthogonal to the faimess constraints. In other words, regardless of the magnitude of the training bias, the faimesss constraints just "soaks it up". On the other hand, fairness constraints play no part in mitigating the remaining parts of the bias.
Our results depend on the assumption that the Bayes decision on test data satisfies a risk-based notion of algorithmic fairness. This assumption is strong, but it is necessary to recover the Bayes classifier. To remove this assumption, we must weaken the goal to merely improving upon the risk minimizer on the biased data. The set of "* for which enforcing the fair constraint improves accuracy is easy to characterize. Let F be fairness constraint and R and Ry be the (unconstrained) and fairness
8

---

Under review as a conference paper at ICLR 2021
constrained risk minimizers with respect to P:
R & migminnest (PA.V, R), R meR (PAV,))
The set of P's for which enforcing the fairness constraint improves accuracy is {P (P, 0}. In other words, long as P is in the preceding set, then enforcing the faimess constraint improves accuracy. Studying the structure of this set is promising area of future work.
We also note that there is another family of algorithmic fairness practices based on robust optimiza- tion (Hashimoto et al.) 2018; Sagawa et al., 2019; Yurochkin et al.) 2020) that are also widely used in practice. Although there are empirical results that demonstrate the efficacy of such practices, there are no theoretical results justifying their use when the training data is biased. This is another promising area of future work.
Taking a step back, the main takeaway for ML practitioners is possibility to encourage fairness and improve accuracy by enforcing risk-based fairness constraints during training. As long they choose the discriminative attribute carefully so that the risk profles are identical the training data and test time, then it is possible to learn the Bayes decision rule on the test data from (biased) training data. This departs from most prior work on algorithmic fairness that starts with the premise that fairness an intrinsically desirable property of an ML model. Unfortunately, although most ML practitioners agree that algorithmic faireess is desirable, they are generally unwilling to sacrifice accuracy of the model for fairness. This is gap between ML practice and algorithmic fairness research, and our work is one way to close this gap. By aligning algorithmic faimesss with the usual goal of MI. practitioners, we hope that this argument enlists the "invisible hand" of accuracy to promote algorithmic fairness practices.
REFERENCES
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. Re- ductions Approach to Fair Classification. arXiv:1803.02455 [cs], July 2018.
Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair Regression: Quantitative Definitions and Reduction-based Algorithms. arXiv:1905. 12843 [cs, stat), May 2019.
Julia Angwin,
Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias.
May 2016.
K. Bache and M. Lichman. UCI machine learning repository. 2013.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in Criminal Justice Risk Assssmmnts:: The State of the Art. arXiv: arXiv: 1703.09207 [stat], March 2017.
Avrim Blum and Kevin Stangl. Recovering from Biased Data: Can Faimess Constraints Improve Accuracy? arXiv: 1912.01094 [cs, stat], December 2019.
Joseph Frédéric Bonnans and Alexander Shapiro. Perturbation Analysis of Optimization Problems. Springer Series in Operations Research. Springer, New York, NY, 2000. ISBN 978-1-4612-7122. 9778-0-38--98005-7.
Joy Buolamwini and Timnit Gebru. Gender Shades: mercial Gender Classification. In Proceedings of Machine Learning Research, volume 87, pages 77-91, 2018.
Intersectional Accuracy Disparities in Com-
L. Elisa Celis, Anay Mehrotra, and Nisheeth K. Vishnoi. Interventions for Ranking in the Presence of Implicit Bias. January 2020.
Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. arXiv: 1000::000000 (cs, stat], February 2017.
Danielle Keats Citron and Frank A. Pasquale. The Scored Society: Due Process for Automated Predictions. SSRN Scholarly Paper ID 2376209, Social Science Research Network, Rochester, NY, 2014.

---

Under review as a conference paper at ICLR 2021
Sam Corbett-Davie and Sharad Goel. The Measure and Mismeasure of Fairness: A Critical Review of Fair Machine Learning, arXiv:1808.00022 [cs], July 2018.
Andrew Cotter, Heinrich Jiang, Maya Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. Optimization with Non-Differentiable Constraints with Applications to Fair- ness, Recall, Churn, and Other Goals. Journal of Machine Learning Research, 20(172):1-59, 2019. ISSN 1533-7928.
Nelson Dunford, Jacob T. Schwartz, William G. Bade, and Robert G. Bartle. Linear Operators. Interscience Publishers, 1958.
Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush R. Varsh- ney. An Information- Theorettic Perspective on the Relationship Between Fairness and Accuracy. 11100:1900007870 [cs, math, stat], October 2019.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel. Fairness Through Awareness.
[cs], April 2011.
Michael Feldman, Sorelle Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubrama- nian. Certifying and removing disparate impact. urXiv: [cs, stat], December 2014.
Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning. arXhv:1610.02413 [cs], October 2016.
Tatsunori B. Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness With- out Demographics in Repeated Loss Minimization.
[cs, stat), June 2018.
Jon Kleinberg and Manish Raghavan. Selection Problems in the Presence of Implicit Bias. January 2018.
Ya acov Ritov, Yuckai Sun, and Ruofei Zhao On conditional parity as a notion non- [es, star], June 2017.
discrimination in machine learning.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Gener- alization. arXiv:1911.08731 [cs, stat]. November 2019.
Tom Simonite. The Best Algorithms Still Struggle to Recognize Black Faces. Wired, July 2019. ISSN 1059-1028.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Pro- ceedings of the 2020 Conference on Fairness, Accountabiliry, and Transparency, pages 547-558, 2020.
Samuel Yeom and Michael Carl Tschantz. Discriminative but Not Discriminatory: A Comparison of Fairness Definitions under Different Worldviews. 1808v:8808.0800 [cs, stat], September 2019.
Mikhail Yurochkin, Amanda Bower, and Yuckai Sun. Training individually fair ML models with sensitive subspace robustness. In International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.
10

---

Under review as a conference paper at ICLR 2021
RF
P
R
R
P
Figure 4: Example in which enforcing algorithmic fairness harms OOD generalization. The triangle the set of risk profiles, and the dotted left side of the triangle intersects the fair constraint (i.e. the risk profiles on the dotted line are algorithmically fair). The training objective P chosen so that the (unconstrained) risk minimizer on biased training data R is the vertex on the right and the fair risk minimizer (also on biased training data) Ry is the vertext on top. The test objective points downward, so points close to the bottom of the triangle have the smallest risk at test time. We that R closer to the bottom of the triangle than R,, so it has better OOD generalization.
A ENFORCING ALGORITHMIC FAIRNESS MAY HARD OOD GENERALIZATION
In this section, provide an example in which enforcing algorithmic faimess harms OOD gener- alization performance. Formally, we provide choices of P, p FF, and R such that (i) R and (ii) (P*, R) < (P*,Ry>) RRr). Figre shows our example. Inspecting the figure reveals such examples are hardly pathological. Just in this figure, there are range of choice for p and P that lead to enforeing algorithmic fairness harming OOD generalization.
B PROOF OF THEOREM 3.4
In this section, we provide proof of Theorem 3.4 under the additional assumption that A and are finite sets. Although less general, we feel that this proof is more instructive because it suggests the origin 3.1.
Proof. "if" direction: Let 1---Pp -PAV If (..1. then it is not hard to check that (R*, -) satisfies the optimality conditions of (.4
PA.v + Z, (stationarity)
Rº E F,
(primal feasibility)
(B.1)
E E& + F (dual feasibility).
Indeed, we have stationarity by the definition Z. strained risk minimizer on unbiased data is algorithmically fair: R E F. We have dual feasibility because
We have primal feasibility because the uncon-
Ż IIx(PA,v PAV) PA.v II (Pā. - PAV)
E NR(HH)) + +
=NR(1(") + +
11

---

Under review as a conference paper at ICLR 2021
where appealed to (3.1) in the second step and recalled JF is subspace in the third step. The FRM problem (2.4) is convex, so [B.I implies R an optimal point of (24).
"nlyy if" direction: Assume R solves (2.4). This implies there is 7 E (RR(R") + F... such that (R", ?) satisfies B.1 By the stationary and dual feasibility conditions,
7 PPAV ENR(R") + F-
We write PA.v as [F - PA.V)
PA.V + TT (PA. - -PAv)) and rearrange to obtain
[T - PAv) - PA.v E E (PÅ. + Wxx + F+ = =N((Rt) + F+
where we recalled F subspace in the second step.
C CONTINUOUS DISCRIMINATIVE ATTRIBUTES
In this section, we state and prove more general verion of Theorem 3.4 that permits continuous discriminative attributes. In this more general setting, risk profiles are (integrable) functions on Z e A V, so the fair risk minimization problem (B.I and its unconstrained counterpart are infinite dimensional optimization problems. We start by setting up the problem and reviewing relevant results from optimization theory.
Let (Z, Σ) be measurable space and S be the set of bounded measurable functions on (Z, Σ). We equip S with the sup norm. The risk set and the fair constraint set J are generally subsets of S. The (topological) dual of S (denoted by S') is the set of finitely additive measures on equipped with the total variation norm Dunford et al. (1958). This result allows us to represent continuous linear functionals such spaces with (finitely additive) measures, it is generalization of the more familiar Riesz-Markov- Kaautain representation theorem to spaces of (possibly discontinuous) measurable functions. We observe that the more familiar set of countably additive measures. is closed subset of Z'.
Definition C.1 (Complemented subspace). Let B be Banach space and A C B be a subspace. We say A is complemented subspace of B, if there exists another subspace Ac C B such that B = A + Ac.
Henceforth, for if A is complemented subset of Banach space IB (i.e., A A. B) then we define IIA.A (resp. (z)c is the component of x in A (resp. Ac), i.. IAA.Ao Φ1 (resp. AC = X2) where I = + 22 with E A, 22 Ac. Recall that, we define F as the fair hyperplane. Previously it was a subspace of the risk set, now becomes a subspace S. We have the following assumption on the fair hyperplane:
Definition C.2 (Annihilator). For any A C B define its annihilator A C B' as the set of bounded linear functions f + such f(x) = for all r A.
Lemma C.3. Let A be a complemented subspace in B. Then A is complemented in IB'.
Proof. Since, A is complemented in B. there exists subspace G C B such that A G B. This implies, each BB has the unique decomposition 1q 22, where Ti E and x2 G. We consider the projection ma II B + B such that [I), = Let us define two following subspaces in B':
HA.G = o IlAG If EEB
HA.G = (f - ff f f B'}.
Note that, H.A.G A Also, for any f A we have f IIA. Og + f=f--folllc H..GG This implies HA,G A+. Furthermore, B' HAGG HAGG and for any f HA,. have ((A) [(G (0). Hence, Oz. This implies BB HAG HA. HA, @ A-.
12

---

Under review as a conference paper at ICLR 2021
Finally, review some relevant background on infinite dimensional optimization. Since mostly concerned with convex optimization problems with linear cost functions, the theory simpli- fies considerably.
Definition C.4 (tangent cone). The tangent cone of a closed convex set C C B at a point I E is the closure of the cone of feasible directions
Te(x) - clfd E B there is 0 such that x + ld E C for all E [0,
There are many notions of tangent cone in variational analysis (e.g. Clarke tangent cone, contingent cone, inner tangent cone etc.), but they all coincide for closed convex sets Bonnans and Shapiro (2000). Notably, this definition is identical to the definition (for convex sets) in finite dimensions.
Definition C.5 (normal cone). The normal cone of a closed convex set C at a point x E the polar cone of the langent cone C at x:
No(x) A (d' E (d', s for all E Tc(x)}.
Proposition C.6. Let C be a closed convex subset of a Banach space B. Consider the convex optimization problem
min
A point x² E C is an optimal point iff
(c,d) 2 for any d E Te(x"),
where (c, ) is the linear cost function and Te(x*) (x") is the tangent cone of C at x². Equivalently. x" is optimal if and only if c Nc(x*).
Recall that in normed vector space (f.x) means the value of the linear functional f at In our problem setting, points in the normed space S are integrable functions/random variables and linear functionals on S are (finitely additive) measures, so (f, x:) means expectation of the random variable x with respect to probability measure f.)
We ready to state the extension of our main result to continuous discriminative attributes. As- sumptions 3.1 3.3 from the main paper remain in effect. For continuous discriminative at- tributes, impose additional assumption.
Assumption C.7. The fair subspace F is complemented in S.
This assumption is usually satisfied by common algorithmic faimeess constraints: when RP is sidered, F is the set of all constant functions from A to R. For CPR, C the set all functions Ax + IR such that f is constant on the first co-ordinate, i.c. fx1 y) = f(z2, y) for all 21 # x2 E A and y E y. We now argue that, in both the cases F is a complemented subset of S the fact that any subspace
under mild assumptions. For RP, codim(A) A) < oo is complemented. As FRP is the set of all constant functions, it has dimension and hence complemented. For CRP, assume that there exists some base measure μ such that f E S is integrable with respect to μ. Then one can write: f fi + J2 where fi E F which is defined as: [a v) = g(v) where g(v) is the marginal of f(., v) with respect to the base measure μ. The function is analogously defined as f f(a, v) g(v).
C S with dim(A) < oc or
Theorem C.S. If the unconstrained risk minimizer on unbiased data algorithmically fair (i.e. its risk profile R satisfy the fairness constraints), then fair risk minimization (..4 learns h E H. such that R(h) = R. under assumptions C.7 and 3.2 if and only if
1 p4 (PAV - PAV) - PA E NR(RR) +
(C.I)
where Pa PA.v (resp.PA.) PAy) is the marginal of P (resp. P) with respect (A,V), (R* is the ) is the projection defined previously.
normal cone of R R and II Fg Fg
(+)
13

---

Under review as a conference paper at ICLR 2021
Table 2: Accuracy Adult data
p
P
Fair
0.852:10.04 0.848=0.005
0.843:10.003 0.847=000003
Base
Proof. For notation simplicity define X - 11 Fi (PAy PA.V) Pay. We show that
minRer
(P) holds if and only if E (Rt). Towards that end, fix R E F:
(P.R) = - P (P*,))
(115g.5- -P- Ry + (P*,
(-P'-X, + +P*,,
(-X,
(C.2)
-(-X,R") + ( R*)
= =(P,R") R + (-X, RR)
[From equation C.2
Hence we have: minner (P, R) (P, R^) if and only if (-X, R²) 2 0 for all R E which holds if and only i ~NRR = NR (1²) +
+F This completes the proof.
L
D EXPERIMENTAL DETAILS
We provide additional details to help reproduce our results. Please also see the code provided with the submission. Code for the Reductions classifier (Agarwal et al.] 2018) is available here: https: //gg:://tttttt com/
We modified the source code prevent it from early stopping, so the baseline classifier runs for same number of iterations as the fair classi- fier. The idea behind the Reductions approach is to translate the problem of learning fair classifier into a constraint optimization problem, where constraints depend on the fairness definition of choice. Reductions method requires base classifier: it learns ensemble of the base classifiers optimize performance subject to the fairness constraints. We used logistic regression as the base classifier in all experiments. The other important parameter is the tolerance that controls the amount of permis- sible constraint violation. Smaller tolerance implies tighter fairness constraints. In all experiments we used Equalized Odds fairness constraint (Hardt et al., (2166) with 10 for the baseline classi- fier (i.e. fairness can be arbitrarily violated) and = 0.1 (for the Adult experiment = 0.02) for the fair classifier
Simulations Simulated data is generated from X|A a, a, Y- NNn~NN(M N(aay Eay) in 2-dimensions with prescribed A, Y joint distribution. We fixed marginals p.o = p.1 = 0.5 and varied joint PA,Y to study different degrees of label bias. Reductions was trained for 25 iterations for both baseline and fair classifiers. We provide code reproducing Figure of the main text s simulat ioons.... py- Please also refer the code for concrete values of (Vay, Eny } and other minor details.
COMPAS experiment Reductions was trained for 50 iterations for both baseline and fair classi- fiers. We provide code reproducing one run of the experiment for Tf the main (results in the table summarize 100 runs) in compas. . py. Please also refer to the code for data pre-processing and other minor details.
Adult experiment
We ran experiment the Adult dataset
We could use simple logistic regression as the baseline classifier, however this would mean that baseline classifier and fair classifier in different hypothesis classes. To avoid this, we used Reductions method for both with same number of iterations, however with loose fairesss constraint for the haseline.
14


Reviews:
[{"review_id": "s84WMiNHgIG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nIn this paper, the authors show that under some assumptions, they can use the fairness constraint to understand some parts of the distribution shift between train and test, and as a result they can improve accuracy at the test time. The main assumptions are (i) the optimal classifier satisfies the fairness constraint and (ii) the risk profiles of all the classifiers in their hypothesis class are the same between biased and unbiased distribution.\n\nThe paper was generally well-written; however, Theorem 3.4 and the assumption 3.3 (eq 2.3) were not sufficiently explained. For Assumption 3.3 the only apparent scenario is that if we group data points according to A and V then the weight of each group can vary between the train and test distribution. What other important distribution shift is supported under this assumption, and why are such distributions important/plausible? Also, the relation to the covariate shift requires some clarification.\n\nAs I explained, one of the supported distribution shifts is when groups (according to A, V) have different weights in train and test data. In this case, this work is similar to work on group DRO where they optimize the worst group risk instead of the average risk and show they can increase the accuracy in the test data. The authors need to compare their work with this line of work. \n\nThe intuition behind Theorem 3.1 (or at least what I understand) is that due to 3.2 we only need to consider the risk over groups according to A and V (RHS of 4.2). Depending on the weights of these groups, the optimal classifier is different.  The authors then assume in the test data the optimal classifier is fair; thus, this assumption leaks some information about the distribution of the test data (weights of these groups at the test data). By projecting the optimal classifier to the fair subspace, we can recover the optimal classifier under some assumption. Again, the only apparent scenario to me is when the distribution in test data has the same weights for the groups with different A but the same V. What other distributions can the test data have, in particular, for what distributions the fair classifier is also the optimal classifier? \n\nI think Theorem 3.2 needs a better explanation. What is the intuition of the normal cone, in what conditions we cannot recover the optimal classifier, etc.? For example, although the paper said that showing 4 dimensions in figure 1 is hard, one can have 4 points with different risks on different groups and show which point is chosen by what algorithm and why. \n\nIf the optimal classifier satisfies some fairness constraint, then enforcing that fairness constraint on biased data should always increase the accuracy, right? So what is the surprising result here? Is it characterizing exactly when the optimal classifier is recovered? \n\nOverall I think this was a good paper, and I enjoyed reading it. However, I believe more intuition and maybe more results will make the paper stronger. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lOkqwViGAQY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although the paper studies a current and very important issue in algorithmic fairness, I have couple of concerns about the paper in general as follows:\n\n1. The assumptions made in the paper makes the stated conclusion almost trivial making the technical contribution of the work not standing out. For instance, assumption 3.1 is not ever guaranteed to hold. Another major assumption is P* and the fact that the test data is coming from this distribution. This makes the problem trivial as it is almost saying that if I satisfy the fairness objective and make sure that my test data is satisfying the very same objective then the performance of the model will boost. This issue can also be observed in the experimental setting as described in my point #2 detailed below.\n\n2. In experiments, the paper claims that we are forcing/incorporating the EO objective in our classification task but also making sure that the test data is satisfying the equal conditional factor. In this case yes the performance of the fair model on P* will increase, but as shown in the results, the performance of baseline model is better on P~ making the results not convincing to me.\n\n3. I think there need to be more controlled and convincing experimental settings to show the significance of the claim. From the COMPAS results specifically, the baseline is clearly outperforming the fair classifier on P~. Yes sure the fair classifier will outperform on P* but this is only because P* is following and satisfying the objective of the fair classifier. This makes me wonder what the paper is trying to prove in this case.\n\n4. Finally more datasets need to be tested specially to show the case. Having COMPAS only is not convincing in addition to the synthetic data which I noticed similar patter in it. The experimental setting is so brief specially on the real world benchmark fairness dataset, COMPAS.\n\n5. A minor comment: The resume screening example brought up in the paper seems to me more like historical bias; however, authors refer to it as sampling bias.\n\n6. Finally, the title makes a very strong claim which is not well supported in the paper considering both the experimental results and assumptions made in the paper.\n\nOverall, the paper needs more experiments with clear and convincing setup with more benchmark datasets utilized in them. The paper can also improve its writing, clarity and accuracy of some minor points such as the sampling bias issue brought up in point 5. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7EeGR0uHQcJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "---------------------After reading the author response and the feedback from the others ------------------------\n\nThank the authors for their response.  \n\nI still think the assumptions made in the paper are strong.  For example, the assumption in equation 2.3 and 3.3 is hardly true. Even in the revised example 2.6, the missing rates might also depend on other factors. It is hard to identify every related attribute in general. \n\nI am still not convinced that \"machine-learning practitioners are unwilling to sacrifice the performance of their ML model for fairness\". First, there might be many other possible reasons that some machine learning applications do not incorporate these fairness considerations. For example, there is no consensus on what fairness constraints we should achieve. In fact, existing works show that considering fairness might has bad effect on protected groups (e.g. Liu et al Delayed impact of fair machine learning). Machine-learning practitioners might just be confused what to do rather than unwilling to do. Second, I have seem many applications that have fairness incorporated. (e.g. Search Google image CEO, we now see many female CEOs on the top). From my perspective, responsible businesses would like to pay extra money to obtain a fair ML solution for the benefit of the society and the reputation of the business as long as they know what to do and how to achieve fairness. \n\nSo, I would like to keep my score. \n\n------------------------------------Original Review---------------------------------------\n\n\nSummary\n\nThe paper proposed risk parity and conditional risk parity as fairness measures that ensure the risk of machine learning models have the same risk for each subgroup. The paper discusses the conditions that minimizing the risk under a biased data distribution with these fairness constraints will lead to Bayes optimal policy in the target test distribution. Empirical evaluations on simulation data and rea-world recidivism prediction data validate the theoretical analysis. \n\n\nStrengths:\n\n1. The paper provides insight on when the optimal fair policy and the optimal utility policy coincide from a biased data point of view, i.e. ensuring some fairness on biased data might learn the optimal policy for the unbiased data. The discussion is very interesting. \n\n2. The paper is clearly written and easy to follow.\n\n3. The paper characterizes the necessary and sufficient for a policy that minimizes the risk under the proposed fairness constraints on the biased data distribution is optimal in the target test distribution. \n\nWeakness:\n\n1. Many assumptions are strong and not likely to be true in practice.\n\n    1.1 For example, assumption in equation (2.3) or assumption (3.3) assumes that the risk of any policy on a subgroup is the same in both the biased data distribution and the true target test data distribution. In example 2.5, positive examples of disadvantaged group should be missing at random to satisfy the assumption. Missing at random is hardly true. For example, they might discard positive training examples based on some features, say the qualification of a female job candidate in the job screening example. Example 2.6 in general does not satisfy this assumption. I do not understand why the authors present this example 2.6. \n\n    1.2 Assumption 3.1 assumes that the unconstrained risk minimizer on unbiased data distribution is algorithmically fair. This is hardly true. \n\n\n2. If these assumptions do not hold, then the fairness definition sometimes does not make sense to me. For example, when assumption 3.1 is violated, the optimal classification accuracy for group A is 90%, the optimal classification accuracy fro group B is 85% and there is a policy \\pi that achieves the optimal accuracy for both groups. But risk parity constraint says that, we cannot choose this optimal policy, we should select a policy that achieves 85% accuracy for both groups. This does not help group B but only harms group A. \n\n3. It would be great to compare with methods that directly model the bias in the data. For example, using importance sampling to correct the distribution difference between the training data distribution and the target test data distribution. \n\n\n\nAdditional feedback\n\nContribution 1 \"which which \"-> \"which\"\n\nThe end of page 4 \"that that\" -> \"that there\"\n\nIt would be great to provide evidence that \"machine-learning practitioners are unwilling to sacrifice the performance of their ML model for fairness\" in the abstract and in the conclusion. \n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The problem as formalized in this paper is essentially a domain adaptation problem. There is a training distrinution P and a test distribution P*. the learner gets training data generated by P and aims to minimize the loss of its hypothesis w.r.t. P*. How is it relate dto fairness? The authors add the assumption "we assume the unbiased Bayes decision rule is algorithmically fair in some sense and hope that enforcing the correct notion of fairness allows us to recover h∗ from P". Under such an assumption, almost by definition "enforcing fairness may improve accuracy". By a similar logic, if we assume the unbiased byes decision rule is biased against a certain group, then enforcing bias against that group will imporve accuracy ....
Decision: Reject