Does enforcing fairness mitigate biases caused by subpopulation shift?
Year: 2021

Abstract:
Many instances of algorithmic bias are caused by subpopulation shifts. For example, ML models often perform worse on demographic groups that are underrepresented in the training data. In this paper, we study whether enforcing algorithmic fairness during training improves the performance of the trained model in the target domain. On one hand, we conceive scenarios in which enforcing fairness does not improve performance in the target domain. In fact, it may even harm performance. On the other hand, we derive necessary and sufficient conditions under which enforcing algorithmic fairness leads to the Bayes model in the target domain. We also illustrate the practical implications of our theoretical results in simulations and on real data.

heading: 1 Introduction
text: There are many instances of distribution shifts causing performance disparities in machine learning (ML) models. For example, Buolamwini and Gebru [7] report commercial gender classification models are more likely to misclassify dark-skinned people (than light-skinned people). This is (in part) due to the abundance of light-skinned examples in training data. Similarly, pedestrian detection models sometimes have trouble recognizing dark-skinned pedestrians [22]. Another prominent example is the poor performance of image processing models on images from developing countries due to the scarcity of images from such countries in publically available image datasets [20].
Unfortunately, many algorithmic fairness practices were not developed with distribution shifts in mind. For example, the common algorithmic fairness practice of enforcing performance parity on certain demographic groups [1, 11] implicitly assumes performance parity on training data generalize to the target domain, but distribution shifts between the training data and target domain renders this assumption invalid.
In this paper, we consider subpopulation shifts as a source of algorithmic biases and study whether the common algorithmic fairness practice of enforcing performance parity on certain demographic groups mitigate the resulting (algorithmic) biases in the target domain. Such algorithmic fairness practices are common enough that there are methods [1, 2] and software (e.g. TensorFlow Constrained Optimization [10]) devoted to operationalizing them. There are other sources of algorithmic biases
*Equal Contribution.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
(e.g. posterior drift [17]), but we focus on algorithmic biases caused by subpopulation shifts in this paper. Our main contributions are:
1. We propose risk profiles as a way of summarizing the performance of ML models on subpopulations. As we shall see, this summary is particularly suitable for studying the performance of risk minimization methods. 2. We show that enforcing performance parity during training may not mitigate performance disparities in the target domain. In fact, it may even harm overall performance. 3. We decompose subpopulation shifts into two parts, a recoverable part orthogonal to the fair constraint and a non-recoverable part, and derive necessary and sufficient conditions on subpopulation shifts under which enforcing performance parity improves performance in the target domain (see Section 4.4).
One of the main takeaways of our study is a purely statistical way of evaluating the notion of algorithmic fairness for subpopulation shift: an effective algorithmic fairness practice should improve overall model performance in the target domain. Our theoretical results characterize when this occurs for risk-based notions of algorithmic fairness.

heading: 2 Problem setup
text: We consider a standard classification setup. Let X ⊂ Rd be the feature space, Y be the set of possible labels, and A be the set of possible values of the sensitive attribute. In this setup, training and test examples are tuples of the form (X,A, Y ) ∈ X × A × Y . If the ML task is predicting whether a borrower will default on a loan, then each training/test example corresponds to a loan. The features in X may include the borrower’s credit history, income level, and outstanding debts; the label Y ∈ {0, 1} encodes whether the borrower defaulted on the loan; the sensitive attribute may be the borrower’s gender or race.
Let P ∗ and P̃ be probability distributions on X ×A× Y . We consider P̃ as the distribution of the training data and P ∗ as the distribution of data in a hypothetical target domain. For example, P ∗ may be the distribution of data in the real world, and P̃ is a biased sample in which certain demographic groups are underrepresented. The difference P ∗ − P̃ is the distribution shift. In practice, distribution shifts often arise due to sampling biases during the (training) data collection process, so we call P ∗ and P̃ unbiased and biased respectively and refer to P ∗ − P̃ as the bias (in the training data). Henceforth E∗ (resp. Ẽ) will denote expectation under P ∗ (resp. P̃ ). The set of all hypotheses under consideration is denoted byH and ` : Y × Y 7→ R+ denotes the loss function under consideration. In Section 3 and 4 we assume the set of sensitive attribute A is discrete. The case with continuous A is relegated to the supplementary document (Appendix 2).

heading: 3 Benefits and drawbacks of enforcing risk parity
text: To keep things simple, we start by considering the effects of enforcing risk parity (RP). This notion is closely related to the notion of demographic parity (DP). Recall DP requires the output of the ML model h(X) to be independent of the sensitive attribute A: h(X) ⊥ A. RP imposes a similar condition on the risk of the ML model. Definition 3.1 (risk parity). A model h satisfies risk parity with respect to the distribution P on X × Y ×A if
EP [ `(h(X), Y ) | A = a ] = EP [ `(h(X), Y ) | A = a′ ] for all a, a′ ∈ A and all h ∈ H.
RP is widely used in practice to measure algorithmic bias in ML models. For example, the US National Institute of Standards and Technology (NIST) tested facial recognition systems and found that the systems misidentify blacks at rates 5 to 10 times higher than whites [21]. By comparing the error rates of the system on blacks and whites, NIST is implicitly adopting RP as its definition of algorithmic fairness.
It is not hard to see that RP is equivalent to linear constraints on the risk profile (RP̃ (h)) of an ML model with respect to P̃ :
RP (h) , { EP [ `(h(X), Y ) | A = a ]} a∈A . (3.1)
For notational simplicity defineR as the set of all risk profiles with respect to the training distribution P̃ , i.e. R , {R(h) | h ∈ H}. The risk profile of a model summarizes its performance on subpopulations. In terms of risk profiles, RP with respect to distribution P requires RP (h) = c1 for some constant c ∈ R. This is a linear constraint. The set of all risk profiles that satisfy the RP constraint with respect to the training distribution P̃ constitutes the following subspace:
FRP ≡ FRP(P̃ ) , {RP̃ (h) ∈ R|A| ∣∣ RP̃ (h) = c1,1 ∈ R|A|, c ∈ R}.
Therefore, we enforce RP by solving (the empirical version of) a constraint risk minimization problem {
minh∈H Ẽ [ `(h(X), Y ) ]
subject to RP̃ (h) ∈ FRP
} ≡ {
minR∈R 〈P̃A, RP̃ (h)〉 subject to RP̃ (h) ∈ FRP
} , (3.2)
where P̃A ∈ [0, 1]|A| is the marginal distribution of A and consequently 〈P̃A, R〉 = Ẽ [ `(h(X), Y ) ] . Note that, in (3.2), the inner product 〈·, ·〉 is euclidean inner product on R|A|. We define the minimizer of (3.2) as h̃F and its corresponding risk profile as R̃F = RP̃ (h̃F ). See Figure 1 for a graphical depiction of (3.2) when there are two groups (A = {0, 1}). Here we see the main benefit of summarizing model performance with risk profiles: risk minimization problems are equivalent to linear optimization problems in terms of risk profiles (objective and constrain functions are linear in terms of risk profile). This allows us to simplify our study of the effects of enforcing risk parity by reasoning about the risk profiles of the resulting models. Hereafter, we refer to this approach as fair risk minimization (FRM). It is not new; similar constrained optimization problems have appeared in the algorithmic fairness literature (e.g. see [1, 12, 10]). Our goal is evaluating whether this approach mitigates algorithmic biases and improves model performance in the target domain. There are efficient algorithms for solving (4.3). One popular algorithm is a reductions approach by Agarwal et al. [1], which solves a sequence of weighted classification problems with appropriately chosen weights to satisfy the desired algorithmic fairness constraints. This algorithm outputs randomized classifiers, which justifies the subsequent convexity assumption on the set of risk profiles.
In order to relate model performance in the training and target domains, some restrictions on the distribution shift/bias is necessary, as it is impossible to transfer performance parity during training to the target domain if they are highly disparate. At a high-level, we assume the distribution shift is a subpopulation shift [16]. Formally, we assume that the risk profiles of the models with respect to P ∗
and the profiles with respect to P̃ are identical:
E∗ [ `(h(X), Y ) | A = a ] = Ẽ [ `(h(X), Y ) | A = a ] for all a ∈ A, h ∈ H. (3.3)
i.e. RP̃ (h) = RP∗(h) for all h ∈ H. We note that this assumption is (slightly) less restrictive than the usual subpopulation shift assumption because it only requires the expected value of the loss (instead of the full conditional distribution of (X,Y ) givenA to be identical in the training and target domains. Furthermore, this assumption is implicit in enforcing risk-based notions of algorithmic fairness. If the risk profiles are not identical in the training and target domains, then enforcing risk-based notions of algorithmic fairness during training is pointless because performance parity during training may not generalize to the target domain. We are now ready to state our characterization of the benefits and drawbacks of enforcing RP. To keep things simple, we assume there are only two groups: a majority group and a minority group. As we shall see, depending on the marginal distribution of the subpopulations in the target domain P ∗A, enforcing RP can harm or improve overall performance in the target domain.
Theorem 3.2. Without loss of generality, let first entry of P̃A be the fraction of samples from the majority group in the training data i.e. P̃ (A = 1). Assume
1. there are only two groups and the set of risk profilesR ⊆ R2; 2. subpopulation shift: the risk profiles with respect to P̃ and P ∗ are identical; 3. (R̃1, R̃0) = R̃ , arg minR∈R〈P̃A, R〉 is the risk profile of the risk minimizer; 4. ((R̃F )1, (R̃F )0) = R̃F , arg minR∈R∩FRP〈P̃A, R〉 is the risk profile of the fair risk minimizer.
Then we have:
〈P ∗A, R̃〉 { ≤ 〈P ∗A, R̃F 〉 if P ∗(A = 1) ≥ R̃0−(R̃F )0R̃0−R̃1 ≥ 〈P ∗A, R̃F 〉 otherwise .
Therefore, enforcing RP harms overall performance in the target domain in the first case, while improves in the second.
In hindsight, this result is intuitive. If P ∗A is close to P̃A (e.g. the minority group is underrepresented in the training data but not by much), then enforcing RP may actually harm overall performance in the target domain. This is mainly due to the trade-off between accuracy and fairness in the IID setting (no distribution shift). If there is little difference between the training and target domains, then we expect the trade-off between accuracy and fairness to manifest (albeit to a lesser degree than in IID settings).

heading: 4 Benefits and drawbacks of enforcing conditional risk parity

heading: 4.1 Risk-based notions of algorithmic fairness
text: In this section, we consider more general risk-based notions of algorithmic fairness, namely Conditional Risk Parity (CRP) which is defined as follows: Definition 4.1 (Conditional Risk Parity). a model h ∈ H is said to satisfy CRP if:
EP [ `(h(X), Y ) | A = a, V = v ] = EP [ `(h(X), Y ) | A = a′, V = v ] (4.1)
for all a, a′ ∈ A, v ∈ V , where V is known as the discriminative attribute [18].
To keep things simple, we assume V is finite-valued, but it is possible to generalize our results to risk-based notions of algorithmic fairness with more general V ’s (see Appendix B). We also point out that this definition of CRP does not cover calibrations where one conditions on the model outcome Ŷ .
It is not hard to see that risk parity is a special case of (4.1) in which V is a trivial random variable. Another prominent instance is when V = Y , i.e. the risk profile satisfies:
EP [ `(h(X), Y ) | A = a, Y = y ] = EP [ `(h(X), Y ) | A = a′, Y = y ]
for all a, a′ ∈ A, y ∈ Y . Definition 4.1 is motivated by the notion of equalized odds (EO) [14] in classification. Recall EO requires the output of the ML model h(X) to be independent of the sensitive attribute A conditioned on the label: h(X) ⊥ A | Y . CRP imposes a similar condition on the risk of the ML model; i.e. the risk of the ML model must be independent of the sensitive attribute conditioned on the discriminative attribute (with label as a special case). Therefore CRP can
be viewed as a general version of EO, where we relax the conditional independence of h to equality of conditional means. CRP is also closely related to error rate balance [9] and overall accuracy equality [5] in classification.
Like RP, (4.1) is equivalent to linear constraints on the risk profiles of ML models. Here (with a slight abuse of notation) we define the risk profile of a classifier h under distribution P as:
RP (h) , { EP [ `(h(X), Y ) | A = a, V = v ]} a∈A,v∈V (4.2)
Compared to (3.1), (4.2) offers a more detailed summary of the performance of ML models on subpopulations that not only share a common value of the sensitive attribute A, but also a common value of the discriminative attribute V . The general fairness constraint (4.1) on the training distribution P̃ is equivalent to RP̃ (h) ∈ FCRP, where FCRP is a linear subspace defined as:
FCRP , {RP̃ (h) ∈ R|A|×|Y| | RP̃ (h) = 1u>,1 ∈ R|A|,u ∈ R|Y|, h ∈ H}. In this section, we study a version of (3.2) with this general notions of algorithmic fairness:
{ minh∈H Ẽ [ `(h(X), Y ) ]
subject to RP̃ (h) ∈ FCRP
} = { minR∈R 〈P̃A,V , R〉 subject to R ∈ FCRP } , (4.3)
where P̃A,V ∈ [0, 1]|A|×|V| is the marginal distribution of (A, V ), i.e. 〈P̃A,V , R〉 = Ẽ [ `(h(X), Y ) ] . As before we define the minimizer of (4.3) as h̃F and its corresponding risk profile as R̃F = RP̃ (h̃F ). We note that (4.2) has the same benefit as the definition in (3.1): the fair risk minimization problem in (4.3) is equivalent to a linear optimization problem in terms of the risk problems. This considerably simplifies our study of the efficacy of enforcing risk-based notions of algorithmic fairness.

heading: 4.2 Subpopulation shift in the training data
text: Similar to equation (3.3), we assume that the risk profiles with respect to P ∗ and P̃ are identical:
E∗ [ `(h(X), Y ) | A = a, V = v ] = Ẽ [ `(h(X), Y ) | A = a, V = v ] ∀ a ∈ A, v ∈ V, h ∈ H .
(4.4) i.e. RP̃ (h) = RP∗(h) for all h ∈ H. This definition of subpopulation shift (equation (4.4)) is borrowed from the domain adaptation literature (see [16, 19]). The difference in our definition is that we require equality of the expectations of the loss functions, whereas these works assume the distributions to be equal for the sub-populations. Note that under subpopulation shift RP̃ are RP? are equal overH. In the remaining part of Section 4 we shall drop the probability in subscript and denote them as R. We note the crucial role of the discriminative attributes in (4.4): the risk profiles are only required to be identical on subpopulations that share a value of the discriminative attribute. A good choice of discriminative attributes keeps the training data informative by ensuring the risk profiles are identical on the training data and at test time. Here are two examples of good discriminative attributes. Example 4.2 (Under-representation bias). In binary classification, training data may suffer from under-representation bias. This kind of bias arises when positive examples from disadvantaged groups are under-represented in the training data. Here is an example of a data generating process that suffers from label bias: (i) sample training examples (Xi, Yi, Ai) from P ∗, (ii) discard training examples from the disadvantaged group (Ai = 0) with positive label (Yi = 1) with probability β. This leads to P̃ (X,Y,A) ∝ P ∗(X,Y,A) · (1− (1− β)1{A = 0, Y = 1}). Because there are fewer positive examples from the disadvantaged group in the training data (compared to test data), this kind of bias causes the ML model to predict mostly negative outcomes for the disadvantaged group. In practice, this kind of bias may creep into the training data more subtly. For example, if human judgements is a crucial part of the data generating process, then implicit biases may lead to over-representation of negative examples from disadvantaged groups in the training data [24].
For training data with underrepresentation bias, a good choice of discriminative attribute is the label. This is because the training data is a filtered version of the data at test time, and the filtering process only depends on the label (and sensitive attribute). Thus the class conditionals at test time are preserved in the training data; i.e. P̃X|a,y = P ∗X|a,y for all a ∈ A, y ∈ Y .

heading: 4.3 Fair risk minimization may not improve overall performance
text: We start by showing that fair risk minimization may not improve overall performance. Without other stipulations, this is implied by a result similar to Theorem 3.2 for more general risk-based notions of algorithmic fairness. Perhaps more surprising, is fair risk minimization may not improve overall performance even if the Bayes decision rule in the target domain is algorithmically fair:
arg minR∈R〈P ∗A,V , R〉 ⊆ FCRP. Figure 2 shows such a problem instance. The triangle is the set of risk profiles, and the dotted bottom of the triangle intersects the fair constraint (i.e. the risk profiles on the dotted line are algorithmically fair). The training objective P̃ is chosen so that the risk profile of (unconstrained) risk minimizer on biased training data R̃ is the vertex on the top and the risk profile of fair risk minimizer (also on biased training data) R̃F is the vertex on left. The test objective points to the right, so points close to the right of the triangle have the best overall performance in the target domain. We see that R̃ is closer to the right of the triangle than R̃F , which immediately implies 〈P ∗, R̃〉 ≤ 〈P ∗, R̃F 〉, i.e. it has better performance in the target domain in comparison to R̃F . This counterexample is not surprising: the assumption that R∗ is fair is a constraint on P ∗,R, and F ; it imposes no constraints on P̃ . By picking P̃ adversarially, it is possible to have 〈P ∗, R̃〉 ≤ 〈P ∗, R̃F 〉.

heading: 4.4 When does fair risk minimization improve overall performance
text: The main result in this section provides necessary and sufficient conditions for recovering the unbiased Bayes’ classifier with (4.3). As the unbiased Bayes’ classifier is the (unconstrained) optimal classifier in the target domain, enforcing a risk-based notion of algorithmic fairness will improve overall performance in the target domain if it recovers the unbiased Bayes’ classifier.
At first blush, it is tempting to think that because the unbiased Bayes classifier satisfies CRP, then enforcing this constraint always increases accuracy, this is not the case as described in the previous paragraph. Our next theorem characterizes the precise condition under which it is possible to improve accuracy on the target domain by enforcing fairness constraint: Theorem 4.3. Under the assumptions
1. The risk setR is convex. 2. The risk profiles with respect to P̃ and P ∗ are
identical. 3. The unconstrained risk minimizer on un-
biased data is algorithmically fair; i.e. arg minR∈R〈P ∗, R〉 ⊆ FCRP.
the fair risk minimization (4.3) obtains h ∈ H such that R(h) = R∗ if and only if
ΠF (P ∗ A,V − P̃A,V )− P ∗A,V ∈ NR(R∗) + F⊥CRP, (4.5)
where P ∗A,V (resp. P̃A,V ) is the marginal of P ∗ (resp. P̃ ) with respect to (A, V ), R∗ is the optimal risk profile with respect to P ∗, NR(R∗) is the normal cone ofR at R∗ and ΠF is the projection on the fair hyperplane.
This assumption that R is convex is innocuous because it is possible to convexify the risk set by considering randomized decision rules. To evaluate a randomized decision rule H , we sample a decision rule h from H and evaluate h. It is not hard to see that the risk profiles of randomized decision rules are convex combinations of the risk profiles of (non-randomized) decision rules, so including randomized decision rules convexifies the risk set. The third assumption is necessary for recovery of the unbiased Bayes classifier. If the unbiased Bayes classifier is not algorithmically fair, then there is no hope for (4.3) to recover it as there will always be a non-negligible bias term. This assumption is also implicit in large swaths of the algorithmic fairness literature. For example, Buolamwini and Gebru [7] and Yang et al. [23] suggest collecting representative training data to improve the accuracy of computer vision systems on individuals from underrepresented demographic groups. This suggestion implicitly assumes the Bayes classifier on representative training data is algorithmically fair.
Theorem 4.3 characterizes the biases in the training data from which can be completely removed by enforcing appropriate algorithmic fairness constraints. The key insight from this theorem is a decomposition of the training bias into two parts: a part orthogonal to the fair constraint and the remaining part in NR(R∗). Enforcing an appropriate risk-based notion of algorithmic fairness overcomes the first part of the bias. This occurs regardless of the magnitude of this part of the bias (see Corollary 4.4), which is also evident from our computational results. The second part of the bias (the part inNR(R∗)) represents the “natural” robustness of R∗ to changes in P ∗: if P̃ is inNR(R∗), then the unconstrained risk minimizer on training data remains R∗. The magnitude of the bias in this set cannot be too large, and enforcing algorithmic fairness constraints does not help overcome this part of the bias. Although we stated our main result only for finite-valued discriminative attributes for simplicity of exposition, please see Appendix 2 for a more general version of Theorem 4.3 that applies to more general (including continuous-valued) discriminative attributes.
Corollary 4.4. A sufficient condition for (4.5) is P̃A,V − P ∗A,V ∈ F⊥CRP.
Corollary 4.4 allows large differences between P̃A,V and its unbiased counterpart P ∗A,V , as long as the differences are confined to F⊥. Intuitively, (4.3) enables practitioners to recover from large biases in F⊥ because the algorithmic fairness constraint “soaks up” any component of P̃A,V in F⊥. We explore the implications of Corollary 4.4 for risk parity and CRP.
Risk Parity: For RP, V is trivial random variable, hence P̃A − P ∗A ∈ F⊥RP means that it has mean 0. This is true for any P̃A as 〈P ∗A, 1〉 = 〈P̃A, 1〉 = 1. Hence, the Bayes’ classifier can be recovered under any perturbation. More specifically, recall the example of women historically underrepresented in STEM fields mentioned in the Introduction. Such training data is biased in its gender representation which differs at test time where women are better represented. Classifiers trained on biased data with the risk Parity fairness constraint will generalize better at test time.
Conditional risk parity: In this case V = Y and the condition P̃A,Y − P ∗A,Y ∈ F⊥CRP implies that the sum of each column of P̃A,Y − P ∗A,Y must be 0. Hence, to recover the Bayes classifier under equalized odds fairness constraints, we are allowed to perturb P ∗A,Y in such a way, that they have the same column sums: i.e. for any label, we are allowed to perturb the distribution of protected attributes for that label, but we have to keep the marginal distribution of the label to be same for both P̃A,Y and P ∗A,Y .
In practice, it is unlikely that the training bias is exactly orthogonal to the fair constraint, which happens only if the second part of the bias (i.e. the part in the normal cone at R∗) is small enough. Theorem 4.3 provides a general characterization along with a precise notion of this “small enough” condition. Remark 4.5. Theorem 4.3 can further be generalized for any discrete/continuous A and V (as defined in (4.1), the proof for the continuous case can be found in the supplementary document). Thus, our theory applies to many fairness constraints which fall under the setup in Equation (4.1), where V can be any discriminative attribute. However, our conditions do not cover calibration where one conditions on the model outcome Ŷ .

heading: 4.5 Related work
text: Most of the prior works on algorithmic fairness assume fairness is an intrinsically desirable property of an ML model, but this assumption is unrealistic in practice [1, 11, 25]. There is a small but growing line of work on how enforcing fairness helps ML models recover from bias in the training data. Kleinberg and Raghavan [15], Celis et al. [8] consider strategies for correcting biases in hiring processes. They show that correcting the biases not only increases the fraction of successful applicants from the minority group but also boosts the quality of successful applicants. Dutta et al. [13] study the accuracy-fairness trade-off in binary classification in terms of the separation of the classes within the protected groups. They explain the accuracy-fairness trade-off in terms of this separation and propose a way of achieving fairness without compromising separation by collecting more features.
Blum and Stangl [6] study how common group fairness criteria help binary classification models recover from bias in the training data. In particular, they show that the equal opportunity criteria [14] recovers the Bayes classifier despite under-representation and labeling biases in the training data. Our results complement theirs. Instead of comparing the effects of enforcing various fairness criteria on training data with two types of biases, we characterize the types of biases that the fairness criteria help overcome. Our results also reveal the geometric underpinnings of the constants that arise in Blum and Stangl’s results. Three other differences between our results and theirs are: (i) they only consider binary classification, while we consider all ML tasks that boil down to risk minimization, (ii) they allow some form of posterior drift (so the risk profiles of the models inH with respect to P ∗ and P̃ may differ in some ways), but only permit marginal drift in the label (V = Y ), (iii) their conditions are sufficient for recovery of the fair Bayes decision rule (in their setting), while our conditions are also necessary (in our setting).

heading: 5 Computational results
text: We verify the theoretical findings of the paper empirically. Our goal is to show that an algorithm trained with fairness constraints on the biased train data P̃ achieves superior performance on the true data generating P ∗ at test time in comparison to an algorithm trained without fairness considerations.
There are several algorithms in the literature that offer the functionality of empirical risk minimization subject to various fairness constraints, e.g. Cotter et al. [11] and Agarwal et al. [1]. Any such algorithm will suffice to verify our theory. In our experiments we use Reductions fair classification algorithm [1] with logistic regression as the base classifier. For the fairness constraint we consider Equalized Odds [14] (EO) — one of the major and more nuanced fairness definitions. We refer to Reductions algorithm trained with loose EO violation constraint as baseline and Reductions trained with tight EO violation constraint as fair classifier (please see Appendix 3 for additional details and supplementary material for the code).
Simulations.
We first verify the implications of Corollary 4.4 using simulation studies. We follow the Conditional risk parity scenario from Section 4. Specifically, consider a binary classification problem with two protected groups, i.e. Y ∈ {0, 1} and A ∈ {0, 1}. We set P ∗ to have equal representation of protected groups conditioned on the label and biased data P̃ to have one of the protected groups underrepresented. Specifically, let pay = PA=a,Y=y , i.e. the a, y indexed element of PA,Y ; pay = 0.25 ∀a, y for P ∗ and p1y = pminor, p0y = pmajor = 0.5 − pminor for P̃ . For both P ∗ and P̃ we fix class marginals p·0 = p·1 = 0.5 and generate Gaussian features X|A = a, Y = y ∼ N (µay,Σay) in 2-dimensions (see additional data generating details in Appendix C). In Figure 5 we show a
qualitative example of simulated train data from P̃ with pminor = 0.1 and test data from P ∗, and the corresponding decision boundaries of a baseline classifier and a classifier trained with the Equalized Odds fairness constraint (irregularities in the decision heatmaps are due to stochasticity in the Reduc-
tions prediction rule). In this example fair classifier is 3% more accurate on the test data and 1% less accurate on a biased test data sampled from P̃ (latter not shown in the figure)
We proceed with a quantitative study by varying degree of bias in P̃ via changing pminor in [0.01, 0.25] and comparing performance of the baseline and fair classifier on test data from P ∗
and P̃ . We present results over 100 runs of the experiment in Figure 4. Notice that the sum of each column of P̃A,Y −P ∗A,Y is 0 for any value of pminor and we observe that the fair classifier has almost constant accuracy on P ∗ (consistently outperforming the baseline), as predicted by Corollary 4.4. The largest bias in the training data corresponds to pminor = 0.01, where baseline is erroneous on the whole a = 1, y = 0 subgroup (cf. Figure 5) resulting in close to 75% accuracy corresponding to the remaining 3 (out of 4) subgroups. For pminor = 0.05 minority group acts as outliers causing additional errors at test time resulting in the worst performance overall. When pminor = 0.25, P̃ = P ∗ and all methods perform the same as expected. Results on P̃ correspond to the case where test data follows same distribution as train data, often assumed in the literature: here baseline can outperform fair classifier under the more extreme sampling bias conditions, i.e. pminor ≤ 0.1. We note that as the society moves towards eliminating injustice, we expect test data in practice to be closer to P ∗ rather then replicating biases of the historical train data P̃ .
Recidivism prediction on COMPAS data. We verify that our theoretical findings continue to apply on real data. We train baseline and fair classifier on COMPAS dataset [3]. There are two binary protected attributes, Gender (male and female) and Race (white and non-white), resulting in 4 protected groups A ∈ {0, 1, 2, 3}. The task is to predict if a defendant will re-
offend, i.e. Y ∈ {0, 1}. We repeat the experiment 100 times, each time splitting the data into identically distributed 70-30 train-test split, i.e. P̃ for train and test, and obtaining test set from P ∗ by subsampling test data to preserve Y marginals and enforcing equal representation at each of the 4 levels of the protected attribute A. Equal representation of the protected groups in P ∗ is sufficient for satisfying the assumption 3 of Theorem 4.3 under an additional condition that noise levels are similar across protected groups. For Conditional Risk Parity, condition in eq. (4.5) of Theorem 4.3 is satisfied as long as P̃ and P ∗ have the same Y marginals. Thus, we expect that enforcing EO will improve test accuracy on P ∗ in this experiment.
We present results in Table 1. We see that our theory holds in practice: accuracy of the fair classifier is 1.8% higher on P ∗. Baseline is expectedly more accurate on the biased test data from P̃ , but only by 0.8%. We present results for the same experimental setup on the Adult dataset [4] in Table 2 in Appendix C. We observe same pattern: in comparison to the baseline, fair classifier increases accuracy on P ∗, but is worse on the biased test data from P̃ .

heading: 6 Summary and discussion
text: In this paper, we studied the efficacy of enforcing common risk-based notions of algorithmic fairness in a subpopulation shift setting. This study is motivated by a myriad of examples in which algorithmic biases are traced back to subpopulations shifts in the training data (see [7, 22]). We show that enforcing risk-based notions of algorithmic fairness may harm or improve the performance of the trained model in the target domain. Our theoretical results precisely characterize when fair risk minimization harms and improves model performance. Practitioners should be careful and actually check that enforcing fairness is improving model performance in the target domain. For example, consider the Gender Shades [7] study which shows that the commercial gender classification algorithms are less accurate on dark-skinned individuals. A practitioner may attempt to mitigate this algorithmic bias by enforcing a RP, which leads to a fair model that sacrifices some performance on lighter-skinned individuals in exchange for improved accuracy on darker skinned individuals. Taking a step back, one of the main takeaways of our study is by considering whether enforcing a particular notion of algorithmic fairness improves model performance in the target domain, it is possible to compare algorithmic fairness practices in a purely statistical way. We hope this alleviates one of the barriers to broader adoption of algorithmic fairness practices: it is often unclear which fairness definition to enforce.
Acknowledgments and Disclosure of Funding
This note is based upon work supported by the National Science Foundation (NSF) under grants no. 1916271, 2027737, and 2113373. Any opinions, findings, and conclusions or recommendations expressed in this note are those of the authors and do not necessarily reflect the views of the NSF.

Review summaries:
The work examines when does a fairness constrained model improve in overall accuracy on a target dataset which has a different distribution (of loss conditioned on some covariates) from the train dataset. It provides necessary and sufficient conditions under which this happens. Experiments on synthetic data validate the theoretical results. Real world experiments on COMPASS and Adult income datasets are promising.

The paper studies the impact of constrained fairness risk minimization on model performance when data are affected by subpopulation shifts. The authors focus on two notions of fairness that are defined with respect to the loss of the model, and that they call "risk parity" and "conditional risk parity" respectively.  Under a series of assumptions, the authors derive the conditions under which a model model trained via constrained risk minimization has overall higher/lower performance on the target data (vs. train data). They then examine the resulting bounds via simulations and experiments. 

The authors analyze the effect of enforcing algorithmic fairness (performance parity) in the presence of subpopulation shift. They argue that model performance can be summarized in terms of risk profiles, as risk minimization problems are equivalent to linear optimization wrt risk profiles. Basically, the authors consider ML problems that can be mapped to risk minimization. The paper also provides a way to analyze subpopulation shift by decomposing the training bias into (1) a portion which is orthogonal to the fair constraint, and (2) a robustness of $R^\star$ to changes in the distribution of the data in the hypothetical target domain ($P^\star$). Lastly, the authors show that fair risk minimization while training does not necessarily improve performance---a theoretical analysis is provided that shows when improvement and harm is done to model performance. Computational results are shown over synthetic data and recidivism data (COMPAS) to support results.