Does enforcing fairness mitigate biases caused by subpopulation shift?
Authors: Subha Maity
Debarghya Mukherjee
Mikhail Yurochkin
Yuekai Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: algorithmic fairness
domain adaptation
sub-population shift
fairness accuracy trade-off

Abstract: Many instances of algorithmic bias are caused by subpopulation shifts. For example, ML models often perform worse on demographic groups that are underrepresented in the training data. In this paper, we study whether enforcing algorithmic fairness during training improves the performance of the trained model in the \emph{target domain}. On one hand, we conceive scenarios in which enforcing fairness does not improve performance in the target domain. In fact, it may even harm performance. On the other hand, we derive necessary and sufficient conditions under which enforcing algorithmic fairness leads to the Bayes model in the target domain. We also illustrate the practical implications of our theoretical results in simulations and on real data.

Does enforcing fairness mitigate biases caused by subpopulation shift?
Subha Maity
Debarghya Mukherjee" Department of Statistics University of Michigan mdeb@umich ch edu
Department of Statistics University of Michigan smai ty@umi ch. edu
Mikhail Yurochkin
Yuekai Sun Department of Statistics University of Michigan yuekai @umich edu
IBM Research
MIT-IBM Watson AI Lab mikhail, www.clkuuochhiiiieee bm. com
Abstract
Many instances of algorithmic bias are caused by subpopulation shifts. For example, ML models often perform worse demographic groups that underrepresented in the training data. In this paper, we study whether enforcing algorithmic fairness during training improves the performance of the trained model in the target domain. On one hand, we conceive scenarios which enforcing fairness does not improve performance in the target domain. In fact, it may even harm performance. On the other hand, we derive necessary and sufficient conditions under which enforcing algorithmic fairness leads to the Bayes model in the target domain. We also illustrate the practical implications of our theoretical results in simulations and on real data.
Introduction
There many instances of distribution shifts causing performance disparities in machine learning (ML.) models. For example, Buolamwini and Gebru [ report commercial gender classification models are more likely misclassify dark-skinned people (than light-skinned people). This is (in part) due to the abundance of light-skinned examples in training data. Similarly, pedestrian detection models sometimes have trouble recognizing dark-skinned pedestrians [22). Another prominent
example is the poor performance of image processing models on images from developing countries
due to the scarcity of images from such countries in publically available image datasets [20].
Unfortunately, many algorithmic faimess practices were not developed with distribution shifts in mind. For example, the common algorithmic fairness practice of enforeing performance parity certain demographic groups [ implicitly assumes performance parity on training data generalize the target domain, but distribution shifts between the training data and target domain renders this assumption invalid.
In this paper, we consider subpopulation shifts as source of algorithmic biases and study whether the common algorithmic fairness practice enforcing performance parity on certain demographic groups mitigate the resulting (algorithmic) biases in the target domain. Such algorithmic fairness practices are common enough that there are methods [ and software (e.g. TensorFlow Constrained Optimization [10] devoted to operationalizing them. There other sources of algorithmic biases
EEqual Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(e.g. posterior drift [7]), but we focus on algorithmic biases caused by subpopulation shifts in this paper. Our main contributions are:
1. We propose risk profiles as a way of summarizing the performance of ML models on subpopula- tions. As we shall see, this summary is particularly suitable for studying the performance of risk minimization methods.
2. We show that enforcing performance parity during training may not mitigate performance dispari- ties in the target domain. In fact, it may even harm overall performance.
3. We decompose subpopulation shifts into two parts, recoverable part orthogonal to the fair con- straint and a non-recoverable part, and derive necessary and sufficient conditions on subpopulation shifts under which enforcing performance parity improves performance in the target domain (see Section 4.4).
One of the main takeaways of our study is purely statistical way of evaluating the notion of algorithmic fairness for subpopulation shift: an effective algorithmic fairness practice should improve overall model performance in the target domain. Our theoretical results characterize when this occurs for risk-based notions of algorithmic fairness.
2 Problem setup
We consider a standard classification setup. Let X' C. Rd be the feature space, y be the set of possible labels, and A be the set of possible values of the sensitive attribute. In this setup, training and test examples are tuples of the form (X, A, Y) E X' x A y. If the ML task is predicting loan, then each training/test example corresponds to loan. The
whether borrower will default features in X may include the borrower's credit history, income level, and outstanding debts; the label Y E {0, } encodes whether the borrower defaulted the loan; the sensitive attribute may be the borrower's gender or race.
Let P and P be probability distributions on X' x A x V. We consider P as the distribution of the training data and P* the distribution of data in a hypothetical target domain. For example, P* may be the distribution of data the real world, and P is biased sample in which certain demographic groups are underrepresented. The difference p shifts often arise due sampling biases during the (training) data collection process, we call P' and P unbiased and biased respectively and refer to p P the bias (in the training data). Henceforth E* (resp. E) will denote expectation under p* (resp. P). The set of all hypotheses under consideration is denoted by H and é: y x y H R., denotes the loss function under consideration. In Section and we assume the set of sensitive attribute A is discrete. The case with continuous A is relegated to the supplementary document (Appendix
the distribution shift. In practice, distribution
Benefits and drawbacks of enforcing risk parity
To keep things simple, we start by considering the effects of enforcing risk parity (RP). This notion closely related to the notion of demographic parity (DP). Recall DP requires the output of the ML model ((X) to be independent of the sensitive attribute A: 1(() 1 A. RP imposes similar condition on the risk of the ML model.
Definition 3.1 (risk parity). A model h satisfies risk parity with respect to the distribution P on A if Er (((((N((),))) A = a] = = (E(((X))) |A=
X x
| a'] for all a, d' e A and all h e H.
RP is widely used in practice measure algorithmic bias in ML models. For example, the US National Institute of Standards and Technology (NIST) tested facial recognition systems and found that the systems misidentify blacks rates 5 to 10 times higher than whites (21). By comparing the eитоr rates of the system on blacks and whites, NIST is implicitly adopting RP as its definition of algorithmic fairness.
is not hard to see that RP equivalent to linear constraints on the risk profile (Rp(h) of an ML model with respect to P:
Rh(h) & ( A A= a]]
(3.1)

---

F
0.70
mine
minorit
Acaracy
6.00
Basciine P
0.3
majority
majonity
05
P
Figure I: Fair risk minimization problem when there are groups. Recall R is set of risk profiles and F is the set of risk profiles that satisfy risk parity. The horizontal and vertical coordinates of the risk profiles represent the risk of the model on the majority and minority subpopulations. In the left plot, we see the empirical risk minimization (ERM) optimal point R and the fair risk minimization (FRM) optimal point Ry. In the center plot, we see that FRM can both improve and harm performance in the target domain (as long the assumptions of are satisfied). The green dotted line separates the PA 's that lead to worse and improved performance in the target domain: if PA falls below the green line, then FRM harms performance in the target domain. In the right plots, we reproduce this effect in a simulation. As the fraction of samples from the minority group decreases in the target domain, there is a point beyond which enforcing fairness harms accuracy (in the target domain). We refer to Appendix for the simulation details.
For notational simplicity define R the set of all risk profiles with respect the training distribution P. i.e. R & {R(h) h H]. The risk profile of model summarizes its performance on subpopulations. In terms of risk profiles, RP with respect to distribution P requires Rp = cl for some constant E R. This is a linear constraint. The of all risk profiles that satisfy the RP constraint with respect the training distribution P constitutes the following subspace:
FR = FRr A (Rp(h) E R| (h(h) cl, E TR E IR..
Therefore, we enforce RP by solving (the empirical version of) constraint risk minimization problem
minAEH subject to Rp(A) FRP
[[((h(x),
minRer (PA,))(h) subject to R₂(h) FRr
(3.2)
where PA E j0, 1| is the marginal distribution of A and consequently (PA, ={((((())))
Note that, in (3.2) the inner product (+-)) is euclidean inner product on R R| We define the minimizer of (3.2) hF and its corresponding risk profile as R2= (hx). See Figure 1 for graphical depiction of (3.2) when there are two groups (A {0. 1J). Here we see the main benefit of summarizing model performance with risk profiles: risk minimization problems are equivalent to linear optimization problems in terms of risk profiles (objective and constrain functions are linear in terms of risk profile). This allows to simplify our study of the effects of enforeing risk parity by reasoning about the risk profiles of the resulting models. Hereafter, we refer to this approach fair risk minimization (FRM). It is not new; similar constrained optimization problems have appeared in the algorithmic fairness literature (e.g. see [ (0). Our goal is evaluating whether this approach mitigates algorithmic biases and improves model performance
12
the target domain. There are efficient algorithms for solving (4.3). One popular algorithm is reductions approach by Agarwal al. M. which solves a sequence of weighted classification problems with appropriately chosen weights to satisfy the desired algorithmic fairness constraints. This algorithm outputs randomized classifiers, which justifies the subsequent convexity assumption on the set risk profiles.
In order to relate model performance in the training and target domains, some restrictions on the distribution shift/bias is necessary, as it impossible to transfer performance parity during training to the target domain if they are highly disparate. At high-level, we assume the distribution shift is subpopulation shift [16]. Formally, we assume that the risk profiles of the models with respect to P and the profiles with respect to P are identical:
E [((h(X), A= E[((h(X), A for A, E H.
(3.3)
3

---

Le. Rp(h)- Rp-(h) (h) for all h E H. We note that this assumption is (slightly) less restrictive than the usual subpopulation shift assumption because only requires the expected value of the loss (instead of the full conditional distribution of (X, Y) given A to be identical in the training and target domains. Furthermore, this assumption is implicit in enforeing risk-based notions of algorithmic fairness. the risk profiles are not identical in the training and target domains, then enforcing risk-based notions of algorithmic fairness during training is pointless because performance parity during training may not generalize to the target domain. We are now ready to state our characterization of the benefits and drawbacks enforcing RP. To keep things simple, we assume there are only two groups: a majority group and minority group. As shall see, depending the marginal distribution of the subpopulations in the target domain PA enforcing RP can harm or improve overall performance in the target domain.
Theorem 3.2. Without loss of generality, let first entry of PA be the fraction of samples from the majority group in the training data i.e. P(A = 1). Assume
1. there are only two groups and the of risk profiles R R²;
2. subpopulation shift: the risk profiles with respect P and p are identical;
(Ra, Ro) -R f arg min (PA, R) is the risk profile of the risk minimizer;
4. ((RF)1, (R=)o)= = R & arg mimmminneenne REROJaP (PA, R) is the risk profile of the fair risk minimizer.
Then we have:
Ro-(Fi)o
$ (PA. Ry)
(A=(A=1)2
(PA, Rt)
Ro- Rs
2 (PA. Rz)
otherwise.
Therefore, enforcing RP harms overall performance in the target domain in the first case, while improves in the second.
In hindsight, this result is intuitive. If PA is close to PA (e.g. the minority group is underrepresentedd in the training data but not by much), then enforeing RP may actually harm overall performance in the target domain. This is mainly due to the trade-off between accuracy and fairness in the IID setting (no distribution shift). If there is little difference between the training and target domains, then expect the trade-off between accuracy and faimess to manifest (albeit to a lesser degree than in IID settings).
4 Benefits and drawbacks of enforcing conditional risk parity
4.1 Risk-based notions of algorithmic fairness
In this section, we consider more general risk-based notions of algorithmic fairness, namely Condi- tional Risk Parity (CRP) which is defined as follows:
Definition 4.1 (Conditional Risk Parity). model h E H. is said to satisfy CRP if:
Ep [e(((()), A= a
E. (E([(((()))) A = d', v'= v]
(4.1)
for all a' E A. v LAVEE V, where V is known as the discriminative attribute [88]
To keep things simple, we assume is finite-valued, but it is possible to generalize our results to risk-based notions of algorithmic faimess with more general V's (see Appendix B). We also point out that this definition of CRP does not cover calibrations where conditions the model outcome Y.
is not hard to see that risk parity is special case of (4.1 in which V is a trivial random variable. Another prominent instance is when V = Y, i.e. the risk profile satisfies:
Ep [e(h(X), V) = a, Y y] = Ep [((M)()))))))))))))) Y) A=
for all a, a E A, D. Definition 4.1 is motivated by the notion of equalized odds (EO) 14 in classification. Recall EO requires the output of the ML model l(X) to be independent of the sensitive attribute A conditioned on the label: l(()) J. A Y. CRP imposes similar condition on the risk of the MI model; i.e. the risk of the ML. model must be independent of the sensitive attribute conditioned on the discriminative attribute (with label as special case). Therefore CRP can

---

be viewed as general version of EO, where we relax the conditional independence of h to equality of conditional means. CRP is also closely related error rate balance 9 and overall accuracy equality 5 in classification.
Like RP, 4. is equivalent to linear constraints the risk profiles of MI models. Here (with slight abuse of notation) we define the risk profile of a classifier h under distribution /' as:
(h) - {Ep (((h(X). Y) = a, = b aE)
(4.2)
offers more detailed summary of the performance of ML models on subpopulations that not only share common value of the sensitive attribute A, but also common value of the discriminative attribute V. The general fairness constraint (4.1) on the training distribution P is equivalent to Rp(h) E FCRP, where FCRP is linear subspace defined as:
Compared [3.
FCRP & (Kp(h) RIAXXDI Rp(h) lu g u E R h E H}.
In this section, we study a version of (32) with this general notions of algorithmic faimess:
minsew
[[((M(x),)) }
minneR
(PAv, R)
subject to R E FCRP S
(4.3)
subject to Rp(h)= E FORP
where PA.v E [0,1]|A is the marginal distribution of (A.
As before we define the minimizer of (4.3) as h5 and its corresponding risk profile as R, (hx)) We note that (4.2) has the same benefit as the definition in (3.1): the fair risk minimization problem in (43 is equivalent to a linear optimization problem in terms of the risk problems. This considerably simplifies our study of the efficacy of enforcing risk-based notions of algorithmic faimess.
4.2
Subpopulation shift in the training data
Similar to equation 3.3 assume that the risk profiles with respect to P and P are identical:
E' [e(((()) A V = v] = ((((M()))Y) A = a, V uj Va AvEV, H.
(4.4)
Lee (h) Rp- (h) for all h H. This definition of subpopulation shift (equation (4.4)) is borrowed from the domain adaptation literature (see [66 19). The difference in our definition that we require equality of the expectations of the loss functions, whereas these works assume the distributions to be equal for the sub-populations. Note that under subpopulation shift Rs are Rp- are equal over H. In the remaining part of Section 4 we shall drop the probability in subscript and denote them as R. We note the crucial role of the discriminative attributes in (4.4): the risk profiles are only required to be identical subpopulations that share value of the discriminative attribute. A good choice of discriminative attributes keeps the training data informative by ensuring the risk profiles identical the training data and at test time. Here two examples of good discriminative attributes.
Example 4.2 cuneettreprrsentttioon bias). binary classification, training data may suffer from rnderrrepresentatioon bias. This kind of bias arises when positive examples from disadvantaged groups are under- that suffers from label bias: (i) sample training examples (X,, frrm P*, (ii) discard training examples from the disadvantaged group (A, = 0) with positive label (Y, 1) with probability B3. This leads to
under-represent represented in the training data. Here an example of a data generating process
P(X,Y,A) x
Because there are fewer positive examples from the disadvantaged group in the training data (com- pared to test data), this kind of bias causes the ML model to predict mostly negative outcomes for the disadvantaged group. In practice this kind of bias may creep into the training data more subtly. For example, if human judgements is crucial part of the data generating process, then implicit biases may lead to overrrepresnntatinn of negative examples from disadvantaged groups in the training data (24).
For training data with underrepresentaaion bias, good choice of discriminative attribute is the label. This is because the training data is a filtered version of the data at test time, and the filtering process only depends on the label (and sensitive attribute). Thus the class conditionals at test time preserved in the training data; i.e. Px a for all A. y y
5

---

A.
Figure 2: Example in which enforcing algorithmic fairness harms performance in the target domain despite the Bayes decision rule in the target domain satisfying the algorithmic fairness constraint.
4.3 Fair risk minimization may not improve overall performance
We start by showing that fair risk minimization may improve overall performance. Without other stipulations, this is implied by result similar to Theorem 3.2 for more general risk-based notions of algorithmic fairness. Perhaps more surprising. is fair risk minimization may not improve overall performance even if the Bayes decision rule in the target domain is algorithmically fair:
arg min (PA.v v. R) C FCRP.
Figure 2 shows such a problem instance. The triangle is the set of risk profiles, and the dotted bottom of the triangle intersects the fair constraint (i.e. the risk profiles the dotted line algorithmically fair). The training objective P is chosen so that the risk profile of (unconstrained) risk minimizer on biased training data R is the vertex on the top and the risk profile of fair risk minimizer (also on biased training data) R is the vertex on left. The test objective points to the right, so points close to the right of the triangle have the best overall performance in the target domain. We see that R is closer to the right of the triangle than Rz, which immediately implies (P*, It) S (P*, R2), i.e. has better performance in the target domain in comparison Ry. This counterexample is not surprising: the assumption that Rt is fair is a constraint on P R. and F; it imposes no constraints on P. By picking P adversarially, it is possible to have (P*, R2) < (P*, (P",
4.4 When does fair risk minimization improve overall performance
The main result in this section provides necessary and sufficient conditions for recovering the unbiased Bayes' classifier with 4.3 As the unbiased Bayes' classifier is the (unconstrained) optimal classifier in the target domain, enforcing risk-based notion of al- gorithmic fairness will improve overall performance the target domain it recovers the unbiased Bayes' classifier.
F
R
R
At first blush, it is tempting to think that because the unbiased Bayes classifier satisfies CRP, then enforc- ing this constraint always increases accuracy, this is not the case as described in the previous paragraph. Our next theorem characterizes the precise condition under which it is possible to improve accuracy on the target domain by enforcing fairness constraint:
-P*
E[((A(X) YIA== 0]
Figure 3: Total recovery from training bias by enforcing risk parity. In this example, the training bias P - P* is always orthogonal to the risk parity constraint (blue line) because P and pa are probability distributions. When the training bias does not affect the risk pro- files, enforcing risk parity allows us to totally overcome the training bias. Unfortunately, to show an example in which the risk decom- poses into recoverable and non-recoverable parts, we need (at least) two more dimensions.
Theorem 4.3. Under the assumptions
1. The risk set R is convex. The risk profiles with respect to P and P are identical.
The unconstrained risk minimizer on un- biased data is algorithmically fair; i.e. arg min (P*, RP, FCRP.
the fair risk minimization 4.. that R(h) R if and only if
obtains h E H such
II PAV) - PAV E NR(H") + Fdee.
(4.5)

---

where Pay (resp. PA.)) is the marginal of P (resp. P) with respect to (A,V), R' is the optimal risk profile with respect to P", (NRRRR) is the normal cone of R at R and Ty is the projection on the fair hyperplane.
This assumption that R is convex is innocuous because it is possible to convexify the risk set by considering randomized decision rules. To evaluate a randomized decision rule H, we sample decision rule from H and evaluate h. is not hard to see that the risk profiles of randomized decision rules are convex combinations of the risk profiles of (non-randomized) decision rules, so including randomized decision rules convexifies the risk set. The third assumption is necessary for recovery of the unbiased Bayes classifier. If the unbiased Bayes classifier is not algorithmically fair, then there is no hope for (4. to recover it as there will always be a non-negligible bias term. This assumption is also implicit in large swaths of the algorithmic fairness literature. For example, Buolamwini and Gebru 77] and Yang et al. [23] suggest collecting representative training data to improve the accuracy of computer vision systems individuals from underrepresented demographic groups. This suggestion implicitly assumes the Bayes classifier on representative training data is algorithmically fair.
Theorem 4.3 characterizes the biases in the training data from which can be completely removed by enforcing appropriate algorithmic fairness constraints. The key insight from this theorem is a decomposition of the training bias into two parts: a part orthogonal to the fair constraint and the remaining part in No RR)) Enforcing appropriate risk-based notion of algorithmic fairness overcomes the first part of the bias. This occurs regardless of the magnitude of this part of the bias (see Corollary (.4). which is also evident from our computational results. The second part of the bias (the part in NR (R+)) represents the "natural" robustness of Ht to changes in p"; P in NR (R²). then the unconstrained risk minimizer on training data remains Rt. The magnitude of the bias in this set cannot be too large, and enforcing algorithmic fairness constraints does not help overcome this part of the bias. Although we stated our main result only for finite-valued discriminative attributes for simplicity of exposition, please see Appendix [oor a more general version of Theorem 4.3 that applies to more general (including continuous-valued discriminative attributes.
Corollary 4.4. A sufficient condition for 4.5 is PA.Y Pa, E FFRR
Corollary 4.4 allows large differences between PA.v and its unbiased counterpart P PA as long as the differences are confined to F+. Intuitively, (43) enables practitioners to recover from large biases in F because the algorithmic fairness constraint "soaks up" any component of Pa.v v in F+. We explore the implications of Corollary for risk parity and CRP.
Risk Parity: For RP, V is trivial random variable, hence PA PA E Fur RP means that it has mean This true for any PA as (P).1) = (PA. = 1. Hence, the Bayes' classifier can be recovered under any perturbation. More specifically, recall the example of women historically underrepresented in STEM fields mentioned the Introduction. Such training data is biased its gender representation which differs at test time where women are better represented. Classifiers trained on biased data with the risk Parity fairness constraint will generalize better at test time.
Conditional risk parity: In this case = Y and the condition PAY PIAy E FORP implies that the sum of each column of PAAY Pā. must be 0. Hence, to recover the Bayes classifier under equalized odds faimess constraints, we are allowed to perturb Phay in such way, that they have the same column sums: i.e. for any label, we are allowed to perturb the distribution of protected attributes for that label, but we have to keep the marginal distribution of the label to be same for both PAA and PAYY
In practice, it is unlikely that the training bias is exactly orthogonal to the fair constraint, which happens only if the second part of the bias (i.e. the part in the normal cone at R*) is small enough. Theorem 4.3 provides a general characterization along with a precise notion of this "small enough" condition.
Remark 4.5. Theorem 4.3 can further be generalized for any discretelcontinuous and V (as defined in (4.1), the proof for the continuous case can be found in the supplementary document). Thus, our theory applies to many fairness constraints which fall under the setup in Equation (4.1), where V can be any discriminative attribute. However, our conditions do not cover calibration where one conditions the model outcome Y.
7

---

4.5 Related work
Most of the prior works on algorithmic faimesss assume faimess is an intrinsically desirable property of an ML model, but this assumption is unrealistic in practice TT 25]. There is a small but growing line of work on how enforcing fairness helps ML. models recover from bias in the training data. Kleinberg and Raghavan [15], Celis al. [ consider strategies for correcting biases in hiring processes. They show that correcting the biases not only increases the fraction of successful applicants from the minority group but also boosts the quality of successful applicants. Dutta et al. [13] study the accuracy-fairness trade-off in binary classification in terms of the separation of the classes within the protected groups. They explain the accuracy- accuracy-fairness trade-off in terms of this separation and propose a way of achieving fairness without compromising separation by collecting more features.
Blum and Stangl 66] study how common group fairness criteria help binary classification models recover from bias in the training data. In particular, they show that the equal opportunity criteria [14] recovers the Bayes classifier despite under-representation and labeling biases in the training data. Our results complement theirs. Instead of comparing the effects of enforcing various fairness criteria on training data with two types of biases, we characterize the types of biases that the fairness criteria help overcome. Our results also reveal the geometric underpinnings of the constants that arise in Blum and Stangl's results. Three other differences between our results and theirs are: (i) they only consider binary classification, while we consider all ML tasks that boil down to risk minimization, (ii) they allow some form of posterior drift (so the risk profiles the models in H with respect to P and P may differ in some ways), but only permit marginal drift in the label (V = Y), (iii) their conditions sufficient for recovery of the fair Bayes decision rule (in their setting), while our conditions also necessary (in our setting).
5 Computational results
We verify the theoretical findings of the paper empirically. Our goal to show that an algorithm trained with fairness constraints on the biased train data P achieves superior performance on the true data generating P* at test time in comparison to an algorithm trained without fairness considerations.
There are several algorithms in the literature that offer the functionality of empirical risk minimization subject to various fairness constraints, e.g. Cotter et al. [1]] and Agarwal et al. [. Any such algorithm will suffice to verify our theory. In our experiments we use Reductions fair classification algorithm [ with logistic regression as the base classifier. For the fairness constraint we consider Equalized Odds [44 (EO)- - one of the major and more nuanced fairness definitions. We refer to Reductions algorithm trained with loose EO violation constraint as baseline and Reductions trained with tight EO violation constraint as fair classifier (please see Appendix for additional details and supplementary material for the code).
Simulations.
We first verify the implications of Corollary 4.4 using simulation studies. We follow the Conditional risk par- ity scenario from Section 4 Specifically, consider a bi- nary classification problem with two protected groups, i.e. Y E 1) and A E (0. 1). We set Pa to have equal rep- resentation of protected groups conditioned on the label
0.95
Fair on P Baseline P on Baseline
0.90
accuracy
0.85
0.80
and biased data P have one of the protected groups un- derrepresented. Specifically, let Pay PA Yo i.e. the a, y indexed element of PA,Y: Pay = 0.25 Va, y for p*
0.70
and Piy = Pminor, Poy Pmajor = 0.5 Pminnr for P. For both pa and P we fix class marginals p.o p.1 0.5 and generate Gaussian features X A = Y = y
0.01
0.05
0.10
0.15
0.20
0.25
Pminor
Figure 4: Test acc. on P and P when trained the (biased) data from P.
M(Hay May) in 2-dimensions (see additional data gen- erating details in Appendix C). In Figure 5 we show a
qualitative example of simulated train data from 1 with Pmiior = 0.1 and test data from Pa and the corresponding decision boundaries of a baseline classifier and a classifier trained with the Equalized Odds faimess constraint (irregularities in the decision heatmaps are due to stochasticity in the Reduc-
8

---

Figure 5: Decision heatmaps for (left) baseline on train data from P; (center left) fair classifier on train data from P: (center right) baseline on test data from P"; (right) fair classifier on test data from Pt Decision boundary the fair classifier has larger slope better accounting for the group a = 1 underrepresented in the train data. Consequently its performance is better the unbiased test data.
tions prediction rule). In this example fair classifier is 3% more accurate on the test data and 1% less accurate biased test data sampled from P (latter not shown in the figure)
We proceed with quantitative study by varying degree of bias in P via changing Pminnor in [0.01, 0.25] and comparing performance of the bascline and fair classifier on test data from P and P. We present results over 100 runs of the experiment in Figure 4 Notice that the sum of each PA,y column of PA.Y -xy--Pxy is O for any value of Pminnr and we observe that the fair classifier has almost constant accuracy on P (consistently outperforming the baseline), as predicted by Corollary 4.4 The largest bias in the training data corresponds to Pminor 0.01, where baseline erroneous on the whole a = 1, = O subgroup (cf. Figure resulting in close to 75% accuracy corresponding the remaining (out of 4) subgroups. For Priiner 0.05 minority group acts as outliers causing additional errors at test time resulting in the worst performance overall. When Pminor
0.25,
p= Ps and all methods perform the same as expected. Results on P correspond to the case where test data follows same distribution as train data, often assumed in the literature: here baseline can outperform fair classifier under the more extreme sampling bias conditions, i.e. Pminor I 0.1. We note that as the society moves towards eliminating injustice, we expect test data in practice to be closer to p* rather then replicating biases of the historical train data P.
Recidivism prediction COMPAS data. We verify that our theoretical findings continue to apply real data. We train baseline and fair classifier on COMPAS dataset [3]. There are two binary protected attributes, Gender (male and female) and Race (white and non-white), resulting in 4 protected groups A E {0, 1, The task is to predict if defendant will re-
Table 1: Accuracy on COMPAS data
TEST ON P
TEST ON
FAIR BASELINE
0.652+0.013 0.63410.011
0.660--0.00 0.668110.010
offend, i.c. Y E {0, 13. We repeat the experiment 100 times, each time splitting the data into identically distributed 70-30 train-test split, i.e. P for train and test, and obtaining test set from P* by subsampling test data to preserve Y marginals and enforcing equal representation at each of the levels of the protected attribute A. Equal representation of the protected groups in P is sufficient for satisfying the assumption 3 of Theorem 4.3 under additional condition that noise levels similar across protected groups. For Conditional Risk Parity, condition in eq. (4.5) of Theorem 4.3 is satisfied as long as P and Pa have the same Y marginals. Thus, we expect that enforcing EO will improve test accuracy on p* in this experiment.
We present results Table I We see that our theory holds in practice: accuracy of the fair classifier is 1.8% higher on pa Baseline is expectedly more accurate on the biased test data from P. but only by 0.8%. We present results for the same experimental setup on the Adult dataset 4 in Table 2 in Appendix C. We observe same pattern: in comparison to the baseline, fair classifier increases accuracy on P but is worse on the biased test data from P.

---

Summary and discussion
In this paper, we studied the efficacy of enforcing common risk-based notions of algorithmic fairness in subpopulation shift setting. This study is motivated by myriad of examples in which algorithmic biases are traced back to subpopulations shifts in the training data (see 7 222). We show that enforcing risk-based notions of algorithmic fairness may harm improve the performance of the trained model in the target domain. Our theoretical results precisely characterize when fair risk minimization harms and improves model performance. Practitioners should be careful and actually check that enforcing fairness is improving model performance in the target domain. For example, consider the Gender Shades [7] study which shows that the commercial gender classification algorithms are less accurate on dark-skinned individuals. A practitioner may attempt to mitigate this algorithmic bias by enforcing RP, which leads to fair model that sacrifices some performance on lighter-skinned individuals in exchange for improved accuracy on darker skinned individuals. Taking step back, one of the main takeaways of our study is by considering whether enforcing particular notion of algorithmic fairness improves model performance in the target domain, it is possible to compare algorithmic fairness practices in purely statistical way. We hope this alleviates one of the barriers to broader adoption of algorithmic faimess practices: it is often unclear which fairness definition to enforce.
Acknowledgments and Disclosure of Funding
This note is based upon work supported by the National Science Foundation (NSF) under grants 1916271, 2027737, and 21 13373. Any opinions, findings, and conclusions recommendations expressed in this note are those of the authors and do not necessarily reflect the views the NSF.
References
[1] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A Reductions Approach to Fair Classification. arXiv:1803.02453 [cs]. July 2018
[2] Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair Regression: Quantitative Definitions and Reduction-based Algorithms. arXiv: /2843 [cs, stat], May 2019.
[3] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias. May 2016.
[4] K. Bache and M. Lichman. UCI machine learning repository. 2013.
[5] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in Criminal Justice Risk Assessments: The State of the Art. UrXiv:1703.09200 [stat], March 2017.
[6] Avrim Blum and Kevin Stangl. Recovering from Biased Data: Can Fairness Constraints Improve Accuracy? 1912:191 [cs, stat], December 2019.
[7] Joy Buolamwini and Timnit Gebru, Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of Machine Learning Research, volume 87, pages 77-91, 2018.
[8] L. Elisa Celis, Anay Mehrotra, and Nisheeth K. Vishnoi. Interventions for ranking in the presence of implicit bias. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pages 369-380, New York, NY, USA, January 2020. Association for Computing Machinery. ISBN 978- 1-500366936-7. doi: 10.1
[9] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. 1703.0 (cs, staf], February 2017.
[10] Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You. Training Well-Gennrrllizing Classifiers for Fairness Metrics and Other Data- Retandeeenddett Constraints. arXiv: [cs, stat], September 2018.
10

---

[11] Andrew Cotter, Heinrich Jiang, Maya Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. Optimization with Don-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals. Journal of Machine Learning Research, 20(172): 1-59, 2019, ISSN 1533-7928.
[12] Michele Donini, Luca Oneto, Shai Ben-David, John S. Shawe-Taylor, and Massimiliano Pontil. Empirical Risk Minimization Under Fairness Constraints. Advances in Neural Information Processing Systems, 12200,, 2018.
[13] Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yi Chen, Sijia Liu, and Kush R. Varshney. An cnnpommmtion-hhhoreee Perspective on the Relationship Between Fairness and Accuracy. arXiv:1910007870 [cs, math, stat], October 2019.
[14] Moritz Hardt, Eric Price, and Nathan Srebro. Equality of Opportunity in Supervised Learning. 111.021610002413 [es], October 2016.
[15] Jon Kleinberg and Manish Raghavan. Selection Problems in the Presence of Implicit Bias. January 2018
[16] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang, WILDS: Benchmark of in-the- Wild Distribution Shifts. arXiv:2012.07421 [cs], December 2020.
[17] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):4477 453, October 2019. ISSN 0036-8075,
doi: 10.1 1122///ienne.aax234.......
[18] Ya'acov Ritov, Yuekai Sun, and Ruofei Zhao. On conditional parity as a notion of non- discrimination in machine learning. arXiv: 1706.08519 [cs, stat), June 2017.
[19] Shibani Santurkar, Dimitris Tsipras, and Alcksander Madry. (BREEDS}: Benchmarks for subpopulation shift. In International Conference Learning Representations, 2021. URL httpp:://ppareviee..eett
[20] Shreya Shankar, Yoni Halpem, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. arXiv:1711.0855. [stat], November 2017.
[21] Tom Simonite. The Best Algorithms Still Struggle Recognize Black Faces. Wired, July 2019. ISSN 1029-1028.
[22] Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. Predictive Inequity in Object Detection. arXiv:/902. 1097 [cs, stat), February 2019.
[23] Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 547-558, 2020.
[24] Samuel Ycom and Michael Carl Tschantz. Discriminative but Not Discriminatory: A Compari- of Fairness Definitions under Different Worldviews. 1808 1808.0890 [cs, stat], September 2019.
[25] Mikhail Yurochkin, Amanda Bower, and Yuekai Sun. Training individually fair ML models with sensitive subspace robustness. In International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
you are including theoretical results...
(a) Did you state the full set assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No]
you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as a URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you'l 're
using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/]]
(b) Did you describe any potential participant risks, with links Institutional Review Board
(IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "ktLrXu2T6s", "reviewer": "Reviewer_3vrA", "summary": "The authors analyze the effect of enforcing algorithmic fairness (performance parity) in the presence of subpopulation shift. They argue that model performance can be summarized in terms of risk profiles, as risk minimization problems are equivalent to linear optimization wrt risk profiles. Basically, the authors consider ML problems that can be mapped to risk minimization. The paper also provides a way to analyze subpopulation shift by decomposing the training bias into (1) a portion which is orthogonal to the fair constraint, and (2) a robustness of $R^\\star$ to changes in the distribution of the data in the hypothetical target domain ($P^\\star$). Lastly, the authors show that fair risk minimization while training does not necessarily improve performance---a theoretical analysis is provided that shows when improvement and harm is done to model performance. Computational results are shown over synthetic data and recidivism data (COMPAS) to support results. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "ORIGINALITY: The main contribution of this paper is to offer insight and an interesting perspective into bias mitigation caused by population shift. The problem is well-motivated; this is not an uncommon phenomenon in practice. The work seems sufficiently original, although I am not intimate with the frontier of knowledge in this area.  There are already works that evaluate whether FRM reduces/impacts algorithmic bias. One key difference is that the authors are looking to evaluate these criteria in the target domain. The related work seems thorough in differentiating between the contributions of this work and similar works. \nQUALITY: The technical content of the main paper appears correct---I thank the authors for the experimental details in the main text. The optimization review in the Appendix B adds to the general readability/understanding of the techniques used in the proof. I did not look over the supplementary proofs in A but focused on B.8, the permissibility of continuous-valued attributes. \nCLARITY: The paper is structured clearly and is well-written! The notation is clear and intuition is provided for different concepts, e.g., enforcing RP by solving the constrained risk minimization problem using two groups...visual explanation is provided and clearly explained. The goal of the computational results is explained clearly, and adequate experimental details are provided in the main text. \nSIGNIFICANCE: I enjoyed this paper. It sheds light on a well-known phenomenon seen in practice---distribution (subpopulation) shift, and could be useful as a tool for analyzing/creating fairness aware algorithms. I believe this work adequately expands on previous work, e.g., ML problems applicable to risk minimization and considering posterior drift, while focusing on the effect on the target domain. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "jD3RpSMFIBd", "reviewer": "Reviewer_M7Vx", "summary": "This paper addresses the following problem: What will the performance of a ML model be on the target domain if one enforces fairness in the training space, while considering the fact that some groups may be underrepresented in the training domain (and hence, biased) and considering subpopulation shift? The paper brings theoretical results as well as experiments to provide insights into this phenomenon. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and well-motivated. The problem statement and the contributions of the paper are clear. Specially, I appreciate the {\\em insight} that this paper can bring to the community. One of the main points of the paper is characterizing and analyzing algorithmic fairness under subpopulation shifts, which is defined in the paper.\n\nThe theoretical results of the paper are backed by the experiments on the simulated data, as well as COMPAS and Adult data.\n \nNote: One should be careful with interpreting results of this paper as the paper assumes the assumption in equation (3.3). This is assumption 2 under Theorem 3.2 and 4.3. It means that this paper does not consider an entire distribution shift from the train domain to the target domain and restricts its attention to subpopulation shift. Hence, one should be cautious about making general statements based on this paper. Perhaps, a discussion about this can help the general reader more.\n\nMinor: Contribution number 3 in Introduction: Line 36, please complete the sentence. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See my main review.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "48"}, "extra_text": {}}, {"review_id": "WtIDXK97nu7", "reviewer": "Reviewer_VZHD", "summary": "The paper studies the impact of constrained fairness risk minimization on model performance when data are affected by subpopulation shifts. The authors focus on two notions of fairness that are defined with respect to the loss of the model, and that they call \"risk parity\" and \"conditional risk parity\" respectively. \nUnder a series of assumptions, the authors derive the conditions under which a model model trained via constrained risk minimization has overall higher/lower performance on the target data (vs. train data). They then examine the resulting bounds via simulations and experiments. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My comprehension of the results was very much hindered by the lack of clearly defined notation (see suggestions below). I also had a hard time understanding how these results differ from prior work and the meaning of the bounds in the results. Two important points:\n* It seems that assumption (3.3) is fairly limiting. For example, this assumption rules out the setting of label-bias as defined by [1] and thus . In addition, I looked at the paper in reference [16] (line 92), and I have the impression that (3.3) is an additional assumption imposed on the subpopulation shifts setting, rather than being “(slightly) less restrictive”. \n* That said, it is unclear to me why the bounds per se, in their current form, should be of interest, or at least I do not think that the result in theorem 3.2 was properly discussed. We should probably interested in understanding by how much performance will change, although I am not sure this is analytically possible. \n\nSome (hopefully) constructive suggestions are: (i) move related work up and clearly explain how this work fits into the existing literature, (ii) clearly define all notation, (iii) improve the explanations of the results, (iv) add an organization paragraph in the introduction. \n\nSome of the typos and other minor issues:\n* definition 3.1: is this definition based on prior work?\n* line 36 “during improves”\n* line 63: needs citation and \\mathcal{l} has never been introduced\n* line 65: the original study should be cited\n* ERM has never been defined\n* (3.2) I am not sure that the inner product on the RHS makes sense\n* line 77: erroneous reference (4.3)\n* it’s unclear what the pedix in the expectation means (compare pedix in (3.1) vs. Line 76)\n* (3.2) \\mathcal{F} has never been defined\n* line 75: the bound between 0 and 1 for \\tilde{P}_A makes sense only in case of discrete distributions\n* (3.3) the expectations E* and \\tilde{E} have not been defined\n* (4.4) V has never been properly defined\n* example 4.2: I believe that the definition of label bias adopted by the authors is one of many, so they may want to cite a paper the authors took this definition from. For example, [1] calls (4.2) “under-representation and labeling bias” (I see that this paper is actually cited in the related work).\n* line 185: “even if the Bayes decision rule in the target domain is algorithmically fair” is unclear\n* line 147: “.” Should be “,” \n\n[1] Blum, Avrim, and Kevin Stangl. \"Recovering from biased data: Can fairness constraints improve accuracy?.\" arXiv preprint arXiv:1912.01094 (2019).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should clearly motivate how their results advance and could contribute to our understanding of risk minimization with fairness constraints, e.g., perhaps by referring to the their case study of the COMPAS data. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "5gVqeS8dU12", "reviewer": "Reviewer_jgBW", "summary": "The work examines when does a fairness constrained model improve in overall accuracy on a target dataset which has a different distribution (of loss conditioned on some covariates) from the train dataset. It provides necessary and sufficient conditions under which this happens. Experiments on synthetic data validate the theoretical results. Real world experiments on COMPASS and Adult income datasets are promising.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Significance: This is my main concern. The significance of the theoretical results to evaluation and design of fair classifiers under shifts is not discussed. The condition provided in Theorem 4.3 (when to expect improvement) are hard to relate to or compute for any practical setting, thus limiting its applicability. Experiments on real world dataset also do not shed light on how the conditions can be computed. The problem of assessing future performance of the classifier under shifts is very important and the work provides a novel perspective based on geometry of risk profiles. Discussing the computable implications of the results will make them more useful for practitioners.\n\nThe work does not discuss the motivation for training fair models to correct subpopulation shift in training data. If we anticipate distribution shift, then we can use transfer learning (e.g. domain adapation, generalization) methods. It does not seem realistic to assume that modeler will not use any such methods and will rely on fairness constraints to potentially correct for the shift. \n\nOriginality: The work adds an original, geometrical perspective to studying fairness under distribution shifts. This is enabled by their focus on the space of risk profiles and the subpopulation shift condition. Related work compares with the most directly relevant papers but does not place this work in the larger fair learning under shifts literature. Some works https://cs.stanford.edu/~jure/pubs/contraction-kdd17.pdf, https://arxiv.org/abs/1806.02887, https://arxiv.org/abs/2007.06029.\n\nQuality: The key results are technically sound. The experiments do not adequately test the results.\nNot much discussion is provided on whether the conditions in Theorems 3.2, 4.3 can be tested. Both the target distribution P* and fair risk set F are unknown.\nExperiments on real data too are not checking whether these conditions hold. The experiments only check for an increase in accuracy in P*. The validation of the theory is incomplete without showing that the conditions that predict this increase also hold or can reasonably be expected to hold for real data.\n\nClarity: The writing is clear. The figures can be explained better. Please see suggestions at the end.\n\n\nOverall, I like the new geometrical perspective to reason about model performance under shifts. I would encourage authors to develop the ideas to make them more practical and to consider their implications on better evaluation or training procedures.\n\n\nMore detailed comments:\n\nReplacing risk with model output in the fairness definition of demographic parity and equalised odds is assuming that the loss function used in learning is the same as that for fairness consideration. The advantage of enforcing independence with model output is that it will ensure independence with any fairness loss function. For many reasons such as ease of optimization (use of surrogate loss) or application context, the two loss functions can be different. This critical assumption should be stated.\n\nThe target for comparison in the unbiased data is assumed to be the unconstrained Bayes optimal model. I disagree with the claim that assumption 3 is implicitly assumed in many previous works, lines 229-235. The cited works are not suggesting to just use the unconstrained Bayes classifier on representative data. The classifier for unbiased data can be trained with fairness constraints too if suitable for the application context.\n\nLabel bias described in Example 4.2 differs from the subpopulation shift assumption. It is usually defined as an invariant distribution of outcome given some or all covariates, whereas the label bias and the resulting risk profile is additionally conditioning on the outcome. Please define subpopulation shifts in the Problem setup to avoid any confusion.\n\n\nQuestions that I would like authors to address:\n\nIs the risk profile in 4.2 defined with respect to biased distribution P tilde?\n\nAre the conditions for checking improvement in the theorems testable? What data does it require? \n\nHow to interpret the threshold on P* in Theorem 3.2 which is used to check whether FRM improves?\n\nIn Theorem 4.3 assumptions, which risk set or profile is used, 3.1 or 4.2?\n\nPlease clarify how the results can be used to better evaluate and design fair classifiers.\n\n---\n\nAfter the response, my concerns on the definition of subpopulation shift, and interpretation of Theorem 3.2 and Assumption 3 are addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please discuss limitations of the work e.g. limited to two fairness metrics (risk parity and conditional risk parity), limited to subpopulation shift (and not label shift), limited ability to verify if the conditions in theorems are satisfied in real data.\n\n\nComments to improve presentation:\n\nPlease consider contrasting the insights from this work with the fair learning under shifts literature. How does this work help in training better models under shifts?\n\nWhat is the space plotted in Figure 2? Although clear from Figure 1, but useful to mention in caption.\n\nIn line 208, what is the fairness constraint the unbiased Bayes classifier will satisfy? Doesn’t it minimize the unconstrained risk? It is explicitly assumed later in Theorem 4.3 assumption 3.\n\nInstead of linear optimization problem e.g. in line 78, please be specific that constraints are linear. Linear optimization suggests that loss function is linear which is not the case here.\n\nIs it necessary to define risk profiles in terms of strict equality of risks across groups? Will the results extend to a more realistic case when there is small difference in risks across groups?\n\nLine 132: “trivial random variables” -> V is an empty set.", "review_text": "", "extra_scores": {"ethical_concerns": "The authors have addressed the ethical concern posed by fair learning without attending to differences between training and deployment settings.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the question of whether enforcing algorithmic fairness in the training domain will improve performance of the model in a target domain in which subpopulations (such as those defined by the sensitive attribute) occur in different proportions from the training domain.  The key assumption behind the results is that the risk profiles (expected loss conditional on group, or group and other discriminative attributes) are the same between the training and test domain.   The authors’ main result crisply characterizes conditions under which target domain performance improves.  Overall this is an interesting paper that studies an important problem, but it is not without its weaknesses.  Given the authors’ responses to concerns raised by reviewers, I believe the major weaknesses can be addressed through moderate revision and am thus recommending this paper for acceptance.  

In revising the submission, the authors should focus on the following dimensions.
1. Clarity in exposition.  Reviewers jgBW VZHD enumerated a lengthy list of typos and confusions that I myself also share.  (Additionally, the difference between distributions, $P^* - \tilde P$ isn’t actually defined where it is first used in line 53.) While the authors note in their response that several reviewers described the work as well-written and clear, I want to emphasize that reviewers who went into depth in their comments noted typos, undefined and ill-defined notation, and confusion about theorem statements.  These issues are minor, but abundant.    The manuscript will be much improved once they are weeded out in revision.
2. Displaying figure 2 sideways is confusing.  If it is desired to keep the figure in this orientation to conserve space, at least rotate the text.  
3. The work would benefit from further discussion of the settings in which the authors believe the results would likely apply.  That is, in what practical settings might the subpopulation shift assumption hold?  Would it hold in something like the motivating gender shades study example?   
4. The COMPAS example is currently highly underdeveloped.  I would either expand on the discussion to explain what this example is illustrating and its implications for training risk assessment tools, or omit it altogether.  
5. In the author response the authors have committed to making a number of revisions to address concerns raised by reviewers.  They are all worthwhile changes that will improve the clarity of the manuscript.  

Decision: Accept (Poster)