Does enforcing fairness mitigate biases caused by subpopulation shift?

1 Introduction . There are many instances of distribution shifts causing performance disparities in machine learning ( ML ) models . For example , Buolamwini and Gebru [ 7 ] report commercial gender classification models are more likely to misclassify dark-skinned people ( than light-skinned people ) . This is ( in part ) due to the abundance of light-skinned examples in training data . Similarly , pedestrian detection models sometimes have trouble recognizing dark-skinned pedestrians [ 22 ] . Another prominent example is the poor performance of image processing models on images from developing countries due to the scarcity of images from such countries in publically available image datasets [ 20 ] . Unfortunately , many algorithmic fairness practices were not developed with distribution shifts in mind . For example , the common algorithmic fairness practice of enforcing performance parity on certain demographic groups [ 1 , 11 ] implicitly assumes performance parity on training data generalize to the target domain , but distribution shifts between the training data and target domain renders this assumption invalid . In this paper , we consider subpopulation shifts as a source of algorithmic biases and study whether the common algorithmic fairness practice of enforcing performance parity on certain demographic groups mitigate the resulting ( algorithmic ) biases in the target domain . Such algorithmic fairness practices are common enough that there are methods [ 1 , 2 ] and software ( e.g . TensorFlow Constrained Optimization [ 10 ] ) devoted to operationalizing them . There are other sources of algorithmic biases * Equal Contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( e.g . posterior drift [ 17 ] ) , but we focus on algorithmic biases caused by subpopulation shifts in this paper . Our main contributions are : 1 . We propose risk profiles as a way of summarizing the performance of ML models on subpopulations . As we shall see , this summary is particularly suitable for studying the performance of risk minimization methods . 2 . We show that enforcing performance parity during training may not mitigate performance disparities in the target domain . In fact , it may even harm overall performance . 3 . We decompose subpopulation shifts into two parts , a recoverable part orthogonal to the fair constraint and a non-recoverable part , and derive necessary and sufficient conditions on subpopulation shifts under which enforcing performance parity improves performance in the target domain ( see Section 4.4 ) . One of the main takeaways of our study is a purely statistical way of evaluating the notion of algorithmic fairness for subpopulation shift : an effective algorithmic fairness practice should improve overall model performance in the target domain . Our theoretical results characterize when this occurs for risk-based notions of algorithmic fairness . 2 Problem setup . We consider a standard classification setup . Let X ⊂ Rd be the feature space , Y be the set of possible labels , and A be the set of possible values of the sensitive attribute . In this setup , training and test examples are tuples of the form ( X , A , Y ) ∈ X × A × Y . If the ML task is predicting whether a borrower will default on a loan , then each training/test example corresponds to a loan . The features in X may include the borrower ’ s credit history , income level , and outstanding debts ; the label Y ∈ { 0 , 1 } encodes whether the borrower defaulted on the loan ; the sensitive attribute may be the borrower ’ s gender or race . Let P ∗ and P̃ be probability distributions on X ×A× Y . We consider P̃ as the distribution of the training data and P ∗ as the distribution of data in a hypothetical target domain . For example , P ∗ may be the distribution of data in the real world , and P̃ is a biased sample in which certain demographic groups are underrepresented . The difference P ∗ − P̃ is the distribution shift . In practice , distribution shifts often arise due to sampling biases during the ( training ) data collection process , so we call P ∗ and P̃ unbiased and biased respectively and refer to P ∗ − P̃ as the bias ( in the training data ) . Henceforth E∗ ( resp . Ẽ ) will denote expectation under P ∗ ( resp . P̃ ) . The set of all hypotheses under consideration is denoted byH and ` : Y × Y 7→ R+ denotes the loss function under consideration . In Section 3 and 4 we assume the set of sensitive attribute A is discrete . The case with continuous A is relegated to the supplementary document ( Appendix 2 ) . 3 Benefits and drawbacks of enforcing risk parity . To keep things simple , we start by considering the effects of enforcing risk parity ( RP ) . This notion is closely related to the notion of demographic parity ( DP ) . Recall DP requires the output of the ML model h ( X ) to be independent of the sensitive attribute A : h ( X ) ⊥ A. RP imposes a similar condition on the risk of the ML model . Definition 3.1 ( risk parity ) . A model h satisfies risk parity with respect to the distribution P on X × Y ×A if EP [ ` ( h ( X ) , Y ) | A = a ] = EP [ ` ( h ( X ) , Y ) | A = a′ ] for all a , a′ ∈ A and all h ∈ H. RP is widely used in practice to measure algorithmic bias in ML models . For example , the US National Institute of Standards and Technology ( NIST ) tested facial recognition systems and found that the systems misidentify blacks at rates 5 to 10 times higher than whites [ 21 ] . By comparing the error rates of the system on blacks and whites , NIST is implicitly adopting RP as its definition of algorithmic fairness . It is not hard to see that RP is equivalent to linear constraints on the risk profile ( RP̃ ( h ) ) of an ML model with respect to P̃ : RP ( h ) , { EP [ ` ( h ( X ) , Y ) | A = a ] } a∈A . ( 3.1 ) For notational simplicity defineR as the set of all risk profiles with respect to the training distribution P̃ , i.e . R , { R ( h ) | h ∈ H } . The risk profile of a model summarizes its performance on subpopulations . In terms of risk profiles , RP with respect to distribution P requires RP ( h ) = c1 for some constant c ∈ R. This is a linear constraint . The set of all risk profiles that satisfy the RP constraint with respect to the training distribution P̃ constitutes the following subspace : FRP ≡ FRP ( P̃ ) , { RP̃ ( h ) ∈ R|A| ∣∣ RP̃ ( h ) = c1,1 ∈ R|A| , c ∈ R } . Therefore , we enforce RP by solving ( the empirical version of ) a constraint risk minimization problem { minh∈H Ẽ [ ` ( h ( X ) , Y ) ] subject to RP̃ ( h ) ∈ FRP } ≡ { minR∈R 〈P̃A , RP̃ ( h ) 〉 subject to RP̃ ( h ) ∈ FRP } , ( 3.2 ) where P̃A ∈ [ 0 , 1 ] |A| is the marginal distribution of A and consequently 〈P̃A , R〉 = Ẽ [ ` ( h ( X ) , Y ) ] . Note that , in ( 3.2 ) , the inner product 〈· , ·〉 is euclidean inner product on R|A| . We define the minimizer of ( 3.2 ) as h̃F and its corresponding risk profile as R̃F = RP̃ ( h̃F ) . See Figure 1 for a graphical depiction of ( 3.2 ) when there are two groups ( A = { 0 , 1 } ) . Here we see the main benefit of summarizing model performance with risk profiles : risk minimization problems are equivalent to linear optimization problems in terms of risk profiles ( objective and constrain functions are linear in terms of risk profile ) . This allows us to simplify our study of the effects of enforcing risk parity by reasoning about the risk profiles of the resulting models . Hereafter , we refer to this approach as fair risk minimization ( FRM ) . It is not new ; similar constrained optimization problems have appeared in the algorithmic fairness literature ( e.g . see [ 1 , 12 , 10 ] ) . Our goal is evaluating whether this approach mitigates algorithmic biases and improves model performance in the target domain . There are efficient algorithms for solving ( 4.3 ) . One popular algorithm is a reductions approach by Agarwal et al . [ 1 ] , which solves a sequence of weighted classification problems with appropriately chosen weights to satisfy the desired algorithmic fairness constraints . This algorithm outputs randomized classifiers , which justifies the subsequent convexity assumption on the set of risk profiles . In order to relate model performance in the training and target domains , some restrictions on the distribution shift/bias is necessary , as it is impossible to transfer performance parity during training to the target domain if they are highly disparate . At a high-level , we assume the distribution shift is a subpopulation shift [ 16 ] . Formally , we assume that the risk profiles of the models with respect to P ∗ and the profiles with respect to P̃ are identical : E∗ [ ` ( h ( X ) , Y ) | A = a ] = Ẽ [ ` ( h ( X ) , Y ) | A = a ] for all a ∈ A , h ∈ H. ( 3.3 ) i.e . RP̃ ( h ) = RP∗ ( h ) for all h ∈ H. We note that this assumption is ( slightly ) less restrictive than the usual subpopulation shift assumption because it only requires the expected value of the loss ( instead of the full conditional distribution of ( X , Y ) givenA to be identical in the training and target domains . Furthermore , this assumption is implicit in enforcing risk-based notions of algorithmic fairness . If the risk profiles are not identical in the training and target domains , then enforcing risk-based notions of algorithmic fairness during training is pointless because performance parity during training may not generalize to the target domain . We are now ready to state our characterization of the benefits and drawbacks of enforcing RP . To keep things simple , we assume there are only two groups : a majority group and a minority group . As we shall see , depending on the marginal distribution of the subpopulations in the target domain P ∗A , enforcing RP can harm or improve overall performance in the target domain . Theorem 3.2 . Without loss of generality , let first entry of P̃A be the fraction of samples from the majority group in the training data i.e . P̃ ( A = 1 ) . Assume 1. there are only two groups and the set of risk profilesR ⊆ R2 ; 2. subpopulation shift : the risk profiles with respect to P̃ and P ∗ are identical ; 3 . ( R̃1 , R̃0 ) = R̃ , arg minR∈R〈P̃A , R〉 is the risk profile of the risk minimizer ; 4 . ( ( R̃F ) 1 , ( R̃F ) 0 ) = R̃F , arg minR∈R∩FRP〈P̃A , R〉 is the risk profile of the fair risk minimizer . Then we have : 〈P ∗A , R̃〉 { ≤ 〈P ∗A , R̃F 〉 if P ∗ ( A = 1 ) ≥ R̃0− ( R̃F ) 0R̃0−R̃1 ≥ 〈P ∗A , R̃F 〉 otherwise . Therefore , enforcing RP harms overall performance in the target domain in the first case , while improves in the second . In hindsight , this result is intuitive . If P ∗A is close to P̃A ( e.g . the minority group is underrepresented in the training data but not by much ) , then enforcing RP may actually harm overall performance in the target domain . This is mainly due to the trade-off between accuracy and fairness in the IID setting ( no distribution shift ) . If there is little difference between the training and target domains , then we expect the trade-off between accuracy and fairness to manifest ( albeit to a lesser degree than in IID settings ) .

Summary: The authors analyze the effect of enforcing algorithmic fairness (performance parity) in the presence of subpopulation shift. They argue that model performance can be summarized in terms of risk profiles, as risk minimization problems are equivalent to linear optimization wrt risk profiles. Basically, the authors consider ML problems that can be mapped to risk minimization. The paper also provides a way to analyze subpopulation shift by decomposing the training bias into (1) a portion which is orthogonal to the fair constraint, and (2) a robustness of $R^\star$ to changes in the distribution of the data in the hypothetical target domain ($P^\star$). Lastly, the authors show that fair risk minimization while training does not necessarily improve performance---a theoretical analysis is provided that shows when improvement and harm is done to model performance. Computational results are shown over synthetic data and recidivism data (COMPAS) to support results.