Attainability and Optimality: The Equalized-Odds Fairness Revisited

1 INTRODUCTION . As machine learning models become widespread in automated decision making systems , apart from the efficiency and accuracy of the prediction , their potential social consequence also gains increasing attention . To date , there is ample evidence that machine learning models have resulted in discrimination against certain groups of individuals under many circumstances , for instance , the discrimination in ad delivery when searching for names that can be predictive of the race of individual ( Sweeney , 2013 ) ; the gender discrimination in job-related ads push ( Datta et al. , 2015 ) ; stereotypes associated with gender in word embeddings ( Bolukbasi et al. , 2016 ) ; the bias against certain ethnicities in the assessment of recidivism risk ( Angwin et al. , 2016 ) . The call for accountability and fairness in machine learning has motivated various ( statistical ) notions of fairness . The Demographic Parity criterion ( Calders et al. , 2009 ) requires the independence between prediction ( e.g. , of a classifier ) and the protected feature ( sensitive attributes of an individual , e.g. , gender , race ) . Equalized Odds ( Hardt et al. , 2016 ) , also known as Error-rate Balance ( Chouldechova , 2017 ) , requires the output of a model be conditionally independent of protected feature ( s ) given the ground truth of the target . Predictive Rate Parity ( Zafar et al. , 2017a ) , on the other hand , requires the actually proportion of positives ( negatives ) in the original data for positive ( negative ) predictions should match across groups ( well-calibrated ) . On the theoretical side , results have been reported regarding relationships among fairness notions . It has been independently shown that if base rates of true positives differ among groups , then Equalized Odds and Predictive Rate Parity can not be achieved simultaneously for non-perfect predictors ( Kleinberg et al. , 2016 ; Chouldechova , 2017 ) . Any two out of three among Demographic Parity , Equalized Odds , and Predictive Rate Parity are incompatible with each other ( Barocas et al. , 2017 ) . At the interface of privacy and fairness , the impossibility of achieving both Differential Privacy ( Dwork et al. , 2006 ) and Equal Opportunity ( Hardt et al. , 2016 ) while maintaining non-trivial accuracy is also established ( Cummings et al. , 2019 ) . In practice , one can broadly categorize computational procedures to derive a fair predictor into three types : pre-processing approaches ( Calders et al. , 2009 ; Dwork et al. , 2012 ; Zemel et al. , 2013 ; Zhang et al. , 2018 ; Madras et al. , 2018 ; Creager et al. , 2019 ; Zhao et al. , 2020 ) , in-processing approaches ( Kamishima et al. , 2011 ; Pérez-Suay et al. , 2017 ; Zafar et al. , 2017a ; b ; Donini et al. , 2018 ; Song et al. , 2019 ; Mary et al. , 2019 ; Baharlouei et al. , 2020 ) , and post-processing approaches ( Hardt et al. , 2016 ; Fish et al. , 2016 ; Dwork et al. , 2018 ) . In accord with the fairness notion of interest , a pre-processing approach first maps the training data to a transformed space to remove discriminatory information between protected feature and target , and then pass on the data to make prediction . In direct contrast , a post-processing approach treats the off-the-shelf predictor ( s ) as uninterpretable black-box ( es ) , and imposes fairness by outputting a function of the original prediction . For inprocessing approaches , various kinds of regularization terms are proposed so that one can optimize the utility function while suppressing the discrimination at the same time . Approaches based on estimating/bounding causal effect between the protected feature and final target have also been proposed ( Kusner et al. , 2017 ; Russell et al. , 2017 ; Zhang et al. , 2017 ; Nabi & Shpitser , 2018 ; Zhang & Bareinboim , 2018 ; Chiappa , 2019 ; Wu et al. , 2019 ) . Focusing on the Equalized-Odds criterion , although various approaches have been proposed to impose the fairness requirement , whether or not it is always attainable is not well addressed . The attainability of Equalized Odds , namely , the existence of the predictor that can score zero violation of fairness in the large sample limit , is an asymptotic property of the fairness criterion . This characterizes a completely different kind of violation of fairness compared to the empirical error bound of discrimination in finite-sample cases . If utilizing a “ fair ” predictor which is actually biased , the discrimination would become a snake in the grass , making it hard to detect and eliminate . Actually , as we illustrate in this paper , Equalized Odds is not always attainable for regression and even classification tasks , if we use deterministic prediction functions . This calls for alternative definitions in the same spirit as Equalized Odds that can always be achieved under various circumstances . Our contributions are mainly : • For regression and classification tasks with deterministic prediction functions , we show that Equalized Odds is not always attainable if certain ( rather restrictive ) conditions on the joint distribution of the features and the target variable are not met . • Under mild assumptions , for binary classification we show that if randomized prediction is taken into consideration , one can always derive a non-trivial Equalized Odds classifier . • Considering the optimality of performance under fairness constraint ( s ) , when exploiting all available features , we show that the predictor derived via an in-processing approach would always outperform the one derived via a post-processing approach ( unconstrained optimization followed by a post-processing step ) . 2 PRELIMINARIES . In this section , we first illustrate the difference between prediction fairness and procedure fairness , and then , we present the formal definition of Equalized Odds ( Hardt et al. , 2016 ) . 2.1 HIERARCHY OF FAIRNESS . Before presenting the formulation of fairness , it is important to see the distinction between different levels of fairness when discussing fair predictors . When evaluating the performance of the proposed fair predictor , it is a common practice to compare the loss ( with respect to the utility function of choice , e.g. , accuracy for binary classification ) computed on target variable and the predicted value . There is an implicit assumption lying beneath this practice : the generating process of the data , which is just describing a real-world procedure , is not biased in any sense ( Danks & London , 2017 ) . Only when we treat the target variable ( recorded in the dataset ) as unbiased can we justify the practice of loss evaluation and the conditioning on target variable when imposing fairness ( as we shall see in the definition of Equalized Odds in Equation 1 ) . One may consider a music school admission example . The music school committee would decide if they admit a student to the violin performance program based on the applicant ’ s personal information , educational background , instrumental performance , and so on . When evaluating whether or not the admission is “ fair ” , there are actually two levels of fairness . First , based on the information at hand , did the committee evaluate the qualification of applicants without bias ( How committee evaluate the applicants ) ? And second , is committee ’ s procedure of evaluating applicants ’ qualification reasonable ( How other people view the evaluation procedure used by the committee ) ? In this paper , we consider prediction fairness , namely , assuming the data recorded is unbiased , the prediction ( made with respect to current reality ) itself should not include any biased utilization of information . The fairness with respect to the data generating procedure as well as the potential future influence of the prediction are beyond the scope of this paper . 2.2 EQUALIZED-ODDS FAIRNESS . Hardt et al . ( 2016 ) proposed Equalized Odds which requires conditional independence between prediction and protected feature ( s ) given ground truth of the target . Let us denote the protected feature byA , with domain of valueA , additional ( observable ) feature ( s ) byX , with domain of value X , target variable by Y , with domain Y , ( not necessarily fair ) predictors by Ŷ , and fair predictors by Ỹ . Equalized-Odds fairness requires Ỹ ⊥ A | Y . ( 1 ) For classification tasks , one can conveniently use the probability distribution form : ∀a ∈ A , t , y ∈ Y : P ( Ỹ = t | A = a , Y = y ) = P ( Ỹ = t | Y = y ) , ( 2 ) or more concisely , PỸ |AY ( t | a , y ) = PỸ |Y ( t | y ) . ( 3 ) For better readability , we also use the formulation in Equation 3 in cases without ambiguity . In the context of binary classification ( Y = { 0 , 1 } ) , Equalized Odds requires that the True Positive Rate ( TPR ) and False Positive Rate ( FPR ) of each certain group match population positive rates . Throughout the paper , without loss of generality we assume there is only one protected feature for the purpose of simplifying notation . However , considering the fact that the protected feature can be discrete ( e.g. , race , gender ) or continuous ( e.g. , the ratio of ethnic group in the population for certain district of a city ) , we do not assume discreteness of the protected feature . Due to the space limit , we will focus on the illustration and implication of our results and defer all the proofs to the appendix . 3 FAIRNESS IN REGRESSION MAY NOT BE ATTAINED . In this section we consider the attainability of Equalized Odds for regression tasks , namely , whether or not it is possible to find a predictor that is conditionally independent from the protected feature given true value of the target . For linearly Gaussian cases , one can attain Equalized Odds by constraining zero partial correlation between the prediction and the protected feature given target variable ( Woodworth et al. , 2017 ) . Various regularization terms have also been proposed to suppress discrimination when predicting a continuous target ( Berk et al. , 2017 ; Mary et al. , 2019 ) . However , whether or not one can always achieve 0-discrimination for regression , even if with an unlimited amount of data , is not clear yet . If “ fair ” predictors are deployed without carefully checking the attainability of fairness , the discrimination would become a hidden hazard , making it hard to detect and eliminate . Actually as we will show in this section , even in the simple setup of linearly correlated continuous data , Equalized Odds is not always attainable . 3.1 UNATTAINABILITY OF EQUALIZED ODDS IN LINEAR NON-GAUSSIAN REGRESSION . As stated in Section 2.1 , in this paper we consider prediction fairness , and therefore any possible bias introduced by the data generating procedure itself is beyond the scope of the discussion . Consider the situation where the data is generated as following ( H is not measured in the dataset ) : X = qA+ EX , H = bA+ EH , Y = cX + dH + EY , ( 4 ) where ( A , EX , EH , EY ) are mutually independent . In fact , if at most one of EX and E : = EY + dEH is Gaussian , then any linear combination of A and X with non-zero coefficients will not be conditionally independent fromA given Y , meaning that it is not possible to achieve Equalized-Odds fairness . Let Z be a linear combination of A and X , i.e. , Z = αA + βX = ( α + qβ ) A + βEX , with linear coefficients α and β , where β 6= 0 . In Theorem 3.1 , we present the general result in linear non-Gaussian cases , where one can not achieve the conditional independence between Z and A given Y . Theorem 3.1 . ( Unattainability of Equalized Odds in the Linear Non-Gaussian Case ) Assume that feature X has a causal influence on Y , i.e. , c 6= 0 in Equation 4 , and that the protected feature A and Y are not independent , i.e. , qc + bd 6= 0 . Assume pEX and pE are positive on R. Let f1 : = log pA , f2 : = log pEX , and f3 : = log pE . Further assume that f2 and f3 are third-order differentiable . Then if at most one of EX and E is Gaussian , Z is always conditionally dependent on A given Y . From Theorem 3.1 , we see that in linear non-Gaussian cases , any non-zero linear combination of the feature ( which is a deterministic function of the input ) will not satisfy Equalized Odds . One may wonder whether Equalized Odds can be achieved by nonlinear regression , instead of a linear model . Although a proof with general nonlinear models is rather complicated , our simulation results in Section 5.1 strongly suggest that the unattainability of Equalized Odds persists in nonlinear regression cases . In light of the unattainability of Equalized Odds for prediction with deterministic functions of A and X , it is desirable to develop general , nonlinear prediction algorithms to produce a probabilistic prediction ( i.e. , with a certain type of randomness in the prediction ) . One possible way follows the framework of Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) : we use random standard Gaussian noise , in addition to A and X , as input , such that the output will have a specific type of randomness . The parameters involved are learned by minimizing prediction error and enforcing Equalized Odds on the “ randomized ” output at the same time . Given that this approach is not essential to illustrate the claims made in this paper and that theoretical properties of such nonlinear regression algorithms with randomized output are not straightforward to establish , this is left as future work .

Summary: This paper studies under what conditions a classifier can satisfy the condition of equalized odds. The authors first prove an impossibility result which shows that (under linear non-gaussian case) any deterministic classifier	cannot achieve equalized odds across the protected groups. This leads to the question of randomized classifier. We can always satisfy equalized odds with randomized classifier  with a trivial classifier. However, the authors show two interesting results — (1) using the post-processing framework developed in Hard et. al. (2016) it is possible to obtain a non-trivial randomized classifier that satisfies EO, and (2) in-processing based classifiers are better than the post-processing based fair classifiers if we compare them in terms of accuracy.