A Near-Optimal Recipe for Debiasing Trained Machine Learning Models

1 INTRODUCTION . Machine learning is increasingly applied to critical decisions which can have a lasting impact on individual lives , such as for credit lending ( Bruckner , 2018 ) , medical applications ( Deo , 2015 ) , and criminal justice ( Brennan et al. , 2009 ) . Consequently , it is imperative to understand and improve the degree of bias of such automated decision-making . Unfortunately , despite the fact that bias ( or “ fairness ” ) is a central concept in our society today , it is difficult to define it in precise terms . In fact , as people perceive ethical matters differently depending on a plethora of factors including geographical location or culture ( Awad et al. , 2018 ) , no universally-agreed upon definition for bias exists . Moreover , the definition of bias may depend on the application and might even be ignored in favor of accuracy when the stakes are high , such as in medical diagnosis ( Kleinberg et al. , 2017 ; Ingold and Soper , 2016 ) . As such , it is not surprising that several definitions of “ unbiased classification ” have been introduced . These include statistical parity ( Dwork et al. , 2012 ; Zafar et al. , 2017a ) , equality of opportunity ( Hardt et al. , 2016 ) , and equalized odds ( Hardt et al. , 2016 ; Kleinberg et al. , 2017 ) . Unfortunately , such definitions are not generally compatible ( Chouldechova , 2017 ) and some might even be in conflict with calibration ( Kleinberg et al. , 2017 ) . In addition , because fairness is a societal concept , it does not necessarily translate into a statistical criteria ( Chouldechova , 2017 ; Dixon et al. , 2018 ) . Statistical parity Let X be an instance space and let Y = { 0 , 1 } be the target set in a standard binary classification problem . In the fair classification setting , we may further assume the existence of a ( possibly randomized ) sensitive attribute s : X → { 0 , 1 , . . . , K } , where s ( x ) = k if and only if x ∈ Xk for some total partition X = ∪kXk . For example , X might correspond to the set of job applicants while s indicates their gender . Here , the sensitive attribute can be randomized if , for instance , the gender of an applicant is not a deterministic function of the full instance x ∈ X ( e.g . number of publications , years of experience , ... etc ) . Then , a commonly used criterion for fairness is to require similar mean outcomes across the sensitive attribute . This property is well-captured through the notion of statistical parity ( a.k.a . demographic parity ) ( Corbett-Davies et al. , 2017 ; Dwork et al. , 2012 ; Zafar et al. , 2017a ; Mehrabi et al. , 2019 ) : Definition 1 ( Statistical Parity ) . Let X be an instance space and X = ∪kXk be a total partition of X . A classifier f : X → { 0 , 1 } satisfies statistical parity across all groups X1 , . . . , XK if : max k∈ { 1,2 , ... , K } Ex [ f ( x ) | x ∈ Xk ] − min k∈ { 1,2 , ... , K } Ex [ f ( x ) | x ∈ Xk ] ≤ To motivate and further clarify the definition , we showcase the empirical results on the Adult benchmark dataset ( Blake and Merz , 1998 ) in Figure 1 . When tasked with predicting whether the income of individuals is above $ 50K per year , all considered classifiers exhibit gender-related bias . One way of removing such bias is to enforce statistical parity across genders . Crucially , however , without taking ethnicity into account , different demographic groups may experience different outcomes . In fact , gender bias can actually increase in some minority groups after enforcing statistical parity . This can be fixed by redefining the sensitive attribute to be the cross product of both gender and ethnicity ( green bars ) . Our main contribution is to present a near-optimal recipe for debiasing models , including deep neural networks , according to Definition 1 . Specifically , we formulate the task of debiasing learned models as a regularized optimization problem that is solved efficiently using the projected SGD method . We show how the algorithm produces thresholding rules with randomization near the thresholds , where the width of randomization is controlled by the regularization parameter . We also show that randomization near the threshold is necessary for Bayes risk consistency . While we focus on binary sensitive attributes in our experiments in Section 5 , our algorithm and its theoretical guarantees continue to hold for non-binary sensitive attributes as well . Statement of Contribution . 1 . We derive a near-optimal post-processing algorithm for debiasing learned models ( Section 3 ) . 2 . We prove theoretical guarantees for the proposed algorithm , including a proof of correctness and an explicit bound on the Bayes excess risk ( Section 4 ) . 3 . We empirically validate the proposed algorithm on benchmark datasets across both classical algorithms and modern DNN architectures . Our experiments demonstrate that the proposed algorithm significantly outperforms previous post-processing methods ( Section 5 ) . In Appendix E , we also show how the proposed algorithm can be modified to handle other criteria of bias as well . 2 RELATED WORK . Algorithms for fair machine learning can be broadly classified into three groups : ( 1 ) pre-processing methods , ( 2 ) in-processing methods , and ( 3 ) post-processing methods ( Zafar et al. , 2019 ) . Preprocessing algorithms transform the data into a different representation such that any classifier trained on it will not exhibit bias . This includes methods for learning a fair representation ( Zemel et al. , 2013 ; Lum and Johndrow , 2016 ; Bolukbasi et al. , 2016 ; Calmon et al. , 2017 ; Madras et al. , 2018 ; Kamiran and Calders , 2012 ) , label manipulation ( Kamiran and Calders , 2009 ) , data augmentation ( Dixon et al. , 2018 ) , or disentanglement ( Locatello et al. , 2019 ) . On the other hand , in-processing methods constrain the behavior of learning algorithms in order to control bias . This includes methods based on adversarial learning ( Zhang et al. , 2018 ) and constraint-based classification , such as by incorporating constrains on the decision margin ( Zafar et al. , 2019 ) or features ( Grgić-Hlača et al. , 2018 ) . Agarwal et al . ( 2018 ) showed that the task of learning an unbiased classifier could be reduced to a sequence of cost-sensitive classification problems , which could be applied to any black-box classifier . One caveat of the latter approach is that it requires solving a linear program ( LP ) and retraining classifiers , such as neural networks , many times before convergence . The algorithm we propose in this paper is a post-processing method , which can be justified theoretically ( Corbett-Davies et al. , 2017 ; Hardt et al. , 2016 ; Menon and Williamson , 2018 ; Celis et al. , 2019 ) . Fish et al . ( 2016 ) and Woodworth et al . ( 2017 ) fall under this category . However , the former only provides generalization guarantees without consistency results while the latter proposes a twostage approach that requires changes to the original training algorithm . Kamiran et al . ( 2012 ) also proposes a post-processing algorithm , called Reject Option Classifier ( ROC ) , without providing any theoretical guarantees . In contrast , our algorithm is Bayes consistent and does not alter the original classification method . In Celis et al . ( 2019 ) and Menon and Williamson ( 2018 ) , instance-dependent thresholding rules are also learned . However , our algorithm also learns to randomize around the threshold ( Figure 2 ( a ) ) and this randomization is key to our algorithm both theoretically as well as experimentally ( Appendix C and Section 5 ) . Hardt et al . ( 2016 ) learns a randomized post-processing rule but our proposed algorithm outperforms it in all of our experiments ( Section 5 ) . Woodworth et al . ( 2017 ) showed that the post-processing approach can , sometimes , be highly suboptimal . Nevertheless , the latter result does not contradict the statement that our post-processing rule is near-optimal because we assume that the original classifier outputs a monotone transformation of some approximation to the posterior probability p ( y = 1 | x ) ( e.g . margin or softmax output ) whereas Woodworth et al . ( 2017 ) assumed in their construction that the post-processing rule had access to the binary predictions only . We argue that the proposed algorithm has distinct advantages , particularly for deep neural networks ( DNNs ) . First , stochastic convex optimization methods are well-understood and can scale well to massive amounts of data ( Bottou , 2010 ) , which is often the case in deep learning today . Second , the guarantees provided by our algorithm hold w.r.t . the binary predictions instead of using a proxy , such as the margin as in some previous works ( Zafar et al. , 2017b ; 2019 ) . Third , unlike previous reduction methods that would require retraining a deep neural network several times until convergence ( Agarwal et al. , 2018 ) , which can be prohibitively expensive , our algorithm operates on learned models that are trained once and does not require retraining . Besides developing algorithms for fair classification , several recent works focused on other related aspects , such as proposing new definitions for fairness ; e.g . demographic parity ( Dwork et al. , 2012 ; Mehrabi et al. , 2019 ) , equalized odds ( Hardt et al. , 2016 ) , equality of opportunity/disparate mistreatment ( Zafar et al. , 2017a ; Hardt et al. , 2016 ) , and individual fairness ( Dwork et al. , 2012 ) . Recent works have also established several impossibility results related to fair classification , such as Kleinberg et al . ( 2017 ) ; Chouldechova ( 2017 ) . In our case , we derive a new impossibility result that holds for any deterministic binary classifier and relate it to the task of controlling the covariance between the classifier ’ s predictions and the sensitive attribute ( Appendix E ) . 3 NEAR-OPTIMAL ALGORITHM FOR STATISTICAL PARITY . Notation We reserve boldface letters for random variables ( e.g . x ) , small letters for instances ( e.g . x ) , capital letters for sets ( e.g . X ) , and calligraphic typeface for universal sets ( e.g . the instance space X ) . Given a set S , 1S ( x ) ∈ { 0 , 1 } is the characteristic function indicating whether x ∈ S. We denote by [ n ] the set of integers { 1 , . . . , n } and [ x ] + = max { 0 , x } . Algorithm Given a classifier f : X → [ −1 , +1 ] our goal is to post-process the predictions made by f 1 in order to control the bias with respect to a sensitive attribute s : X → [ K ] as in Definition 1 . To this end , instead of learning a deterministic classifier , we consider randomized prediction rules of the form h̃ : X × { 1 , 2 , . . . , K } × [ −1 , 1 ] → [ 0 , 1 ] , where h̃ ( x ) represents the probability of predicting the positive class given ( i ) instance x ∈ X , ( ii ) sensitive attribute s ( x ) , and ( iii ) classifier ’ s output f ( x ) . As discussed in Appendix B , for post-processing rule h̃ ( x ) , and for each group Xk ⊆ X , the fairness constraint in Definition 1 can be written as |Ex [ h̃ ( x ) | x ∈ Xk ] − ρ| ≤ , where ρ ∈ [ 0 , 1 ] is a hyper-parameter tuned via a validation dataset . On the other hand , minimizing the probability of altering the predictions of the original classifier can be achieved by maximizing the inner product Ex [ h̃ ( x ) ·f ( x ) ] . Instead of optimizing this quantity directly , which would lead to a pure thresholding rule , we minimize the regularized objective : ( γ/2 ) Ex [ h̃ ( x ) 2 ] −Ex [ h̃ ( x ) · f ( x ) ] for some regularization parameter γ > 0 . This regularization leads to randomization around the threshold , which we show to be critical , both theoretically ( Section 4 and Appendix C ) and experimentally ( Section 5 ) . Using Lagrange duality we show that the solution reduces to the update rules in Equation 2 with optimization variables { λk , µk } k∈ [ K ] and the corresponding predictor which outputs +1 for group Xk with probability h̃γ ( x ) is given by h̃γ ( x ) = 0 , f ( x ) ≤ λk − µk ( f ( x ) − λk + µk ) /γ , λk − µk ≤ f ( x ) ≤ λk − µk + γ 1 , f ( x ) ≥ λk − µk + γ ( 1 ) where ξγ is given by Eq . ( 3 ) . Update rules To learn these parameters , one can apply the following update rules ( Appendix B ) : λs ( x ) ← max { 0 , λs ( x ) − η ( 2 + ρ+ ∂ ∂λs ( x ) ξγ ( f ( x ) − ( λs ( x ) − µs ( x ) ) ) ) } µs ( x ) ← max { 0 , µs ( x ) − η ( 2 − ρ+ ∂ ∂µs ( x ) ξγ ( f ( x ) − ( λs ( x ) − µs ( x ) ) ) ) } , ( 2 ) where , again , ρ ∈ [ 0 , 1 ] is a hyperparameter tuned via a validation dataset , s : X → [ K ] is the sensitive attribute , and γ > 0 is a regularization parameter that controls the level of randomization . In addition , the function ξγ : R→ R+ is given by : ξγ ( w ) = w2 2γ · I { 0 ≤ w ≤ γ } + ( w − γ 2 ) · I { w > γ } ( 3 ) Note that ξγ is convex and its derivative ξ′γ is ( 1/γ ) -Lipschitz continuous ; it can be interpreted as differentiable approximation to the ReLU unit ( Nair and Hinton , 2010 ) . A full pseudocode of the proposed algorithm is presented in Appendix A .

Summary: In this paper, the authors propose a post-processing method for removing bias from a trained model. The bias is defined as conditional statistical parity — for a given partitioning of the data, the predicted label should be conditionally uncorrelated with the sensitive (bias inducing) attribute for each partition. The authors relax this strong requirement to an epsilon-constraint on the conditional covariance for each partition. As an example, race (sensitive attribute) should be conditionally uncorrelated to whether an individual will default on their loan (predicted target) for each city (data partition). The authors propose a constrained optimization problem that takes the input data, sensitive attribute, partitioning and a trained model to yield a probabilistic decision rule. Subsequently, they propose an iterative solution to the problem, proving some theoretical properties as well as showing how the method compares to different baselines.