A Near-Optimal Algorithm for Debiasing Trained Machine Learning Models

1 Introduction . Background . Machine learning is increasingly applied to critical decisions which can have a lasting impact on individual lives , such as for credit lending [ Bruckner , 2018 ] , medical applications [ Deo , 2015 ] , and criminal justice [ Brennan et al. , 2009 ] . Consequently , it is imperative to understand and improve the degree of bias of such automated decision-making . Unfortunately , despite the fact that bias ( or “ fairness '' ) is a central concept in our society today , it is difficult to define it in precise terms . In fact , as people perceive ethical matters differently depending on a plethora of factors including geographical location or culture [ Awad et al. , 2018 ] , no universallyagreed upon definition for bias exists . Moreover , bias may depend on the application and might even be ignored in favor of accuracy when the stakes are high , such as in medical diagnosis [ Kleinberg et al. , 2016 ] . As such , it is not surprising that several measures of bias have been introduced , such as statistical parity [ Dwork et al. , 2012 , Zafar et al. , 2017a ] , equality of opportunity [ Hardt et al. , 2016 ] , and equalized odds [ Hardt et al. , 2016 , Kleinberg et al. , 2016 ] , and these are not generally mutually compatible [ Chouldechova , 2017 , Kleinberg et al. , 2016 ] . Let X be an instance space and let Y = { 0 , 1 } be the target set in a binary classification problem . In the fair classification setting , we may further assume the existence of a sensitive attribute s : X → { 1 , . . . , K } , where s ( x ) = k if and only if x ∈ Xk for some total partition X = ∪kXk . For example , X might correspond to the set of job applicants while s indicates their sex . Then , a commonly used criterion for fairness is to require similar mean outcomes across the sensitive attribute ( a.k.a . statistical parity ) [ Dwork et al. , 2012 , Zafar et al. , 2017a , Mehrabi et al. , 2019 ] : Definition 1 ( Statistical Parity ) . Let X be an instance space and X = ∪kXk be a total partition of X . A predictor h : X → [ 0 , 1 ] satisfies statistical parity across all groups X1 , . . . , XK if : max k∈ [ K ] Ex [ h ( x ) | x ∈ Xk ] − min k∈ [ K ] Ex [ h ( x ) | x ∈ Xk ] ≤ , where [ K ] denotes the set { 1 , . . . , K } . Our main contribution is to derive a near-optimal recipe for debiasing models , including deep neural networks , according to Definition 1 . Specifically , we formulate the task of debiasing learned models 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as a regularized optimization problem that is solved efficiently using the projected SGD method . We show how the algorithm produces thresholding rules with randomization near the thresholds , where the width of randomization is controlled by a regularization hyperparamter . We also prove that randomization near the threshold is , in general , necessary for Bayes risk consistency . In Appendix D , we show how the proposed algorithm can be modified to handle a weaker notion of bias as well . We refer to the proposed algorithm as Randomized Threshold Optimizer ( RTO ) . Besides the theoretical guarantees , we empirically validate RTO on benchmark datasets across both classical algorithms as well as modern DNN architectures . Our experiments demonstrate that the proposed algorithm significantly outperforms previous post-processing methods and performs competitively with in-processing ( Section 5 ) . While we focus on binary sensitive attributes in the experiments , our algorithm and its guarantees continue to hold for non-binary attributes as well . In addition , we show that RTO is particularly effective for models trained at scale where postprocessing is a natural and practical choice . Qualitatively speaking , for a fixed downstream task D , such as predicting facial attributes in the CelebA dataset [ Liu et al. , 2015 ] , we say that the model is “ trained at scale '' if it is both : ( 1 ) heavily overparameterized , and ( 2 ) pretrained on large datasets before fine-tuning on the downstream task D. We show that the impact of debiasing models on their performance using the proposed algorithm can be improved with scale . Remark . Because “ bias '' is a societal concept that can not be reduced to metrics such as statistical parity [ Chouldechova , 2017 , Dixon et al. , 2018 , Selbst et al. , 2019 ] , our conclusions do not necessarily pertain to “ fairness '' in its broader sense . Rather , they hold for the narrow technical definition of statistical parity . Similarly , we conduct experiments on standard benchmark datasets , such as CelebA [ Liu et al. , 2015 ] and COCO [ Lin et al. , 2014 ] , which are commonly used in the literature , as a way of validating the technical claims of this paper . Our experiments are , hence , not to be interpreted as an endorsement of those visions tasks , such as predicting facial attributes . 2 Related Work . Algorithms for fair machine learning can be broadly classified into three groups : ( 1 ) pre-processing methods , ( 2 ) in-processing methods , and ( 3 ) post-processing methods [ Zafar et al. , 2019 ] . Preprocessing algorithms transform the data into a different representation such that any classifier trained on it will not exhibit bias . This includes methods for learning a fair representation [ Zemel et al. , 2013 , Lum and Johndrow , 2016 , Bolukbasi et al. , 2016 , Calmon et al. , 2017 , Madras et al. , 2018 , Kamiran and Calders , 2012 ] , label manipulation [ Kamiran and Calders , 2009 ] , data augmentation [ Dixon et al. , 2018 ] , or disentanglement [ Locatello et al. , 2019 ] . On the other hand , in-processing methods constrain the behavior of learning algorithms in order to control bias . This includes methods based on adversarial training [ Zhang et al. , 2018 ] and constraintbased classification , such as by incorporating constraints on the decision margin [ Zafar et al. , 2019 ] or features [ Grgić-Hlača et al. , 2018 ] . Agarwal et al . [ 2018 ] showed that the task of learning an unbiased classifier could be reduced to a sequence of cost-sensitive classification problems , which could be applied to any black-box classifier . One caveat of the latter approach is that it requires solving a linear program ( LP ) and retraining classifiers , such as neural networks , many times before convergence . The algorithm we propose in this paper is a post-processing method , which can be justified theoretically [ Corbett-Davies et al. , 2017 , Hardt et al. , 2016 , Menon and Williamson , 2018 , Celis et al. , 2019 ] . Fish et al . [ 2016 ] and Woodworth et al . [ 2017 ] fall under this category . However , the former only provides generalization guarantees without consistency results while the latter proposes a twostage approach that requires changes to the original training algorithm . Kamiran et al . [ 2012 ] also proposes a post-processing algorithm , called Reject Option Classifier ( ROC ) , without any theoretical guarantees . In contrast , our algorithm is Bayes consistent and does not alter the original classification method . In Celis et al . [ 2019 ] and Menon and Williamson [ 2018 ] , instance-dependent thresholding rules are also learned . However , our algorithm also learns to randomize around the threshold ( Figure 1 ( a ) ) and this randomization is key to our algorithm both theoretically as well as experimentally ( Appendix B and Section 5 ) . Hardt et al . [ 2016 ] learns a randomized post-processing rule but our proposed algorithm outperforms it in all of our experiments ( Section 5 ) . Also , [ Wei et al. , 2019 ] is a post-processing method but it requires solving a non-linear optimization problem ( for the dual variables ) via ADMM and provides guarantees for approximate fairness only . Woodworth et al . [ 2017 ] showed that the post-processing approach can be suboptimal . Nevertheless , the latter result does not contradict the statement that our post-processing rule is near-optimal because we assume that the original classifier outputs a score ( i.e . a monotone transformation of an approximation to the posterior p ( y = 1 | x ) such as margin or softmax output ) whereas Woodworth et al . [ 2017 ] assumed that the post-processing rule had access to the binary predictions only . We argue that the proposed algorithm has distinct advantages , particularly for deep neural networks ( DNNs ) . First , stochastic convex optimization methods can scale well to massive amounts of data [ Bottou , 2010 ] , which is often the case in deep learning today . Second , the guarantees provided by our algorithm hold w.r.t . the binary predictions instead of using a proxy , such as the margin as in some previous works [ Zafar et al. , 2017b , 2019 ] . Third , unlike previous reduction methods that would require retraining a deep neural network several times until convergence [ Agarwal et al. , 2018 ] , which can be prohibitively expensive , our algorithm does not require retraining . Also , post-processing can be the only available option , such as when using machine learning as a service with out-of-the-box predictive models or due to various other constrains in data and computation [ Yang et al. , 2020b ] . 3 Near-Optimal Algorithm for Statistical Parity . Notation . We reserve boldface letters for random variables ( e.g . x ) , small letters for instances ( e.g . x ) , capital letters for sets ( e.g . X ) , and calligraphic typeface for universal sets ( e.g . the instance space X ) . Given a set S , 1S ( x ) ∈ { 0 , 1 } is its characteristic function . Also , we denote [ N ] = { 1 , . . . , N } and [ x ] + = max { 0 , x } . We reserve η ( x ) for the Bayes regressor : η ( x ) = p ( y = 1 | x = x ) . Algorithm . Given a classifier outputting a probability score p̂ ( y = 1 | x = x ) , let f ( x ) = 2p̂ ( y = 1 | x = x ) − 1 . We refer to f ( x ) ∈ [ −1 , +1 ] as the classifier ’ s output . Our goal is to post-process the predictions made by the classifier to control statistical parity with respect to a sensitive attribute s : X → [ K ] according to Definition 1 . To this end , instead of learning a deterministic rule , we consider randomized prediction rules h ( x ) , where h ( x ) is the probability of predicting the positive class given f ( x ) and s ( x ) . Note that we have the Markov chain : x→ ( s ( x ) , f ( x ) ) → h ( x ) . A simple approach of achieving statistical parity is to output a constant prediction in each subpopulation , which is clearly suboptimal in general . As such , there is a tradeoff between accuracy and fairness . The approach we take in this work is to modify the original classifier such that the original predictions are matched as much as possible while satisfying the fairness constraints . Minimizing the probability of altering the binary predictions of the original classifier can be achieved by maximizing the inner product Ex [ h ( x ) ·f ( x ) ] ( cf . Appendix B ) . However , maximizing this objective alone leads to deterministic thresholding rules which have a major drawback as illustrated in the following example . Example 1 ( Randomization is necessary ) . Suppose that X = { −1 , 0 , 1 } where p ( x = −1 ) = 1/2 , p ( x = 0 ) = 1/3 and p ( x = 1 ) = 1/6 . Let η ( −1 ) = 0 , η ( 0 ) = 1/2 and η ( 1 ) = 1 . In addition , let s ∈ { 0 , 1 } be a sensitive attribute , where p ( s = 1|x = −1 ) = 1/2 , p ( s = 1|x = 0 ) = 1 , and p ( s = 1|x = 1 ) = 0 . Then , the Bayes optimal prediction rule h ? ( x ) subject to statistical parity ( = 0 ) satisfies : p ( h ? ( x ) = 1|x = −1 ) = 0 , p ( h ? ( x ) = 1|x = 0 ) = 7/10 and p ( h ? ( x ) = 1|x = 1 ) = 1 . As a result , randomization close to the threshold is necessary in the general case to achieve Bayes risk consistency . This conclusion in Example 1 remains true with approximate fairness ( < 12/70 ) . In this work , we propose to achieve this by minimizing the following regularized objective for some hyperparameter γ > 0 : ( γ/2 ) Ex [ h ( x ) 2 ] − Ex [ h ( x ) · f ( x ) ] . ( 4 ) We prove in Appendix A that this regularization term leads to randomization around the threshold , which is critical , both theoretically ( Section 4 and Appendix B ) and experimentally ( Section 5 ) . Informally , γ controls the width of randomization as illustrated in Figure 1 . Let X = ∪kXk be a total partition of the instance space according to the sensitive attribute s : X → [ K ] . Denote a finite training sample by S = { ( x1 , y1 ) , . . . , ( xN , yN ) } and write Sk = S ∩Xk . For each group Sk , the fairness constraint in Definition 1 over the training sample can be written as : 1 |Sk| ∣∣∣ ∑ xi∈Sk ( h ( xi ) − ρ ) ∣∣∣ ≤ 2 , ( 5 ) for some hyper-parameter ρ ∈ [ 0 , 1 ] . Precisely , if the optimization variables h ( xi ) satisfy the constraint ( 5 ) , then Definition 1 holds over the training sample by the triangle inequality . Conversely , Algorithm 1 : Pseudocode of the Randomized Threshold Optimizer ( RTO ) . Input : γ > 0 ; ρ ∈ [ 0 , 1 ] ; ≥ 0 ; f : X → [ −1 , 1 ] ; s : X → [ K ] Output : Prediction rule : hγ ( x ) Training : Initialize ( λ1 , µ1 ) , . . . , ( λK , µK ) to zeros . Then , repeat until convergence : 1 . Sample an instance x ∼ p ( x ) 2 . Perform the updates : λs ( x ) ← [ λs ( x ) − ηgλs ( x ) ] + , µs ( x ) ← [ µs ( x ) − ηgµs ( x ) ] + ( 1 ) where : gλs ( x ) = 2 + ρ+ ∂ ∂λs ( x ) ξγ ( f ( x ) − ( λs ( x ) − µs ( x ) ) ) gµs ( x ) = 2 − ρ+ ∂ ∂µs ( x ) ξγ ( f ( x ) − ( λs ( x ) − µs ( x ) ) ) . and : ξγ ( w ) = w2 2γ · I { 0 ≤ w ≤ γ } + ( w − γ 2 ) · I { w > γ } ( 2 ) Prediction : Given an instance x in the group Xk , predict the label +1 with probability hγ ( x ) , where : hγ ( x ) = [ min { 1 , ( f ( x ) − λk + µk ) /γ ] + ( 3 ) if Definition 1 holds , then the constraint ( 5 ) also holds where : 2ρ = max k∈ [ K ] Ex [ h ( x ) | x ∈ Sk ] + min k∈ [ K ] Ex [ h ( x ) | x ∈ Sk ] . Therefore , to learn the post-processing rule h ( x ) , we solve the optimization problem : min 0≤h ( xi ) ≤1 ∑ xi∈S ( γ/2 ) h ( xi ) 2 − f ( xi ) h ( xi ) s.t . ∀k ∈ [ K ] : ∣∣ ∑ xi∈Sk ( h ( xi ) − ρ ) ∣∣ ≤ k , ( 6 ) in which k = |Sk| /2 for all k ∈ [ K ] . Using Lagrange duality we show in Appendix A that solving the above optimization problem is equivalent to Algorithm 1 . In Appendix D , we show that if = 0 , an alternative formulation can be used to minimize the same objective while satisfying the fairness constraint but without introducing a hyperparameter ρ . To reiterate , ρ ∈ [ 0 , 1 ] is tuned via a validation dataset and γ > 0 is a hyperparameter that controls randomization .

Summary: In this paper, the authors propose a post-processing method for debiasing trained models. The authors defined statistical parity as a 'bias'. The paper proposes a constrained optimization problem that takes the input data, sensitive attribute, partitioning and a trained model to yield a probabilistic decision rule. The authors proved theoretical properties and empirically validate its advantages on standard benchmark datasets across both classical algorithms.