Impossibility results for fair representation

With the growing awareness to fairness in machine learning and the realization1 of the central role that data representation has in data processing tasks , there is2 an obvious interest in notions of fair data representations . We provide a formal3 framework for examining the fairness of data representations through the lens of4 their effect on decisions ( mainly classification ) made based on data represented that5 way . Using that framework , we prove that several desiderata for fair representations6 can not be achieved . While some of our conclusions are intuitive , we formulate7 ( and prove ) crisp statements of such impossibilities , often contrasting impressions8 conveyed by many recent works on fair representations.9 1 Introduction10 Automated decision making has become more and more successful over the last few decades and11 has therefore been used in an increasing number of domains , either as stand alone , or to support12 human decision makers . This includes many sensitive domains which significantly impact people ’ s13 livelihoods , such as loan applications , university admissions , recidivism predictions , or insurance rate14 settings . It has been found that many such decision tools have , often unintentionally , biases against15 minority groups , and therefore lead to discrimination . In response to these concerns , the machine16 learning research community has been devoting effort to developing clear notions of fair decision17 making , and coming up with algorithms for implementing fair machine learning.18 19 A common approach to address the important issue of fair algorithmic decision making is through fair20 data representation . The idea is that some regulator or a responsible data curator transforms collected21 data to a format ( or representation ) , that can then be used for solving downstream classification tasks22 providing guarantees of fairness . This approach was proposed by the seminal paper of Zemel et23 al . [ 15 ] . In their words : `` our intermediate representation can be used for other classification tasks24 ( i.e. , transfer learning is possible ) '' ... `` We further posit that such an intermediate representation is25 fundamental to progress in fairness in classification , since it is composable and not ad hoc ; once26 such a representation is established , it can be used in a blackbox fashion to turn any classification27 algorithm into a fair classifier , by simply applying the classifer to the sanitized representation of28 the data '' . Many followup papers aim to realize this paradigm , solving technical and algorithmic29 issues [ 10 , 6 , 11 , 14 , 3 ] ( to mention just a few ) . The main contribution of this paper is showing that,30 basically , it is impossible to achieve this goal . Namely , no data representation can guarantee that for31 every classification task a classifier trained on data under the given representation will be fair for32 that task . This impossibility applies even if one restricts the downstream tasks in question to share33 the same labeling rule , or for fairness notions like Odds Equality , to share the same marginal data34 distribution with the data on which the representation was trained . Our results answer negatively the35 main two questions posed in the discussion section of Creager et al . [ 3 ] .36 While many papers in this domain propose algorithmic solutions to fairness related issues , the main37 contributions of this paper are conceptual . We believe that , to a much larger extent than many other38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . facets of machine learning , fundamental concepts of fairness in machine learning require better39 understanding . Some basic questions are still far from being satisfactorily elucidated ; What should40 be considered fair decision making ? ( various mutually incompatible notions have been proposed , but41 how to pick between them for a given real life application is far from being clarified ) . What is a fair42 data representation ? To what extent should accuracy or other practical utilities be compromised for43 achieving fairness goals ? and so on . The answers to these questions are not generic . They vary with44 the principles and the goals guiding the agents involved ( decision makers , subjects of such a decision,45 policy regulators , etc . ) , as well as with what can be assumed regarding the underlying learning setup.46 We view these as the primary issues facing the field , deserving explicit research attention ( in addition47 to the more commonly discussed algorithmic and optimization aspects ) . This is a theoretical work,48 our discussion is grounded in definitions and proofs rather than heuristics and experimental results.49 1.1 What is fair representation ? 50 The term ‘ fair data representation ’ encompasses a wide range of different meanings . When word51 embeddings results in smaller distance between the vectors representing ‘ woman ’ and ‘ nurse ’ relative52 to the distance between the representations of ‘ woman ’ and ‘ doctor ’ and the other way around for53 ‘ man ’ , is it an indication of bias in the representation or is it just a faithful reflection of a bias in54 society ? Rather than delving into such issues , we discuss an arguably more concrete facet of data55 representation ; We examine representation fairness from the perspective of its effect on the fairness56 of classification rules that agents using data represented that way may come up with . Such a view57 takes into consideration two setup characteristics:58 The objective of the agent using the data We distinguish three types of classification prediction59 agents ( formal definitions of these aspects of fairness are provided in section 3.2 ) :60 Malicious - driven by a bias against a group of subjects . To protect against such an agent,61 a fair representation ( or feature set ) should be such that every classifier based on data62 represented that way is fair . This is apparently the most common approach to fair63 representations in the literature e.g. , [ 15 , 10 ] .64 Accuracy Driven - focusing on traditional measures of learning efficiency , ignoring fair-65 ness considerations . A representation is accuracy-driven fair if every loss minimizing66 classifier based on that representation is fair.67 Fairness Driven - aiming to find a decision rule that is fair while maintaining meaningful68 accuracy . A representation is fairness-driven fair if there exists a loss minimizing ( or69 an approximate minimizer ) classifier based on that representation is fair.70 The notion of group fairness applied to the classification decisions The wide range of group fair-71 ness notions ( for classification ) can be taxonomized along several dimensions : Does the72 notion depend on the ground truth classification or only on the agents decision ( like demo-73 graphic parity ) ? Is perfectly accurate decision ( matching the ground truth classification ) 74 always considered fair ( like in odds equality ) ? Does the fairness notion depend on unobserv-75 able features ( like intention or causality ) ? In this work we focus on fairness notions that76 are ground-truth-dependent , view the ground truth classification as fair and depend only on77 observable features . The decision which notion of fairness one wishes to abide by depends78 on societal goals and may vary from one task to another and is outside the scope of this79 paper . Just the same , let us briefly explain why the requirements listed above are natural in80 many situations.81 The dependence on the ground truth classification is almost inevitable from a utilitarian82 perspective - taking into account the probability that a student succeed or fail when83 making acceptance decisions should not be considered unfair . Put more formally , when-84 ever there is any correlation between membership and the ground truth classification,85 any classifier that is fair w.r.t . a notion that ignored the ground truth ( like demographic86 parity ) is bound to suffer prediction error proportional to that correlation.87 Viewing perfectly accurate decisions as fair can be viewed as a distinction between no-88 tions that do or do not try to inflict affirmative action . It makes a lot of sense in tasks89 like conviction in a crime - if you convict all criminals and no one else , you should not90 be accused on unfairness.91 Relying only on observable features fosters objectivity and allows scrutiny of the decisions92 made . Our running example of such a notion is odds equality [ 8 ] , however our results93 hold as well for other common notions of fairness that meet the above conditions ( like94 Calibrations Within Groups [ 9 ] ) .95 1.2 Our results96 We prove the following inherent limitations of notions of fair representations ( under the above97 taxonomy ) :98 1 . The impossibility to be task-independent . There is a host of literature proposing methods99 of coming up with data representation that guarantees the fairness of classifier based on100 that representation ( e.g. , [ 18 , 3 , 10 , 12 ] ) . We elaborate on these works in our Previous101 Work section . Contrasting the impression conveyed by many such papers , we show that102 the ability to guarantee multi-task fairness is inherently limited . Much of that work ad-103 dresses Demographic parity ( DP ) . We prove that if two tasks have different marginal data104 distributions ( that is , the distribution of unlabeled instances ) and different success rates105 of the protected group , then no representation can guarantee that any non-trivial classifier106 trained on it satisfies DP for both . We show that the only classifiers that are guaranteed to107 satisfy any significant level of DP fairness w.r.t . all marginal distributions are the redundant108 constant functions . From a practical point of view , since DP fairness of some decision ( say,109 acceptance to some university program ) requires the ratio of positive decisions between110 groups to match the ratio of applicants from those groups , a representation that guarantees111 DP fairness can not be a priory constructed - it must have access to the distribution of groups112 among applicants for that specific program . Furthermore , we prove that for every fixed113 marginal data distribution , if two ground truth classifications differ with non-zero probability114 over it , there can be no data representation that enjoys Odds Equality fairness and accuracy115 with respect to both tasks over that shared marginal distribution ( except for the redundant116 case where the success rates of both groups are equal for both tasks ) . These results answer117 negatively the main two open problems posed in the Discussion section of [ 3 ] .118 2 . The impossibility to evaluate the fairness contribution of a given feature devoid of the other119 features used ( again , for each agent objective and several common group fairness notions ) .120 3 . The inherent dependence of the effect on fairness of adding/deleting a feature on the type of121 agent using the representation ( on top of the above mentioned dependence on other features ) ,122 even when the feature in question does not correlate with membership in the protected group.123 ( These come on top of the obvious dependence on the notion of fair classification sought ) .124 Concerning potential negative societal impact : We can not foresee any potential negative societal125 impact of our work . The main message of this paper is a cautionary statement . We alert potential126 users that approaches based on task independent fair representations can not guarantee the fairness of127 arbitrary predictors based on them . As such , we are only guarding against potential negative impact128 of previously published work.129 Our paper is organized as follows : Section 2 gives an overview of the related work . Section 3130 introduces our setup including our taxonomy for fair representations . Section 4 contains our main131 results on the impossibility of generic fairness of a representation . Section 5 addressed the impos-132 sibility of defining the fairness effect of a single feature without considering the other components133 of a representation . Section 6 briefly shows the impossibility of having fair representations w.r.t.134 Predictive Rate Parity . Section 7 is our concluding remarks.135 We defer proofs to the appendix.136 2 Related Work137 Since our paper goes against messages conveyed by many previous papers , we wish to address in138 detail more related works than space here allows . We therefore provide a more elaborate section on139 previous work in the supplementary material.140 Much of the recent work on fair representation for learning classifiers focuses on algorithms . ( and141 demonstrating the viability of those algorithms though experimental results ) [ 15 , 10 , 17 , 1 , 16 ] . As142 explained before , our focus is different . We discuss what should be considered fair representation in143 that context , what is the scope of such notions and what are the inherent limitations of defining such144 representations.145 Almost all the work on fair representations focuses on the demographic parity ( DP ) notion of fairness146 [ 6 , 10 , 15 , 14 ] . Not having to take ground truth into account makes this notion independent of the147 classification task carrying both advantages and limitations . However , any positive result in these148 papers assumes that the marginal data distribution is available to the designer of the fair representation.149 Such an assumption severely restricts the applicability of such representations . To achieve DP fairness,150 a classifier has to induce success ratio between the two groups that match the ratio between these151 groups in the input data . However , that ratio , say a set of applicants for a bank loan or to some152 university program varies from one application to another and can not be determined a priori . Our153 results on this inherent limitation of fair representation for DP ( see section 4 ) do not seem to have154 been stated before.155 When the data marginal distribution is fixed , and available to the designer of a representation , DP156 fairness is possible . However , in such a setup , we show that fairness with respect to notions of fairness157 that do rely on the correct ground truth , such as equalized odds ( EO ) [ 8 ] , can not be guaranteed for158 arbitrary tasks ( see Section 4 ) . This fact also has not been explicitly stated ( and proved ) before,159 although it seems that some of the previous work worried about it . Instead , previous work either160 focus only on DP fairness , or , when it comes to discuss other notions of fairness , the algorithms that161 design the representations are assumed to have access to task specific labeled data ( e.g . [ 16 , 2 , 14 , 5 ] ,162 which defies the goal of having a fixed representation that guarantees fairness for many tasks.163 The effect of the motivation of the decision maker using the representation on the fairness of the164 resulting decision rule has been considered by Madras et al . [ 10 ] and Zhang et al . [ 16 ] . These papers165 identify two motivations . The first is malicious , which is the intent to discriminate without regard166 for accuracy . The second is accuracy-driven , which is the intent to maximize accuracy . We address167 these effects as part of our taxonomy of notions of fair representations . Additionally , we discuss168 fairness-driven agents that aim to achieve fairness while maintaining some level of accuracy.169 A natural question that arises in this context is about the inherent trade-offs between fairness and170 accuracy . When the notion of fairness is demographic parity , such trade-offs are clearly expected -171 they surface whenever there exists correlation between membership in the protected group and the172 ground truth classification . Zhao et al . [ 18 ] and Mcnamara et al . [ 11 ] analyze such scenarios and173 demonstrate situations in which there exists a more accurate and more fair classifier based on an174 original representation than any classifier built using a learnt representation.175 The question of feature deletion has also been considered in real world examples , such as in the `` ban176 the box '' policy which disallowed employers using criminal history in hiring decisions [ 4 ] . The effect177 of allowing or disallowing features on fairness has been studied before , for example in Grgic-Hlaca et178 al . [ 7 ] . However in previous works , the effect of a feature on fairness , has been discussed in isolation.179 In contrast , we show that fairness of a feature should not be considered in isolation , but should also180 take into account the remaining features available.181 3 Formal Setup182 We consider a binary classification problem with label set { 0 , 1 } over a domain X of instances we183 wish to classify , e.g . individuals applying for a loan . We assume the task to be given by some184 distribution P over X × { 0 , 1 } from which instances are sampled i.i.d . We denote the ground-truth185 labeling rule as t : X → [ 0 , 1 ] . We will think of the label 1 as denoting ‘ qualified ’ and the label 0 as186 ‘ unqualified ’ and t ( x ) = P [ y = 1|x ] . For concreteness , we focus here on the case of deterministic187 labeling ( that is t : X → { 0 , 1 } ) . Most of our discussion can readily be extended to the probabilistic188 labeling case . In a slight abuse of notation we will sometimes use t ( w ) to indicate the label coordinate189 of an instance w ∈ X × { 0 , 1 } 190 A data representation is determined by a mapping F : X → Z , for some set Z , and the learner only sees F ( x ) for any instance x ( both in the training and the test/decision stages ) .We denote the hypothesis class of all feature based decision rules asHF = { h : Z → { 0 , 1 } } . As a loss function we consider a weighted sum of false positives and false negatives , i.e . lα ( h , x , y ) = { α , if h ( x ) = 0 , y = 1 1− α , if h ( x ) = 1 , y = 0 0 , otherwise for some weight α ∈ ( 0 , 1 ) . We denote the true risk with respect to this loss as LαP and the empirical191 risk as LαS .192 3.1 Notions of group fairness193 For our fairness analysis we assume the population X to be partitioned into two subpopulation A and194 D ( namely , we restrict our discussion the case of one binary protected attribute ) . We sometimes use a195 function notation G : X → { A , D } to indicate the group-membership of an instance . Of course in196 reality there are often many protected attributes with more than two values . However , as our goal is to197 show limitations and impossibility results for fair representation learning , it suffices to only consider198 one binary protected attribute – the same impossibilities readily follow for the more complex settings.199 We now define two widely used notions of group-fairness that we will refer to throughout the paper,200 namely , equalized odds and demographic parity . In the following we will denote with Xg , l the subset201 of X with label l and group membership g , i.e . Xg , l = X ∩ t−1 ( l ) ∩G−1 ( g ) .202 Definition 1 ( Group fairness ; Equalized odds ) The notion of group-fairness we will focus on in203 this paper is the ground-truth-dependent notion of odds equality as introduced by [ 8 ] .204 A classifier h is considered fair w.r.t . to odds equality ( LEO ) and a distribution P if for x ∼ P we have the statistical independence h ( x ) ⊥⊥ G ( x ) |t ( x ) . For g ∈ { A , D } let the false positive rate and the false negative rate be defined as FPRg ( h , t , P ) = Px∼P [ h ( x ) = 1|x ∈ Xg,0 ] and FNRg ( h , t , P ) = Px∼P [ h ( x ) = 0|x ∈ Xg,1 ] respectively . The EO unfairness is given then by the sum of differences in false positive rate and false negative rate between groups : LEOP ( h ) = 1 2 |FNRA − FNRD|+ 1 2 |FPRA − FPRD| . If we say a classifier is fair , without referring to any particular group-fairness notion , we mean205 fairness w.r.t . equalized odds.206 Definition 2 ( Demographic parity ) A classifier h is considered fair w.r.t . to demographic parity207 ( LDP ) and a distribution P if h ( x ) ⊥⊥ G ( x ) . The respective unfairness is given by difference in208 positive classification rates between groups209 LDPP ( h ) = |Px∼P [ h ( x ) = 1|G ( x ) = A ] − Px∼P [ h ( x ) = 1|G ( x ) = D ] |.210 3.2 The role of the agent ’ s objective211 We will phrase our definitions of representation fairness in terms of a general group fairness notion212 Lfair with unfairness measure LfairP .213 We start by considering a malicious decision maker who tries to actively discriminate against one214 group . To protect against this kind of decision maker , we need to give a guarantee such that based on215 the feature set it is not possible to discriminate against one group . This corresponds to the notion of216 adversarial fairness.217 Definition 3 ( Adversarial fairness ) A representation F is considered to be adversarial fair w.r.t . the218 distribution P and group fairness objective Lfair , if every classifier h ∈ HF is group-fair . We define219 the adversarial unfairness of a representation F by Uadv ( F ) = maxh∈HF L fair P ( h ) .220 Furthermore , we consider an accuracy-driven decision maker , who aims to label instances correctly221 and is agnostic about fairness . For this kind of decision maker , we only need to make sure that222 optimizing for correct classification results in a fair classifier . The following definition ensures that223 the Bayes optimal classifier for a representation is fair.224 Definition 4 ( Accuracy-driven fairness ) A representation F is considered to be accuracy-driven225 fair w.r.t . the fairness objectiveLfair and distribution P , if for every threshold α ∈ ( 0 , 1 ) , every classi-226 fier h ∈ HF with LαP ( h ) = minh∈HF LαP ( h ) is group-fair . The accuracy-driven unfairness for a par-227 ticular threshold parameter α is given by Uαacc ( F ) = max { L fair P ( h ) : h ∈ argminh∈HF LαP ( h ) } .228 The general accuracy-driven unfairness is given by Uacc ( F ) = maxα∈ [ 0,1 ] Uαacc ( F ) .229 We note that in cases where the decision maker does not have access to the distribution P , but230 only to a labelled sample , this requirement is might not sufficient for guaranteeing that an accuracy-231 driven decision maker arrives at a fair decision . In the Appendix we propose another fairness notion232 ( λ-robustness ) that formalizes the desired fairness guarantee for this scenario.233 Lastly , we also consider a fairness-driven decision maker who actively tries to find a fair and234 accurate decision rule , while maintaining some accuracy guarantees . For such a decision maker235 a representation should allow for fair and accurate decision rules . If a representation fulfills this236 requirement , we call it fairness-enabling.237 Definition 5 ( ( , η ) -fairness-enabling representation ) A representation F is considered to be238 ( , η ) -fairness-enabling w.r.t . a fairness objective Lfair , if there exists a classifier h ∈ HF that239 such that LαP ( h ) ≤ and L fair P ( h ) ≤ η.240 Our discussion focuses primarily on the case of malicious and indifferent decision makers . These241 notions of fair representation can be defined with respect to any group-fairness notion . In our paper242 we will mainly focus on the equalized odds notion of fairness [ 8 ] . We also note that all the above243 definitions can be given with respect to a fixed modelH in a continuous space.244 4 Can there be a generic fair representation ? 245 We address the existence of a multi-task fair representation . We prove that for the adversarial agent246 scenario ( which is the setup that most fairness representation previous work is concerned with ) ,247 it is impossible to have generic non-trivial fair representations - no useful representation can248 guarantee fairness for all `` downstream '' classification that are based on that representation ( even if249 the ground truth classification remains unchanged and only the marginal may change between tasks ) .250 We start by considering scenarios in which only the marginals shift between two tasks , e.g . two251 openings for different jobs , requiring similar skills , for which different pools of people would apply.252 Such a distribution shift can likely affect one group more than another and would thus affect the253 classification rates of both groups differently . We show that we can not guarantee fairness of a fixed254 data presentation for general shifts of this kind , even for the simplest case of demographic parity.255 Claim : 1 Pick any domain set X and any partition of X into non-empty subsets A , D . For every256 non-constant function f : X → { 0 , 1 } there exists a probability distribution P over X such that f is257 arbitrarily DP-unfair w.r.t . P ( say , LDPP ( h ) > 0.9 ) .258 In particular , when a shift in marginal occurs between tasks , fairness for previous tasks does not259 imply a fairness guarantee for a new task.260 Proof : If f is constant on any of the groups A or D then , since f is not a constant over X there is261 are points in the other group on which f has the opposite value . Let P assigns probability 0.5 to the262 group on which f is constant and probability 0.5 to the set of points to which f assigns the other263 value . Clearly f fails DP w.r.t . this P . Otherwise , both values are assigned in both groups , so let P264 assign probability 0.5 to { x ∈ A : f ( x ) = 0 } and probability 0.5 to { x ∈ D : f ( x ) = 1 } . Clearly,265 f fails DP w.r.t . this P .266 Corollary 1 No data representation can guarantee the DP fairness of any non-trivial classifier w.r.t.267 all possible data generating distributions ( over any fixed domain set with any fixed partition into268 non-empty groups ) . That is , any non-constant representation F , can not be adversarially fair with269 respect to LDP and any arbitrary task P .270 Claim : 2 Pick any domain set X and any partition of X into non-empty subsets A , D . For every271 non-constant function f : X → { 0 , 1 } and every classifier h : X → { 0 , 1 } such that h 6= f272 there exists a probability distribution P over X such that h is arbitrarily EO-unfair w.r.t . P , f , say273 LEOP , f > 0.9.274 Corollary 2 No data representation can guarantee EO fairness of any non-constant predictor based275 on that representation for all `` downstream '' classification learning tasks . That is , any non-constant276 representation F , can not be adversarially fair with respect to LEO and any arbitrary task P . This277 holds even if one restricts the claim to tasks sharing a fixed marginal data distribution.278 We will now look at a slightly more restricted setting and analyse the case of multi-task learning,279 where instead of asking for a representation that is fair for every task , we only consider fairness with280 respect to a fixed ( finite ) set of tasks that we want to learn . We find that for the adversarial case , even281 this less ambitious goal is not achievable for generic tasks and the equalized odds notion of fairness.282 We say a distribution P has equal success rates if P ( XA,1 ) P ( A ) = P ( XD,1 ) P ( D ) .283 Lemma 1 Let P1 and P2 be the distributions defining two different tasks with the same marginal284 PX = P1 , X = P2 , X such that at least one of the tasks does not have equal success rates . Let285 h1 , h2 : X → { 0 , 1 } be such that LP1 ( h1 ) = LP2 ( h2 ) = 0 , and assume that tasks are non-negligibly286 different ( namely , LP1 ( h2 ) 6= 0 ) . Then , it can not be the case that both h1 and h2 are EO fair w.r.t.287 both P1 and P2.288 The proof ( in the appendix ) has a similar flavour as the proof of incompetability of different fairness289 notions of [ 9 ] .290 Theorem 1 There can be no data representation F such that for some P1 , P2 as above , the following291 criteria simultaneously hold:292 1 . F is adversarially fair w.r.t . P1 and EO293 2 . F is adversarially fair w.r.t . P2 and EO294 3 . F allows for perfect accuracy w.r.t . to P1 and P2 , i.e . there are h1 , h2 both expressible over295 the representation F , such that LP1 ( h1 ) = LP2 ( h2 ) = 0.296 This result follows directly from Lemma 1 . Therefore , if the goal is to prevent discrimination from a297 possibly adversarial decision maker , while also enabling accurate prediction , each task requires its298 task-specific feature representation.299 5 Fairness of a feature set vs. fairness of a feature300 In this section we discuss feature deletion and its impact on the fairness of a representation . For301 this we assume our representation F to consist of finitely many features fi : X → Yi i.e . for302 every x ∈ X : F ( x ) = ( f1 ( x ) , . . . , fn ( x ) ) and Z = Y1 × . . . × Yn . We limit our discussion to303 cases where all Yi are finite . While this assumption facilitates our analysis , we do not expect our304 results to be different in the cases of continuous features . We will denote the set of features as305 F = { f1 , . . . , fn } and will denote by Uadv ( F ) and Uαacc ( F ) the adversarial and accuracy-driven306 fairness of the representation induced by the feature set F respectively . We show that it is in general307 not possible to determine the effect a single feature has on the fairness of a representation without308 considering the full representation . This is the case even if our considered feature is not correlated309 with the protected attribute.310 5.1 Opposing effects of a feature for accuracy-driven fairness of a representation311 We start our discussion with accuracy-driven fairness w.r.t . equalized odds . In this case we show that312 the deletion of a feature f can lead to an increase in accuracy-driven unfairness for some set of other313 given features F and that the deletion of the same feature f can lead to a decrease in accuracy-driven314 unfairness for another set of other available features F ′ . This implies that the fairness of the feature315 f can not be evaluated without context . We show that this phenomena holds for a general class of316 features that satisfy some non-triviality properties ( That on the one hand do not reveal too much317 information about group membership and labels ( non-committing ) , and on the other hand does not318 reveal identity when label and group information is given ( k-anonymity [ 13 ] ) ) . The exact definitions319 of these properties can be found in the appendix.320 Theorem 2 ( Context-relevance for fairness of features ) For every 6-anonymous non-committing321 feature f , there exists a probability function P over X and feature sets F and F ′ such that:322 • The accuracy-driven fairness w.r.t LEO , P and α = 0.5 of F ∪ { f } is greater than that of323 F , i.e.324 Uαacc ( F ∪ { f } ) < Uαacc ( F ) Thus , deleting f in this context will increase unfairness.325 • The accuracy-driven fairness w.r.t LEO , P and α = 0.5 of F ′ ∪ { f } is less than that of F ′,326 i.e.327 Uαacc ( F ′ ∪ { f } ) > Uαacc ( F ′ ) Thus , deleting f in this context will decrease unfairness.328 This phenomenon can happen even if { f } is adversarially fair w.r.t . to P and equalized odds.329 5.2 The fairness of a feature for different notions of fairness330 We will now briefly discuss the effect of a single feature on fairness for the cases of a malicious331 or a fairness-driven decision makers . In contrast to the accuracy-driven case , adding features has a332 monotone effect on the fairness of a fairness-driven and the malicious decision maker . As Theorem 3,333 adding any feature in the malicious case , will only give the decision maker more information and334 thus give the decision maker more chances of discrimination . Similarly in the fairness driven case,335 any feature will only give the decision maker another option for fair decision making ( Theorem 4 ) .336 However , the quantitative effect of adding a feature on the unfairness can still range from having no337 effect to achieving perfect fairness/unfairness for both the fairness-driven and the malicious case.338 As in the accuracy-driven case , we will show ( Theorem 4 and Theorem 3 ) that it is impossible to339 evaluate the quantitative effect of a feature on the fairness of a representation without considering the340 context of other available features.341 Theorem 3 1 . For every distribution P and feature f , there exists a feature set F , such that342 adding f will not impact the fairness of the distribution , e.g . Uadv ( F ) = Uadv ( F ∪ { f } ) .343 2 . There exist distributions P , features f and F ′ , such that Uadv ( F ′ ) = 0 and Uadv ( { f } ) = 0,344 but Uadv ( F ′ ∪ { f } ) = 1 .345 Theorem 4 1 . For any feature f and any featureset F we have Uadv ( F ) ≤ Uadv ( F ∪ { f } ) .346 Similarly , if the representation F is ( , η ) -fairness-enabling , the representation F ∪ { f } is347 also ( , η ) -fairness-enabling.348 2 . For every distribution P and every feature f , there exists a feature set F , such that F ∪ { f } 349 is ( η , ) -fairness-enabling , if and only if F is ( , η ) -fairness-enabling . Furthermore , there350 exists a distribution P , a feature f and a feature set F ′ , such that both F ′ and { f } are not351 ( , η ) -fairness-enabling for any , η < 12 , but such that F ′ ∪ { f } is ( 0 , 0 ) -fairness-enabling.352 While this section focused on fairness with respect to equalized odds , we note that many of these353 results can be replicated for other notions of fairness . For a more general version of Theorem 3,354 which takes into account other fairness notions , like demographic parity , we will refer the reader to355 the Appendix.356 6 Impossibility of adversarially fair representations with respect to357 predictive rate parity358 We now show that not all acceptable notions of group fairness always allow a adversarially fair359 representation , even in a single-task setting . One such notion is predictive rate parity.360 Definition 6 ( Predictive rate parity ( PRP ) ) A classifier h is considered PRP fair w.r.t . to a marginal361 data distribution P and true classification t if the random variable t ( x ) is independent of the group362 membership , G ( x ) given the classification h ( x ) . We denote this fairness objective with LPred.363 Theorem 5 Adversarial fairness w.r.t . P and LPred is only possible , if P has equal success rates364 for both groups.365 This theorem results from the fact that the classifier which maps every instance to label 1 is not366 fair w.r.t . to LPred if P does not have equal success rates . The quantitative version of predictive367 rate parity as well as a more general version of Theorem 5 , giving a characterization of adversarial368 fairness in the case of equal success rates can be found in the appendix.369 7 Conclusion370 In this paper we introduced a general taxonomy of notions of fair representation , taking into considera-371 tion both different objectives of decision makers using the representation , and different group fairness372 notions . Within this taxonomy we showed several impossibility results about fair representation373 learning.374 Our main result addressed the existence of generic fair representations and of fair transfer learn-375 ing . We show that even seemingly task-independent fairness notions like demographic parity are376 vulnerable to shifts in marginals between tasks . We conclude the impossibility of having generic377 data representations that guarantee ( even just ) DP fairness with respects to tasks whose marginal378 distributions are not considered when designing the representation . Furthermore , we show that it is379 impossible to have an adversarially fair representation with respect to several tasks and the equalized380 odds notion of fairness , if those tasks do not all fulfill statistical parity . These insights stand in contrast381 to the impression arising from recent papers [ 10 ] that claim to learned transferable fair decisions.382 We also considered the question of `` fairness of a feature '' , which has been used in legal scenarios . We383 showed that for notions of decision-making fairness other than demographic parity , the fairness of a384 single feature is an ill defined notion . Namely , the impact of a feature on the fairness of a decision385 can not be determined without considering the other features of the representation.386 Lastly , we show that some fairness notions , like predictive rate parity , do not always allow an387 adversarially fair representation , even if it is just for a single task.388 References389 [ 1 ] Tameem Adel , Isabel Valera , Zoubin Ghahramani , and Adrian Weller . One-network adversarial390 fairness . In AAAI , 2019.391 [ 2 ] Alex Beutel , Jilin Chen , Zhe Zhao , and Ed H. Chi . Data decisions and theoretical implications392 when adversarially learning fair representations . CoRR , abs/1707.00075 , 2017.393 [ 3 ] Elliot Creager , David Madras , Joern-Henrik Jacobsen , Marissa Weis , Kevin Swersky , Toniann394 Pitassi , and Richard Zemel . Flexibly fair representation learning by disentanglement . In ICML,395 2019.396 [ 4 ] Jennifer L Doleac and Benjamin Hansen . Does “ ban the box ” help or hurt low-skilled work-397 ers ? statistical discrimination and employment outcomes when criminal histories are hidden.398 Technical report , National Bureau of Economic Research , 2016.399 [ 5 ] Flávio du Pin Calmon , Dennis Wei , Bhanukiran Vinzamuri , Karthikeyan Natesan Ramamurthy,400 and Kush R. Varshney . Optimized pre-processing for discrimination prevention . In Advances in401 Neural Information Processing Systems 30.402 [ 6 ] Harrison Edwards and Amos J. Storkey . Censoring representations with an adversary . In ICLR,403 2016.404 [ 7 ] Nina Grgic-Hlaca , Muhammad Bilal Zafar , Krishna P. Gummadi , and Adrian Weller . Beyond405 distributive fairness in algorithmic decision making : Feature selection for procedurally fair406 learning . In AAAI , 2018.407 [ 8 ] Moritz Hardt , Eric Price , and Nathan Srebro . Equality of opportunity in supervised learning . In408 NIPS , 2016.409 [ 9 ] Jon M. Kleinberg , Sendhil Mullainathan , and Manish Raghavan . Inherent trade-offs in the fair410 determination of risk scores . CoRR , abs/1609.05807 , 2016.411 [ 10 ] David Madras , Elliot Creager , Toniann Pitassi , and Richard Zemel . Learning adversarially fair412 and transferable representations . In ICML , 2018.413 [ 11 ] Daniel McNamara , Cheng Soon Ong , and Robert C Williamson . Costs and benefits of fair414 representation learning . In Proceedings of the 2019 AAAI/ACM Conference on AI , Ethics , and415 Society , pages 263–270 , 2019.416 [ 12 ] Luca Oneto , Michele Donini , Andreas Maurer , and Massimiliano Pontil . Learning fair and417 transferable representations . arXiv preprint arXiv:1906.10673 , 2019.418 [ 13 ] Pierangela Samarati and Latanya Sweeney . Protecting privacy when disclosing information:419 k-anonymity and its enforcement through generalization and suppression . Technical report,420 1998.421 [ 14 ] Jiaming Song , Pratyusha Kalluri , Aditya Grover , Shengjia Zhao , and Stefano Ermon . Learning422 controllable fair representations . In The 22nd International Conference on Artificial Intelligence423 and Statistics , pages 2164–2173 , 2019.424 [ 15 ] Rich Zemel , Yu Wu , Kevin Swersky , Toni Pitassi , and Cynthia Dwork . Learning fair representa-425 tions . In ICML , 2013.426 [ 16 ] Brian Hu Zhang , Blake Lemoine , and Margaret Mitchell . Mitigating unwanted biases with427 adversarial learning . In AAAI/ACM Conference on AI , Ethics , and Society , 2018.428 [ 17 ] Han Zhao , Amanda Coston , Tameem Adel , and Geoffrey J. Gordon . Conditional learning of429 fair representations . CoRR , abs/1910.07162 , 2019.430 [ 18 ] Han Zhao and Geoffrey J. Gordon . Inherent tradeoffs in learning fair representations . CoRR,431 abs/1906.08386 , 2019.432 Checklist433 1 . For all authors ... 434 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s435 contributions and scope ? [ Yes ] We provide theorems with proofs for all claims made436 in the introduction and appendix437 ( b ) Did you describe the limitations of your work ? [ Yes ] The introduction clearly details438 the scope of our paper439 ( c ) Did you discuss any potential negative societal impacts of your work ? [ Yes ] We discuss440 potential negative societal impacts at the end of the introduction.441 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to442 them ? [ Yes ] 443 2 . If you are including theoretical results ... 444 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] Every result445 clearly states the assumptions used . In one instance ( Theorem 2 ) we refer to a detailed446 definition of the assumptions in the appendix.447 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Many of them were448 refered to the appendix449 3 . If you ran experiments ... 450 ( a ) Did you include the code , data , and instructions needed to reproduce the main experi-451 mental results ( either in the supplemental material or as a URL ) ? [ N/A ] 452 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they453 were chosen ) ? [ N/A ] 454 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-455 ments multiple times ) ? [ N/A ] 456 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type457 of GPUs , internal cluster , or cloud provider ) ? [ N/A ] 458 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 459 ( a ) If your work uses existing assets , did you cite the creators ? [ N/A ] 460 ( b ) Did you mention the license of the assets ? [ N/A ] 461 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 462 463 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re464 using/curating ? [ N/A ] 465 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable466 information or offensive content ? [ N/A ] 467 5 . If you used crowdsourcing or conducted research with human subjects ... 468 ( a ) Did you include the full text of instructions given to participants and screenshots , if469 applicable ? [ N/A ] 470 ( b ) Did you describe any potential participant risks , with links to Institutional Review471 Board ( IRB ) approvals , if applicable ? [ N/A ] 472 ( c ) Did you include the estimated hourly wage paid to participants and the total amount473 spent on participant compensation ? [ N/A ] 474

Summary: This paper demonstrates two impossibilities raised in a fair machine learning context: (i) the impossibility of achieving a generic fair data representation; (ii) the impossibility of achieving fairness notions (except disparate parity) in the case of manipulating a single feature. The paper also answers the two questions raised in Creager et. al. [3] via theoretical analysis under binary classification setting w.r.t. varying fairness notions.