Characterizing the risk of fairwashing

1 Introduction . As machine learning models are increasingly integrated into the pipeline of high-stakes decision processes , concerns about their transparency are becoming prominent and difficult to ignore for the actors deploying them . As a result , post-hoc explanation techniques have recently gained popularity as they may appear as a potentially viable solution to regain trust in machine learning models ’ predictions . More precisely , post-hoc explanation techniques refer to methods used to explain how black-box ML models produce their outcomes [ 26 , 9 ] . Current existing techniques for post-hoc explanations include global and local explanations . In a nutshell , global explanations focus on explaining the whole logic of the black-box model by training a surrogate model that is interpretable by design ( e.g. , linear models , rule-based models or decision trees ) while maximizing its fidelity to the black-box model . In contrast , local explanations aim at explaining a single decision by approximating the black-box model in the vicinity of the input point through an interpretable model . However , a growing body of works has recently shown that post-hoc explanation techniques not only can be misleading [ 38 , 46 ] but are also vulnerable to adversarial manipulations , wherein an adversary misleads users ’ trust by devising deceiving explanations . This phenomenon has been demonstrated for a broad range of post-hoc explanation techniques , including global and local explanations [ 4 , 49 ] , example-based explanations [ 25 ] , visualization-based explanations [ 31 , 20 ] and counterfactual explanations [ 36 ] . For instance , in a fairwashing attack [ 4 ] , the adversary manipulates the explanations to under-report the unfairness of the black-box models being explained . This attack can significantly impact individuals who have received a negative outcome following the model ’ s prediction while depriving them of the possibility of contesting it . ⇤Work done while at Université du Québec à Montréal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The fundamental question regarding fairwashing attacks is their manipulability , which we define as the ability to maximize the fidelity of an explanation model under an unfairness constraint . The manipulability of fairwashing attacks directly impacts the possibility to detect them . Indeed , if the manipulability is so low that the explanation manipulation can be detected , the risk of fairwashing is small and misleading decisions can be avoided . By contrast , if the manipulability is high enough the manipulation is undetectable , we will be under threat of the use of unfair models whose unfairness is hidden by malicious model producers through manipulated explanations . In this work , in the context of fairwashing for global explanations , we provide the first empirical results demonstrating that the manipulability of fairwashing is likely to be high . To assess the manipulability of fairwashing , we used the fidelity-unfairness trade-off and evaluated two characteristics of fairwashing , namely generalization and transferability . • Generalization of fairwashing beyond the suing group . In a fairwashing attack , a manipulated explanation is tailored specifically for a suing group of interest so that the explanation is fair within this group . As the explanation is specific to that group , we hypothesize that the same explanation can fail for another group . Based on this hypothesis , we assess the manipulability of fairwashing through its generalization capability . Our results suggest that the fidelity of the fairwashed explanation evaluated on another group is comparable to the one evaluated on the suing group . This means that the above hypothesis is negative , in the sense that explanations built to fairwash a suing group can generalize to another group not explicitly targeted by the attack . • Transferability of fairwashing beyond the targeted model . In the fairwashing attack , a manipulated explanation is targeted specifically for the deployed black-box model . However , in practical machine learning , it is usually the case that the deployed model is updated frequently . Thus , we hypothesize that there can be an inconsistency between the manipulated explanations provided to the suing group in the past and the currently deployed model . Based on this hypothesis , we quantify the manipulability of fairwashing through its transferability . Our results suggest that the fidelity of the fairwashed explanation evaluated on another model is comparable to the one evaluated on the deployed black-box model . Thus , the above hypothesis is also negative as fairwashed explanations designed for a specific model can also transfer to another model . Implications to undetectability . We observed the generalization and transferability of fairwashing attacks on several datasets , black-box models , explanation models and fairness criteria . As a consequence , our results indicate that detecting manipulated explanations based on the change of fidelity alone is not a viable solution ( or at least it is very difficult ) . Another way of quantifying fairwashing manipulability . In the above experiments , the manipulability of fairwashing was evaluated using the fidelity of explanation and its changes . Our negative results suggest that fidelity alone may not be an effective metric for quantifying the manipulability of fairwashing . Thus , we further investigated a different way of quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set ( FaiRS ) [ 17 ] framework . Our results indicate that this framework can be effectively used to quantify the manipulability of fairwashing . Related work . In the context of example-based explanations ’ manipulation , Fukuchi et al . [ 25 ] have demonstrated the risk of stealthily biased sampling , which occurs when a model producer explains the behaviour of its black-box model by sampling a subset of its training dataset . Slack et al . [ 49 ] have shown that variants of LIME [ 45 ] and SHAP [ 39 ] , two popular post-hoc local explanation techniques , can be manipulated to underestimate the unfairness of black-box models . Following the same line of work , Le Merrer and Trédan [ 37 ] have demonstrated that a malicious model producer can always craft a fake local explanation to hide the use of discriminatory features . Another work by Laugel et al . [ 36 ] has focused explicitly on the use of counterfactual explanations , which is a form of example-based explanation . They have demonstrated that generated instances can be unjustified ( i.e. , not supported by ground-truth data ) for most techniques existing for such type of explanation . Visualization-based explanation techniques can also be manipulated , as shown by recent works on saliency map-based explanations ’ manipulation . Heo et al . [ 31 ] have demonstrated that these types of explanations are vulnerable to the so-called adversarial model manipulation while Dombrowski et al . [ 20 ] have shown their vulnerability to adversarial input manipulation . In our previous work [ 4 ] , we introduced the notion of fairwashing as a rationalization exercise . We devised LaundryML , an algorithm that can systematically rationalize black-box models ’ decisions through global or local explanations . The details of this method , which form the basis of the fairwashing attacks in this study , are given in Section 2.2 . A subsequent work [ 35 ] has also investigated the possibility that black-box models can be explained with high fidelity by global interpretable models whose features are very different from that of the black-box and look innocuous . Finally , several negative aspects of post-hoc explanations have been reported in recent studies . For instance , post-hoc explanations can mislead model designers when they are used for debugging purposes [ 1 , 2 ] or can be leveraged to perform powerful model stealing attacks [ 42 , 6 ] . Outline . In Section 2 , we review the preliminary notions necessary to understand this work , and we describe how to explore the fidelity-unfairness trade-offs of a fairwashing attack by using an ✏-constraint method to solve its underlying multi-objective optimization problem . In Section 3 , we present the results obtained from our study for a diverse set of datasets , black-box models , explanation models and fairness metrics . In Section 4 , we consider quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set ( FaiRS ) [ 17 ] framework . Finally , we discuss the main implications of our findings in Section 5 as well as its limitations and societal impact in Section 6 . 2 Setting and problem formulation . 2.1 Notations . Let X 2 X ⇢ Rn denote a feature vector , Y 2 Y = { 0 , 1 } its associated binary label ( for simplicity we assume a binary classification setup without loss of generalization ) and G 2 G = { 0 , 1 } a feature defining a group membership ( e.g. , with respect to a sensitive attribute ) for every data point sampled from X . In addition , we assume that b : X ! Y refers to a black-box classifier of a particular model class B ( e.g. , neural network or ensemble model ) mapping any input X 2 X to its associated prediction Ŷ 2 Y . Finally , let e : X ! Y be a global explanation model from a particular model class E ( e.g. , linear model , rule list or decision tree ) designed to explain b . In the context of fairwashing attacks , which we formalize in Definition 2 , we refer to the data instances on which the attack is performed as the suing group . In this work , we measure the unfairness unfD ( f ) of a modelf on a dataset D by using statistical notions of fairness [ 13 , 15 , 16 , 30 ] , which require a model to exhibit approximate parity according to a statistical measure across the different groups defined by the group membership G. In particular , we consider four different statistical notions of fairness , namely statistical parity [ 22 , 13 , 33 , 23 , 51 ] , predictive equality [ 15 , 16 ] , equal opportunity [ 30 ] and equalized odds [ 15 , 34 , 30 , 50 ] . The definitions of these fairness metrics are listed in Table 1 . Table 1 : Summary of the different statistical notions of fairness considered . Fairness notion Definition Statistical Parity ( SP ) |P ( Ŷ = 1|G = 0 ) P ( Ŷ = 1|G = 1 ) | Predictive Equality ( PE ) |P ( Ŷ = 1|Y = 0 , G = 0 ) P ( Ŷ = 1|Y = 0 , G = 1 ) | Equal Opportunity ( EOpp ) |P ( Ŷ = 1|Y = 1 , G = 0 ) P ( Ŷ = 1|Y = 1 , G = 1 ) | Equalized Odds ( EOdds ) |P ( Ŷ = 1|Y = 1 , G = 0 ) P ( Ŷ = 1|Y = 1 , G = 1 ) | and |P ( Ŷ = 1|Y = 0 , G = 0 ) P ( Ŷ = 1|Y = 0 , G = 1 ) | 2.2 Problem formulation . Our investigation is motivated by our previous work [ 4 ] in which we defined fairwashing in global and local explanations as a manipulation exercise in which high-fidelity and fairer explanations can be designed to explain unfair black-box models . Definition 1 ( Global explanation fidelity ) Let b be a black-box model , e a global explanation model for b , and X a set of data instances . Following the definition in [ 18 ] , the fidelity of e with respect to b on X is expressed as : fidelity ( e ) = 1 |X| X x2X I ( e ( x ) = b ( x ) ) . Definition 2 ( Global fairwashing attack ) Let b be a black-box model and Xsg a set of data instances hereafter referred to as suing group . A global fairwashing attack consists in finding an interpretable global model e = p ( b , Xsg ) derived from the black-box b and the suing group Xsg using some attack process p ( · , · ) , such that e is fairer than b for a given fairness metric . To realize this , in our previous work [ 4 ] , we devised LaundryML , an algorithm that can systematically fairwash unfair black-box models ’ decisions through both global and local explanations . LaundryML is a constrained model enumeration technique [ 29 , 28 ] that searches for explanation models maximizing the fidelity while minimizing the unfairness for a given unfair black-box model . In this study , we go a step further by determining the fidelity-unfairness trade-offs of the fairwashing attack and characterizing the manipulability of the fairwashed explanations . For this purpose , we compute the set of Pareto optimal explanation models describing all the achievable fidelity-unfairness trade-offs by solving the following problem : minimize E ( xi , b ( xi ) ) ⇠Dsg [ l ( e ( xi ) , b ( xi ) ) ] , subject to unfDsg ( e ) ✏ , ( 1 ) in which e is the explanation model , l ( e ( x ) , b ( x ) ) is the loss function ( e.g. , cross entropy ) , Dsg = { Xsg , b ( Xsg ) } is formed by the suing group and the prediction of the black-box model b on the suing group , and ✏ is the value of the unfairness constraint .

Summary: This paper studies "fairwashed" explanation of black-box ML models -- interpretable models that achieve high fidelity with respect to the black-box predictions but substantially better scores on various fairness metrics. The authors empirically demonstrate how fairwashing can be achieved with very small decreases in fidelity, and how a fairwashed explanation of one type of model (say, random forests) can be successfully repurposed to explain a different model (eg AdaBoost). Because of this, they argue, auditors need to be very careful about how they use interpretable models in fairness assessments.