Characterizing the risk of fairwashing

1 Introduction . As machine learning models are increasingly integrated into the pipeline of high-stakes decision processes , concerns about their transparency are becoming prominent and difficult to ignore for the actors deploying them . As a result , post-hoc explanation techniques have recently gained popularity as they may appear as a potentially viable solution to regain trust in machine learning models ’ predictions . More precisely , post-hoc explanation techniques refer to methods used to explain how black-box ML models produce their outcomes [ 26 , 9 ] . Current existing techniques for post-hoc explanations include global and local explanations . In a nutshell , global explanations focus on explaining the whole logic of the black-box model by training a surrogate model that is interpretable by design ( e.g. , linear models , rule-based models or decision trees ) while maximizing its fidelity to the black-box model . In contrast , local explanations aim at explaining a single decision by approximating the black-box model in the vicinity of the input point through an interpretable model . However , a growing body of works has recently shown that post-hoc explanation techniques not only can be misleading [ 38 , 46 ] but are also vulnerable to adversarial manipulations , wherein an adversary misleads users ’ trust by devising deceiving explanations . This phenomenon has been demonstrated for a broad range of post-hoc explanation techniques , including global and local explanations [ 4 , 49 ] , example-based explanations [ 25 ] , visualization-based explanations [ 31 , 20 ] and counterfactual explanations [ 36 ] . For instance , in a fairwashing attack [ 4 ] , the adversary manipulates the explanations to under-report the unfairness of the black-box models being explained . This attack can significantly impact individuals who have received a negative outcome following the model ’ s prediction while depriving them of the possibility of contesting it . ⇤Work done while at Université du Québec à Montréal 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The fundamental question regarding fairwashing attacks is their manipulability , which we define as the ability to maximize the fidelity of an explanation model under an unfairness constraint . The manipulability of fairwashing attacks directly impacts the possibility to detect them . Indeed , if the manipulability is so low that the explanation manipulation can be detected , the risk of fairwashing is small and misleading decisions can be avoided . By contrast , if the manipulability is high enough the manipulation is undetectable , we will be under threat of the use of unfair models whose unfairness is hidden by malicious model producers through manipulated explanations . In this work , in the context of fairwashing for global explanations , we provide the first empirical results demonstrating that the manipulability of fairwashing is likely to be high . To assess the manipulability of fairwashing , we used the fidelity-unfairness trade-off and evaluated two characteristics of fairwashing , namely generalization and transferability . • Generalization of fairwashing beyond the suing group . In a fairwashing attack , a manipulated explanation is tailored specifically for a suing group of interest so that the explanation is fair within this group . As the explanation is specific to that group , we hypothesize that the same explanation can fail for another group . Based on this hypothesis , we assess the manipulability of fairwashing through its generalization capability . Our results suggest that the fidelity of the fairwashed explanation evaluated on another group is comparable to the one evaluated on the suing group . This means that the above hypothesis is negative , in the sense that explanations built to fairwash a suing group can generalize to another group not explicitly targeted by the attack . • Transferability of fairwashing beyond the targeted model . In the fairwashing attack , a manipulated explanation is targeted specifically for the deployed black-box model . However , in practical machine learning , it is usually the case that the deployed model is updated frequently . Thus , we hypothesize that there can be an inconsistency between the manipulated explanations provided to the suing group in the past and the currently deployed model . Based on this hypothesis , we quantify the manipulability of fairwashing through its transferability . Our results suggest that the fidelity of the fairwashed explanation evaluated on another model is comparable to the one evaluated on the deployed black-box model . Thus , the above hypothesis is also negative as fairwashed explanations designed for a specific model can also transfer to another model . Implications to undetectability . We observed the generalization and transferability of fairwashing attacks on several datasets , black-box models , explanation models and fairness criteria . As a consequence , our results indicate that detecting manipulated explanations based on the change of fidelity alone is not a viable solution ( or at least it is very difficult ) . Another way of quantifying fairwashing manipulability . In the above experiments , the manipulability of fairwashing was evaluated using the fidelity of explanation and its changes . Our negative results suggest that fidelity alone may not be an effective metric for quantifying the manipulability of fairwashing . Thus , we further investigated a different way of quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set ( FaiRS ) [ 17 ] framework . Our results indicate that this framework can be effectively used to quantify the manipulability of fairwashing . Related work . In the context of example-based explanations ’ manipulation , Fukuchi et al . [ 25 ] have demonstrated the risk of stealthily biased sampling , which occurs when a model producer explains the behaviour of its black-box model by sampling a subset of its training dataset . Slack et al . [ 49 ] have shown that variants of LIME [ 45 ] and SHAP [ 39 ] , two popular post-hoc local explanation techniques , can be manipulated to underestimate the unfairness of black-box models . Following the same line of work , Le Merrer and Trédan [ 37 ] have demonstrated that a malicious model producer can always craft a fake local explanation to hide the use of discriminatory features . Another work by Laugel et al . [ 36 ] has focused explicitly on the use of counterfactual explanations , which is a form of example-based explanation . They have demonstrated that generated instances can be unjustified ( i.e. , not supported by ground-truth data ) for most techniques existing for such type of explanation . Visualization-based explanation techniques can also be manipulated , as shown by recent works on saliency map-based explanations ’ manipulation . Heo et al . [ 31 ] have demonstrated that these types of explanations are vulnerable to the so-called adversarial model manipulation while Dombrowski et al . [ 20 ] have shown their vulnerability to adversarial input manipulation . In our previous work [ 4 ] , we introduced the notion of fairwashing as a rationalization exercise . We devised LaundryML , an algorithm that can systematically rationalize black-box models ’ decisions through global or local explanations . The details of this method , which form the basis of the fairwashing attacks in this study , are given in Section 2.2 . A subsequent work [ 35 ] has also investigated the possibility that black-box models can be explained with high fidelity by global interpretable models whose features are very different from that of the black-box and look innocuous . Finally , several negative aspects of post-hoc explanations have been reported in recent studies . For instance , post-hoc explanations can mislead model designers when they are used for debugging purposes [ 1 , 2 ] or can be leveraged to perform powerful model stealing attacks [ 42 , 6 ] . Outline . In Section 2 , we review the preliminary notions necessary to understand this work , and we describe how to explore the fidelity-unfairness trade-offs of a fairwashing attack by using an ✏-constraint method to solve its underlying multi-objective optimization problem . In Section 3 , we present the results obtained from our study for a diverse set of datasets , black-box models , explanation models and fairness metrics . In Section 4 , we consider quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set ( FaiRS ) [ 17 ] framework . Finally , we discuss the main implications of our findings in Section 5 as well as its limitations and societal impact in Section 6 . 2 Setting and problem formulation . 2.1 Notations . Let X 2 X ⇢ Rn denote a feature vector , Y 2 Y = { 0 , 1 } its associated binary label ( for simplicity we assume a binary classification setup without loss of generalization ) and G 2 G = { 0 , 1 } a feature defining a group membership ( e.g. , with respect to a sensitive attribute ) for every data point sampled from X . In addition , we assume that b : X ! Y refers to a black-box classifier of a particular model class B ( e.g. , neural network or ensemble model ) mapping any input X 2 X to its associated prediction Ŷ 2 Y . Finally , let e : X ! Y be a global explanation model from a particular model class E ( e.g. , linear model , rule list or decision tree ) designed to explain b . In the context of fairwashing attacks , which we formalize in Definition 2 , we refer to the data instances on which the attack is performed as the suing group . In this work , we measure the unfairness unfD ( f ) of a modelf on a dataset D by using statistical notions of fairness [ 13 , 15 , 16 , 30 ] , which require a model to exhibit approximate parity according to a statistical measure across the different groups defined by the group membership G. In particular , we consider four different statistical notions of fairness , namely statistical parity [ 22 , 13 , 33 , 23 , 51 ] , predictive equality [ 15 , 16 ] , equal opportunity [ 30 ] and equalized odds [ 15 , 34 , 30 , 50 ] . The definitions of these fairness metrics are listed in Table 1 . Table 1 : Summary of the different statistical notions of fairness considered . Fairness notion Definition Statistical Parity ( SP ) |P ( Ŷ = 1|G = 0 ) P ( Ŷ = 1|G = 1 ) | Predictive Equality ( PE ) |P ( Ŷ = 1|Y = 0 , G = 0 ) P ( Ŷ = 1|Y = 0 , G = 1 ) | Equal Opportunity ( EOpp ) |P ( Ŷ = 1|Y = 1 , G = 0 ) P ( Ŷ = 1|Y = 1 , G = 1 ) | Equalized Odds ( EOdds ) |P ( Ŷ = 1|Y = 1 , G = 0 ) P ( Ŷ = 1|Y = 1 , G = 1 ) | and |P ( Ŷ = 1|Y = 0 , G = 0 ) P ( Ŷ = 1|Y = 0 , G = 1 ) | 2.2 Problem formulation . Our investigation is motivated by our previous work [ 4 ] in which we defined fairwashing in global and local explanations as a manipulation exercise in which high-fidelity and fairer explanations can be designed to explain unfair black-box models . Definition 1 ( Global explanation fidelity ) Let b be a black-box model , e a global explanation model for b , and X a set of data instances . Following the definition in [ 18 ] , the fidelity of e with respect to b on X is expressed as : fidelity ( e ) = 1 |X| X x2X I ( e ( x ) = b ( x ) ) . Definition 2 ( Global fairwashing attack ) Let b be a black-box model and Xsg a set of data instances hereafter referred to as suing group . A global fairwashing attack consists in finding an interpretable global model e = p ( b , Xsg ) derived from the black-box b and the suing group Xsg using some attack process p ( · , · ) , such that e is fairer than b for a given fairness metric . To realize this , in our previous work [ 4 ] , we devised LaundryML , an algorithm that can systematically fairwash unfair black-box models ’ decisions through both global and local explanations . LaundryML is a constrained model enumeration technique [ 29 , 28 ] that searches for explanation models maximizing the fidelity while minimizing the unfairness for a given unfair black-box model . In this study , we go a step further by determining the fidelity-unfairness trade-offs of the fairwashing attack and characterizing the manipulability of the fairwashed explanations . For this purpose , we compute the set of Pareto optimal explanation models describing all the achievable fidelity-unfairness trade-offs by solving the following problem : minimize E ( xi , b ( xi ) ) ⇠Dsg [ l ( e ( xi ) , b ( xi ) ) ] , subject to unfDsg ( e ) ✏ , ( 1 ) in which e is the explanation model , l ( e ( x ) , b ( x ) ) is the loss function ( e.g. , cross entropy ) , Dsg = { Xsg , b ( Xsg ) } is formed by the suing group and the prediction of the black-box model b on the suing group , and ✏ is the value of the unfairness constraint .

Summary: This paper attempts to empirically characterize the risk of "fair washing" attacks, in which an unfair model can be described with high fidelity by a global explainer that exhibits substantially less unfairness. The authors determine the fidelity-unfairness Pareto curve for several black box models, explanation models, datasets, and fairness criteria and find that across all tested groups, it is generally possible to reduce apparent unfairness by 50% with a minimal reduction in fidelity to the black box model. The authors additionally show that fair washing attacks trained on a specific subgroup (the suing group) extrapolate well to an unseen test group and that fairwashed explanations trained on one black box model may still have high fidelity on other black box models. Finally, the authors determined the range of unfairness metric values that can be achieved at various levels of fidelity to the black box in order to quantify how susceptible different black box-dataset-explainer combinations may be to fair washing.