Characterizing the risk of fairwashing
Authors: Ulrich Aïvodji
Hiromi Arai
Sébastien Gambs
Satoshi Hara
Venue: NeurIPS 2021 Poster (2021)
Keywords: fairwashing
post-hoc explanation
fairness

Abstract: Fairwashing refers to the risk that an unfair black-box model can be explained by a fairer model through post-hoc explanation manipulation. In this paper, we investigate the capability of fairwashing attacks by analyzing their fidelity-unfairness trade-offs. In particular, we show that fairwashed explanation models can generalize beyond the suing group (i.e., data points that are being explained), meaning that a fairwashed explainer can be used to rationalize subsequent unfair decisions of a black-box model. We also demonstrate that fairwashing attacks can transfer across black-box models, meaning that other black-box models can perform fairwashing without explicitly using their predictions. This generalization and transferability of fairwashing attacks imply that their detection will be difficult in practice. Finally, we propose an approach to quantify the risk of fairwashing, which is based on the computation of the range of the unfairness of high-fidelity explainers.

Characterizing the risk of fairwashing
UIrich Aïvodji
Hiromi Arai
École de Technologie Supéricure ulrich aivodj feeejitettsttt...
RIKEN Center for Advanced Intelligence Project hiromi. arai @riken. jp
Sébastien Gambs Université du Québec à Montréal gambs. sebastieaeeeee ennbuat.. ca
Satoshi Hara
Osaka University .aanaan
satchara@ar sanken. osaka-u.ac.
Abstract
Fairwashing refers the risk that an unfair black-box model can be explained by a fairer model through post-hoc explanation manipulation. In this paper, we inves- tigate the capability of fairwashing attacks by analyzing their fidelity-unfairness trade-offs. In particular, we show that fairwashed explanation models can general- ize beyond the suing group (i.e., data points that are being explained), meaning that fairwashed explainer can be used to rationalize subsequent unfair decisions of a black-box model. We also demonstrate that fairwashing attacks can transfer across black-box models, meaning that other black-box models can perform fairwashing without explicitly using their predictions. This generalization and transferability fairwashing attacks imply that their detection will be difficult in practice. Finally, we propose an approach to quantify the risk of fairwashing, which is based on the computation of the range of the unfairness of high-fidelity explainers.
Introduction
As machine learning models are increasingly integrated into the pipeline of high-stakes decision processes, concerns about their transparency are becoming prominent and difficult to ignore for the actors deploying them. As result, post-hoc explanation techniques have recently gained popularity as they may appear as potentially viable solution to regain trust in machine learning models' predictions. More precisely, post-hoc explanation techniques refer to methods used explain how black-box ML models produce their outcomes 266] 9]. Current existing techniques for post-hoc explanations include global and local explanations. In nutshell, global explanations focus explaining the whole logic of the black-box model by training a surrogate model that interpretable by design (e.g., linear models, rule-based models decision trees) while maximizing its fidelity the black-box model. In contrast, local explanations aim at explaining a single decision by approximating the black-box model in the vicinity of the input point through an interpretable model.
However, a growing body of works has recently shown that post-hoe explanation techniques not only can be misleading 38 46) but are also vulnerable to adversarial manipulations, wherein adversary misleads users' trust by devising deceiving explanations. This phenomenon has been demonstrated for broad range of post-hoc explanation techniques, including global and local explanations | 49], example-batt based explanations
[25]. visualization based explanations [T 200 and counterfactual explanations [36]]. For instance, in fairwashing attack [4] the adversary manipulates the explanations to under-report the unfairness of the black-box models being explained. This attack significantly impact individuals who have received negative outcome following the model's
prediction while depriving them of the possibility of contesting it.
Work done while at Université du Québec Montréal
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

The fundamental question regarding fairwashing attacks is their manipulability, which we define as the ability to maximize the fidelity of an explanation model under an unfairness constraint. The manipulability of fairwashing attacks directly impacts the possibility to detect them. Indeed, if the manipulability is so low that the explanation manipulation can be detected, the risk of fairwashing is small and misleading decisions can be avoided. By contrast, if the manipulability is high enough the manipulation is undetectable, we will be under threat of the use of unfair models whose unfairness is hidden by malicious model producers through manipulated explanations. In this work, in the context of fairwashing for global explanations, we provide the first empirical results demonstrating that the manipulability of fairwashing is likely to be high. To assess the manipulability of fairwashing, trade-off and evaluated two characteristics of fairwashing, namely generalization and transferability.
we used the
Generalization of fairwashing beyond the suing group. In a fairwashing attack, a manipulated explanation is tailored specifically for a suing group of interest so that the explanation is fair within this group. As the explanation is specific to that group, we hypothesize that the same explanation can fail for another group. Based on this hypothesis, we assess the manipulability of fairwashing through its generalization capability. Our results suggest that the fidelity of the fairwashed explanation evaluated on another group comparable to the one evaluated the suing group. This means that the above hypothesis is negative, in the sense that explanations built to fairwash a suing group can generalize to another group not explicitly targeted by the attack.
Transferability of fairwashing beyond the targeted model. In the fairwashing attack, a manipu- lated explanation is targeted specifically for the deployed black-box model. However, in practical machine learning, is usually the case that the deployed model is updated frequently. Thus, we hypothesize that there can be an inconsistency between the manipulated explanations provided to the suing group in the past and the currently deployed model. Based on this hypothesis, we quantify the manipulability of fairwashing through its transferabilitty Our results suggest that the fidelity of the fairwashed explanation evaluated on another model is comparable to the one evaluated on the deployed black-box model. Thus, the above hypothesis is also negative as fairwashed explanations designed for a specific model can also transfer to another model.
Implications to undetectability. We observed the generalization and transferability of fairwashing attacks on several datasets, black-box models, explanation models and faimesss criteria. As a consequence, our results indicate that detecting manipulated explanations based on the change of fidelity alone is not a viable solution (or at least it is very difficult).
Another way of quantifying fairwashing manipulability. In the above experiments, the manipu- lability of fairwashing was evaluated using the fidelity of explanation and its changes. Our negative results suggest that fidelity alone may not be an effective metric for quantifying the manipulability of fairwashing. Thus, we further investigated different way of quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set (FaiRS) [17] framework. Our results indicate that this framework can be effectively used to quantify the manipulability of fairwashing.
Related work. In the context of example- based explanations' manipulation, Fukuchi et al. (5 have demonstrated the risk of stealthily biased sampling, which occurs when model producer explains the behaviour of its black- box model by sampling a subset of its training dataset. Slack et al. [49] have shown that variants of LIME 45 and SHAP (39] two popular post-hoe local explanation techniques, can be manipulated to underestimate the unfairness of black-box models. Following the same line of work, Le Merrer and Trédan (37 have demonstrated that a malicious model producer can always craft a fake local explanation to hide the use of discriminatory features. Another work by Laugel al. [36] has focused explicitly on the use of counterfactual explanations, which is form of example- explanation. They have demonstrated that generated instances can be unjustified (i.e., not supported by ground-trutt -tuth data) for most techniques existing for such type of explanation. Visualization based explanation techniques can also be manipulated, as shown by recent works on saliency map-based explanations' manipulation. Heo et al. [31) have demonstrated that these types of explanations are vulnerable to the so- so-called adversarial model manipulation while Dombrowski et al. (20) have shown their vulnerability to adversarial input manipulation. In our previous work (4, we introduced the notion of fairwashing as rationalization exercise. We devised LaundryML, an algorithm that can systematically rationalize black-box models' decisions through global or local explanations. The details of this method, which form the basis of the fairwashing attacks in this study,

---

are given in Section 2.2 A subsequent work (35) has also investigated the possibility that black-box models can be explained with high fidelity by global interpretable models whose features are very different from that of the black-box and look innocuous. Finally, several negative aspects of post-hoe explanations have been reported in recent studies. For instance, post-hoc explanations can mislead model designers when they are used for debugging purposes [ or can be leveraged to perform powerful model stealing attacks [42]
Outline. In Section? 2 we review the preliminary notions necessary understand this work, and we describe how to explore the fidelity-unfairness trade-offs of a fairwashing attack by using an e-constraint method solve its underlying multi-objective optimization problem. In Section 3 we present the results obtained from our study for diverse set of datasets, black-box models, explanation models and fairness metrics. In Section 4 we consider quantifying the manipulability of fairwashing using the Fairness In The Rashomon Set (FaiRS) [77 framework. Finally, we discuss the main implications of our findings in Section as well as its limitations and societal impact in Section 6
2 Setting and problem formulation
2.1 Notations
Let X E X' C Rn denote feature vector, Y = {0,1} its associated binary label (for simplicity
we assume a binary classification setup without loss of generalization) and G g {0. 1} a feature defining group membership (e.g., with respect to sensitive attribute) for every data point sampled
from X. In addition, we assume that b : X + y refers to a black-box classifier of a particular model class B (e.g., neural network or ensemble model) mapping any input X X to its associated prediction Y V. Finally, let X' + )' be global explanation model from particular model class (e.g.. linear model, rule list or decision tree) designed to explain b. In the context of fairwashing attacks, which we formalize in Definition 2
we refer to the data instances on which the attack is
performed as the suing group.
In this work, we measure the unfairness wff) of a model on dataset 'D by using statistical
notions of fairnsss [13 15 16 30), which require a model to exhibit approximate parity according to
statistical measure across the different groups defined by the group membership G. In particular,
we consider four different statistical notions of fairness, namely statistical parity 12
13
33 23, [T].
predictive equality [[3 [6),, equal opportunity [30] and equalized odds 11
30
50
The
definitions of these fairness metrics are listed in Table
Table 1: Summary of the different statistical notions of fairness considered.
Fairness notion
Definition
Statistical Parity (Asp)
IP(Ý 1|G 0) P(ý = 1)]
Predictive Equality (Apg)
[P(Ý = 1|Y = =)) - -P- = 1|Y = 0,G 1)
Equal Opportunity (AεOрp)
IP(Ý = 1- 1=, -P(Ý'
= 1,G 1)]
IP(Ý =
-P(Y' =1)==
Equalized Odds (Agodds)
and LP(Y = 1|1 = O, = 0) p( = 12 = 0, 1)
2.2 Problem formulation
Our investigation is motivated by our previous work 44 in which we defined fairwashing in global and local explanations as manipulation exercise in which high-fidelity and fairer explanations can be designed to explain unfair black-box models.
Definition (Global explanation fidelity) Let b be a black-box model, e a global explanation model for b, and X a set of data instances. Following the definition in [T8), the fidelity of e with respect to b on X is expressed as:
fidelity(e)
Σ ZEX
[(e(x) b(x)).
X
3

---

Definition 2 (Global fairwashing attack) Let b be black-box model and Xeg a set of data in- stances hereafter referved to as suing group. global fairwashing attack consists in finding an interpretable global model = p(b, Xxg) derived from the black-box b and the suing group Xag using some attack process :): such that e. is fairer than for given fairness metric.
To realize this, in our previous work 1 we devised LaundryMI, an algorithm that can system- atically fairwash unfair black-box models' decisions through both global and local explanations. LaundryML is constrained model enumeration technique [29] [28] that searches for explanation models maximizing the fidelity while minimizing the unfairness for given unfair black-box model.
In this study, we go a step further by determining the fidelity-unfaimen trade-offs of the fairwashing attack and characterizing the manipulability of the fairwashed explanations. For this purpose, compute the set of Pareto optimal explanation models describing all the achievable fidelity-unfainen trade-offs by solving the following problem:
minimize ((x,,b(r))) [((((()) b(x+)),, subject to unf (e) S
(1)
in which is the explanation model, (e((z), b(z)) is the loss function (e.g.. cross entropy), Day {Xags b(Xog)} is formed by the suing group and the prediction of the black-box model on the suing
group, and e is the value of the unfairness constraint.
Experimental evaluation
In this section, we evaluate the manipulability of the fairwashing attack over several datasets, black- box models, explanation models, and fairness metrics. We start by replicating the results of our previous study [4], which demonstrate that explanations can be fairwashed. Then, we evaluate two fundamental properties of fairwashing, namely generalization and transferability]
Datasets. We have investigated four real-world datasets commonly used in the faimess literature, namely Adult Income, Marketing, COMPAS, and Default Credit. Table summarizes the main characteristics of these datasets.
Adult Income. The UCI Adult Income (1 dataset contains demographic information about 48,842 individuals from the 1994 U.S. census. The associated classification task consists in predicting whether a particular individual earns more than 50,000$ per year. We used gender (Female, Male) as group membership for investigating fairness.
Bank Marketing. The UCI Bank Marketing 143 dataset contains information about 41,175 customers of a Portuguese banking institution contacted as part of a marketing campaign (i.e., phone calls), whose goal was to convince them to subscribe to a term deposit. The classification task consists of predicting who will subscribe to term deposit. We used age (30-60, not30-60) as group membership.
COMPAS. The COMPAS 8 dataset gathers 6,150 records from criminal offenders in Florida during 2013 and 2014. Here, the classification task consists in inferring who will re-offend within two years. We used race (African-Amete Amaricannnnn Caucasian) as group membership.
Default Credit. The Default Credit [21] dataset is composed of information of 29,986 Taiwanese credit card users. The classification task is to predict whether a user will default in its payments. We used sex (Female, Male) as group membership.
Preprocessing. Before the experiments, each dataset is split into three subsets, namely the training set (67%). the suing group (16.5%) and the test set (16.5%). We created 10 different samplings of the three subsets using different random seeds and averaged the results of over these 10 samples. The training set is used to learn the black-box models, while the suing group dataset is used to prepare the explanation models as well evaluate their fidelity-unfairnes trade-offs. Finally, the objective of the test set is to estimate the accuracy of the black-box models as well as the generalization of the explanation models beyond their suing groups. For all models (i.e., black-boxes and explanation models), we used one-hot encoding of the features of the datasets.
Our implementations are available at httpiilleltttps/gggtubooo//////e htt.::///ggtuuu..ooo////////eeer

---

Table 2: Summary of the datasets used. N, R, Roke and n,- denote respectively the numbers of data points, the number of features, the number of one-hot encoded features and the number of rules.
Dataset
Category
N
f
Noke
f
Adult Income
Finance
48,842
11
40
177
Marketing
Commerce
41,175
20
61
178
COMPAS
Justice
6,150
8
11
121
Default Credit
Finance
29,986
23
35
188
Black-box models. We have trained four different types of black-box models on each dataset, namely an AdaBoost classifier [24], a Deep Neural Network (DNN), a Random Forest (RF) [12) and XgBoost classifier [14). To tune the hyperparameters of these models, during their training, performed a hyperparameter search with 25 iterations using HyperOpt [10]. The performances (i.e. accuracy and unfairness) of the four black-box models on the suing group are provided in Table 3
Table 3: The performances of the black-box models evaluated on the suing group, Each cell is of the form
Asp
APE
accuracy
Agodds.
AdaBoost
DNN
RF
XgBoost
.17 .06
.16
.06
.16 .06
.17 .06
Adult Incomde
0.85
0.85
0.86
0.86
.12 .12
.06
.07
.09
.09
.09 .09
.10
.04
.09
.04
.10 .04
.10 .04
Marketing
0.91
0.91
0.91
0.91
.13 .13
.09
.09
.04
.06
.08 .09
.24
.25
.28
.31
.26
.28
.27 .30
COMPAS
0.68
0.68
0.67
0.68
.14 .25
.18
.31
.16 .28
.18 .30
.03
.04
.03
.04
.03
.03
.02
.02
Default Credit
0.80
0.81
0.81
0.81
.01
.04
.01
.04
.01
.03
.01
.03
Fairwashed explanation models. We solved the optimization problem defined in Equation for three model classes, namely logistic regression, rule lists and decision trees. For rule lists, we used FairCORELS [5]. modified version of CORELS [7]. which trains rule lists under faimess constraints. For both logistic regression and decision trees, we used the exponentiated gradient technique (3]. which is model agnostic technique to train any classifier under fairness constraints
3.1 Experiment 1: Replicating the result of Aïvodji et al.
We first demonstrate that the explanations can be fairwashed by replicating the results of our prior study 4 over a wide range of datasets, black-box models, explanation models and faimesss metrics.
Setup. Given suing group Xeg. for each black-box model b and each fairness metric m, the Pareto fronts are obtained by sweeping over 300 values of faimnsss constraints E [0, 1]. For each value of 6ns an explanation model ec.. trained to satisfy the unfairness constraint 6n on X by solving the problem in Equation [T for logistic regression, rule list and decision tree. Then, the effective unfairness and fidelity (with respect to b) on Xag are obtained. Finally, the set of non-dominated points is computed.
Results Top rows in Figure 1 show the fidelity-unfainen trade-off's of fairwashed logistic regres- sion explainers found for the four black-box models, respectively on Adult Income and COMPAS, for the suing group. using four different fairness metrics. Figure 2 displays the fidelity of the fairwashed logistic regression explainers when they are designed to be at least 50% less unfair than the black-box models they are explaining. Results are shown for all datasets, fairness metrics and three families
We used the Fairlearn lihrary [T] implementation: http:///ggsttaa.ecom////errrrrrs
'See Appendix C for the results of all the datasets and models.
5

---

of black-box models (AdaBoost, DNN and RF). Consistently over all these results, we observe that the fairwashed explanation models found for the suing groups were significantly less unfair than the black-box models while maintaining high fidelity (i.e., the fairwashing attacks were effective as shown in Figure D.. More precisely, for any combination of fairness metric m and black-box model b, a fairwashed logistic regression displays unfairness less than 50% of the unfairness of b while maintaining a fidelity greater than 92%. 96%, 81% and 90% respectively for Adult Income, Marketing, COMPAS and Default Credit(c.f. Figure Furthermore, the small change in percentage of the fidelity of the fairwashed explanation models indicates that fairwashing does not introduce an important loss in fidelity when compared to non-fairwashed explanations models.
Adult Income
Logistic Regression
Model
AdaBoost
DNN
RF
XgBocet
EOdds
EOpp
SP
0.94
0.90
Fidality
0.96
0.94
JPS T99l
0.92
0.88
0.05
0.10
0.00
0.05
0.10
0.00
0.02
0.04
0.06
0.00
0.05
0.10
0.16
Unfairness
COMPAS Logistic Regression
Model AdaBoost
DNN
RF XyBoost
ECods
EOpp
0.9
1+
Staing Group
0.8
0.7
Fidellyy
0.8
0.7
0.8
0.1
0.2
0.3
0.05
0.10
0.15
0.20
0.0
0.1
0.2
0.3
0.1
0.2
Unfairesss
Figure Fndelity-unfairness trade-off of fairwashing attacks for equalized odds (Agodds), equal opportunity (AgOpp). predictive equality (Arg) and statistical parity (Asr) metrics on Adult Income and COMPAS datasets, using logistic regression explanation models. Vertical lines denote the unfairness of the black-box models. Results are averaged over 10 fairwashing attacks. The standard deviations are shown as shaded regions.
3.2 Experiment 2: Generalization of f'airwashing beyond suing groups
We now assess the manipulability of fairwashing attacks in terms of generalization. As a fairwashed explanation tailored specifically for a suing group of interest, could be the case that the same explanation can fail for another group. Our results detailed below suggest that this is not the case. That is, the above hypothesis negative as explanations built to fairwash suing group can generalize to another group not explicitly targeted by the fairwashing attack.

---

Logistic Regression
AdeBoost
DNN
RF
Markaling
97.77
97.74
97.71
97.17
97.26
97.26
97.3
96.81
97:03
96.95
97.09
96.29
Fidellly
Datanets
Default Crodit
96.05
94.75
93.51
91.
51.52
B2 90
93.11
92.27
92.45
91.65
10.79
COMPAS
93:92
81.51
83
92
02.76
92 92.91
81 81.25
0.08
063
0.66
-236
94. .4
92.95
95.51
95
95.57
ECd
EDsp
EOcls
ECdda: EOpp
Metrics
Figure 2: Fidelity of the fairwashed logistic regression explainers that are 50% less unfair than the black-box models they are explaining. Results (averaged over 10 fairwashing attacks) are shown for AdaBoost, DNN and RF black-box models, for all datasets and fairness metrics. The content of each cell is in the form of 2 in which represents the fidelity of the fairwashed explanation model, and y its percentage change with respect to the fidelity of the unconstrained explainer, used here as a baseline.
Sctup. We used the same experimental setup in Experiment 1. However, the unfairness and fidelity of the explanation model are computed on the test set Xtest such that X O Xtest = 0. As Xteat not disclosed to the model producer, the fairwashed explanation may fail to generalize on Xteat and exhibit unfairness.
Results 5 Bottom rows in Figure 1 show the fidelity-unfairnee trade-offs of fairwashed logistic regression explainers found for the four black-box models on Adult Income and COMPAS, on the test set, using four different faimess metrics. Overall, the results show that the explanation models designed for a particular suing group generalize well also to the test set by achieving similar fidelity-uffarrness trade-offs.
Implications to undetectability. One might try to detect manipulated explanations by preparing second suing group unknown to the model producer, with the expectation that the manipulated explanations will fail for that second group. However, the subtle gaps between the trade-offs of the suing group Xag and the test set Xteat suggest that this would not be convincing evidence of the occurrence of fairwashing. More precisely, only a fraction cases exhibiting considerable drops in fidelity could be detected, while the majority of manipulated explanations will go unnoticed.
In addition, the subtle gaps could be made even smaller by adopting robust sermess-enhancing techniques (e.g., [40).. The fairwashing attack defined in Equation i equivalent to problem of training an explanation model under fairness constraint, in which the training pair (X, Y)) formed by the suing group ag and the predictions b( (X.,, of the black-box b. As a result, the issue of generalizing heyond the suing group can be reduced to the problem of generalizing fairness beyond the training set. Thus, robust fairness-enhancing techniques could potentially enable malicious model producers to obtain explanation models with better generalizations and smaller gaps.
3.3 Experiment 3: Transferability of fairwashing beyond the targeted model
In this section, we assess the manipulability of the fairwashing attack in terms of transferability. In practical machine learning, is usually the case that the deployed model is updated frequently. Thus, an inconsistency could occur between the manipulated explanations generated before the model update and the currently deployed model. Our results below suggest that this is not the case, in the sense that the fairwashed explanation for specific model can transfer to another model.
Setup. Given a suing group Xag. a teacher black-box model breacher (corresponding to an old model), fairness metric m, its associated fairness constraint em and set of student black-box models bi b'rudents with i = I,..., ...,, (corresponding to updated models), an explanation model e...
SSee Appendix C for the results of all the datasets and models.
7

---

is trained to satisfy the unfairness constraint 6n on Xzg. by solving the problem in Equation f logistic regression, rule lists and decision trees. Afterwards, the unfairness and fidelity of e are evaluated with respect to each of the student black-box models ii sttudent on X xxg. For this experiment, we considered four black-box models (AdaBoost, DNN, RF and XgBoost). First, we fixed one model as the teacher model and used the remaining ones as student models. We conducted the experiments for all four possible combinations of the (teacher, student) models. We evaluated the results on a number of unfairness constraints (e {0.03, 0.05, 0.13) to simulate both strong and loose fairness constraints.
Results F Figure 3 displays the fidelity and unfairness of fairwashed logistic regression explainers with respect to both the teacher and the set of student black-box models on the suing group Xag- Results are shown for Adult Income and COMPAS, for all fairness metrics and different values of the unfairness constraint (c 0.05). Overall, our results demonstrate that fairwashed explanations can generalize well to student black-box models by displaying high fidelity in many cases. For instance, on Adult Income with a predictive equality constraint set to 0.05, fairwashed logistic regression explainer that had a fidelity of 95% for a DNN teacher model successfully transferred to AdaBoost, RF and XgBoost student models with fidelity of respectively 94%, 95% and 93%.
Implications to undetectability. If the hypothesis that manipulated explanations are vulnerable to model updates true, one would expect that the manipulated explanations generated before the model update exhibit an inconsistency with the updated model. However, we observed that high-fidelity explanations can be transferred to another model in many situations. Hence, the change in fidelity after the model update would not be reliable evidence for detecting fairwashing attacks as it will overlook several manipulated explanations with small drop of fidelity.
Adult Income -- Logistic Regression eps=0.05
AM
AM
100
CR:
XiBoooe
0.94
0.95
0.95
0.94
0.95
0.95
0.94
0.94
0.95
0.95
0.94
0.91
0.92
0.92
0.91
Fidelity
Teacher model
1M
KM
0M
8M
KM
mT
0.94
0.96
0.94
0.98
0.93
0.94
0.95
0.96
0.93
0.91
0.92
0.92
0.9
1/0
403
48
AD
60
500
0.93
0.94
0.95
0.83
0.94
0.91
0.92
6:0
0.1
0.1
Adafooot
0.95
0.95
0.95
0.95
0.94
0.95
0.93
0.92
0.92
0.92
0.9
AdaBoos: DNN
AF
XgBoos: AdaBooss DNN
PF
XaBoost
AduBooot DAN RF
XyBcoot AdsBocst DNN
RF
XgBoost
Student model
COMPAS -- Logistic Regression eps=0.05
XgBoost
0.93
0.98
0.93
0.98
0.73
0.78
Fidelity
634
0.93
0.92
0.92
0.92
0.93
0.92
0.93
0.92
0.75
0.74
Teacher
0.05
65
DNN
0.94
0.91
0.94
0.91
0.91
0.94 k0
0.74
1/2
4.
076
0.94
0.82
0.92
0.94
0.92
0.82
0.94
0.92
0.92
0.92
0.74
0.73
AdaBoost
DNN
XgBooss
AduBoost
XyBoors
AduBcoot
XyBoooi AduBoost
MyBooot
Student model
Figure 3: Analysis of the transferability of fairwashing attacks for equalized odds, equal opportunity, predictive parity and statistical parity on Adult Income and COMPAS datasets, for e = 0.05, for fairwashed logistic regression explainers. The result in each cell is in the form of 2. in which denotes the label agreement between the teacher black-box model and the student black-box model, is the fidelity of the fairwashed explanation model and is its unfairness. Blank cells denotes the absence of transferability for the unfairness constraint imposed. Results are averaged over 10 fairwashing attacks.
SSee Appendix for the results of all the datasets and models.
8

---

4 Another way of quantifying the manipulability of fairwashing
Our experimental results in the previous section revealed that fairwashed explanations can generalize to unseen suing groups and can transfer to other black-box models with high fidelity. This suggests that fidelity alone may not be an effective metric for quantifying the manipulability of fairwashing. In this section, we consider another way of quantifying the manipulability of fairwashing using the Fairness The Rashomon Set (FaiRS) [17] framework.
At a high level, one can say that fairwashing is possible because the set of all the explanations models for particular black-box model (at a particular level of fidelity) is diverse in terms of unfairness (i.e., contains both fair and unfair explanation models). This phenomenon has been observed for a broad range of models and domains, and received different names including predictive multiplicity [4]. underspecification [9] and multiplicity in the Rashomon set [48] 27 47].
Given model class F, loss function (D(.) over a dataset D of interest, reference model f (e.g.. optimal explanation model with at least 95% fidelity), and a performance threshold T E [0, 1]. the Rashomon set R. (F. f",
f 7) is defined as
(", (S ( S (1)))((1++()) (1+T)
(2)
We quantify the manipulabilityy of fairwashing by seeking the models with the lowest unfairness within the Rashomon set. More precisely, if the Rashomon set contains explanation models with sufficiently low unfairness, then the risk of fairwashing is high high-fidelity explanation models with low unfaimess. Conversely, if all the models in the Rashomon set have unfairness close to that of the black-box model, the fairwashing will fail as it is not possible to find high-fidelity explanation models with an unfairness significantly lower than that of the black-box model.
can adopt such a model as
A possible way to identify the model with the lowest unfairness in the Rashomon set is to use the method of Coston et al. [17], which computes the range of the unfairness of high-fidelity explanation models by solving the following problem:
minimize unf unf (e), subject to LD.,, (e) S
in which is the explanation model, Day (X.gs b(X,g)) is formed by the suing group and the prediction of the black-box model b on the suing group, while v is the value of the constraint on the loss explore different levels of fidelity.
Setup. In this experiment, we explore different values for the fidelity ranging from 70% to 98%. In particular, solve the problem of minimizing the disparity (and not the absolute disparity) under constraints of the loss [17] Problem 2] using the reduction approach of Agarwal et al. [3] to explore
the range of unfairness for fixed value of the loss. We used this approach and swept over different values of the loss function to compute the range of unfairness of high-fidelity explanation models] [
Results g Figure 4 shows that logistic regression explainers Adult Income have a wider range of
unfairness in the Rashomon set compared to that of COMPAS. This result implies that the fairwashing attack has a higher manipulability on Adult Income. This observation is coherent with the results observed in Experiments and 2 (c.f., Figures [nd in which, although fairwashing is possible for both datasets, it is casier (i.e., in terms of the possibility to have high-fidelity with low unfairness) to perform fairwashing on Adult Income than COMPAS.
5 Conclusion
In this paper, we have investigated the manipulability of fairwashing attacks by analyzing their fidelity- unfaimess trade-off's in diverse situations. In particular, we have demonstrated that fairwashing attacks have high manipulability. Furthermore, we showed that fairwashed explanations can generalize to unseen suing groups and can transfer across black-box models by displaying a high fidelity. The lesson draw from our investigation is that relying on the fidelity alone and its changes as proxies
We used the implementation provided by the authors in the public Github repository available at pattps //github. com/
In_ The Rashomon, [estt "See Appendix E for the results of all the datasets and black-box models.

---

Adult Income - DNN
COMPAS DNN
02
Unbanios
-33
Figure 4: Range of the statistical parity of logistic regression explainers for different values of the fidelity for DNN black-box trained on Adult Income and COMPAS. Horizontal lines denote the unfairness of the black-box models.
for the quality of post-hoc explanation can be misleading as fairwashed explanation model can exhibit high fidelity even to unseen suing groups and other black-box models while being significantly less unfair than the black-box model being explained. Our first result obtained for the generalization of fairwashed explanations beyond suing groups demonstrates that fairwashed explanation model can also rationalize subsequent unfair decisions made by the original black-box model for free. This precludes the possibility of designing fairwashing detection techniques that leverage the instability of the unfairness with respect to variations in the suing group. Indeed, such a technique will most likely fail against fairwashed explanation models designed using stable fair classification algorithms [32 [Q]. Our second result, the transferability of fairwashed explanations across black- black-box models, revealed that model producers can use fairwashed explanation models to rationalize unfair decisions of future black-box models. Since fidelity alone did not prove to be effective measure, we investigated another possible way to quantify the manipulability of fairwashing. To this end, we used the Fairness In The Rashomon Set (FaiRS) framework of Coston et al. [I7] to compute the range of the unfairness of high-fides fidelity explainers, and observed that it is possible to quantify the manipulability of fairwashing by using this framework.
Future work. In this study, we focused on statistical notions of fairness defined over binary output and binary attributes. These definitions of faimess be extended continuous output (e.g., predicted class probability) these extended problems remains open. We also hypothesize that one may be able design meta-classifier for detecting fairwashing. Despite the fact that the fidelity of fairwashed explanations considerably high, it may be possible that some implicit patterns specific fairwashing could be detected and exploited (e.g., the fidelity can be low on specific subgroups). Thus, if there are several pairs of fairwashed/honest explanations, one might be able train a meta-classifier to distinguish between fairwashed and non-fairwashed explainers by secking such patterns.
However, investigating the manipulability of fairwashing attacks
6 Limitations and societal impact
Limitations. In this paper, fidelity is defined as the lahel agreement between the black-box and explanation models. This definition was first introduced in [18] and is still the popular quality measure used in global explanation techniques. While there are several criticisms regarding the use fidelity quality measure, we believe that considering its popularity in the community, it still remains reasonable choice to raise awareness about the risk of fairwashing.
Societal impact. One of the objectives of this paper is to raise awareness about the risks that can occur when post-hoc explanations are used to assess fairness claims. As such. aims at averting the potential negative societal impacts of such assessments. Nonetheless, it is possible that malicious model producers could use the manipulation attacks presented in this paper to perform fairwashing in believe that
the real world. However, in addition to increasing the vigilance of the community, this paper makes a significant step towards detecting and preventing such fairwashing attacks.
10

---

Acknowledgments
We thank the anonymous NeurIPS reviewers for their insightful feedback. Hiromi Arai is supported by JST, PRESTO Grant Number JPMJPR 1752, Japan. Satoshi Hara is supported by JSPS KAKENHI Grant Number 20K 19860, and JST, PRESTO Grant Number JPMJPR20C8, Japan. Sébastien Gambs is supported by the Canada Research Chair program, a Discovery Grant from NSERC, the Legalia project from the AUDACE program funded by the FQRNT, the project Privacy and Ethics: Understanding the Convergences and Tensions for the Responsible Development of Machine Learning funded by the Office of the Privacy Commissioner of Canada (OPC) as well as the NSERC-RDA -DDC DEEL project. The opinions expressed in this paper only the one of the authors and do not necessarily reflect those of the OPC.
References
[1] J. Adebayo, J. Gilmer, M. Muelly. I. Goodfellow. M. Hardt, and B. Kim. Sanity checks for saliency maps. In Advances Neural Information Processing Systems, pages 9505-9515. 2018.
[2] Adebayo, M. Muelly, I. Liccardi, and Kim. Debugging tests for model explanations. In Advances in Neural Information Processing Systems, 2020.
[3] A. Agarwal, A. Beygelzimer. M. Dudik, J. Langford, and H. Wallach. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning, pages 60-69, 2018.
[4] U. Aïvodji, H. Arai, O. Fortineau, S. Gambs, S. Hara, and A. Tapp. Fairwashing: the risk of rationalization. In Proceedings of the 36th International Conference on Machine Learning. pages 161-170, 2019.
[5] U. Aïvodji, J. Ferry, S. Gambs, M.-. Huguet, and M. Siala. Learning fair rule lists. arXiv preprint arXiv:
2019.
[6] U. Aivodji, A. Bolot, and S. Gambs. Model extraction from counterfactual explanations. arXiv preprint w00.i:20009008880 2020.
[7] E. Angelino, N. Larus-Stone, D. Alabi, M. Seltzer, and C. Rudin. Learning certifiably optimal rule lists. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 35 44, 2017.
[8] J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias. ProPublica, May, 23, 2016.
[9] A. B. Arrieta, N. Díaz Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, R. Chatila, and F. Herrerag. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 50:82-115, 2020.
[10] J. Bergstra, D. Yamins, and D. Cox. Making a science of model search:
Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, pages 115-123, 2013.
[11] S. Bird, M. Dudík, R. Edgar, B. Horn, R. Lutz, V. Milan, M. Sameki, H. Wallach, and K. Walker. Fairlearn: A toolkit for assessing and improving fairness in ai. Microsoft Technical Report 2020.
[12] L. Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
[13] T. Calders and S. Verwer. Three naive bayes approaches for discrimination-fi -rreeiiioo-oeeeeeeee classification.
Data Mining and Knowledge Discovery, 21 2010.
[14] Chen and C. Guestrin. Xgboost: scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 785-794, 2016.
[15] A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2):153-163, 2017.
[16] S. Corbett-Davies, H. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 797-806, 2017.

---

[17] A. Coston, A. Rambachan, and A. Chouldechova, Characterizing faimesss over the set of good models under selective labels. In Proceedings the 38th International Conference on Machine Learning, pages 2144-21555 2021.
[18] M. Craven and J. W. Shavlik. Extracting tree-structured representations of trained networks. In Advances in Neural Information Processing Systems, pages 24-30, 1996.
[19] A. D'Amour, K. Heller, D. Moldovan, B. Adlam, B. Alipanahi, A. Beutel, C. Chen, J. Deaton, J. Eisenstein, M. D. Hoffmann, et al. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:201 2020.
[20] A.-K. Dombrowski, M. Alber, C. Anders, M. Ackermann, K.-R. Müller, and P. Kessel. Ex- planations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, pages 25589-13600, 2019.
[21] D. Dua and C. Graff. UCI machine learning repository. 2017. URL http::/arrcive..ikt uci edu/ml
[22] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, and R. Zemel. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pages 214-226, 2012.
[23] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, and S. Venkatasubramanian Certifying and removing disparate impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 259-268, 2015.
[24] Y. Freund and R. E. Schapire. A application to boosting. Journal of Computer and System Sciences, 55(1):119-135 1997.
generalization of on-line learning and an
[25] K. Fukuchi, S. Hara, and T. Machara. Faking fairness via stealthily biased sampling. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 419,419, 2020.
[26] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, Giannotti, and D. Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5):93, 2019.
[27] L. Hancox-Li. Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 Conference on Fairness, Accountability, and Transpurency, pages 640-647, 2020.
[28] S. Hara and M. Ishihata. Approximate and exact enumeration of rule models. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 3157-3164, 2018
[29] S. Hara and T. Maehara. Enumerate Lasso solutions for feature selection. In Proceedings of the st AAAI Conference on Artificial Intelligence, pages 1985-1991, 2017.
[30] M. Hardt, E. Price, N. Srebro, et al. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, pages 3315-3323. 2016.
[31] J. Heo, S. Joo, and T. Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems, pages 2925-2936, 2019.
[32] I. Huang and N. Vishnoi. Stable and fair classification. In Proceedings of the 36th International Conference Machine Learning, pages 2879-28999 2890, 2019.
[33] T. Kamishima, S. Akaho, H. Asoh, and J. Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pages 35-50, 2012.
[34] Kleinberg, Mullainathan, and M. Raghavan. Inherent trade-off's in the fair determination of risk scores. In Proceedings of the 8th Innovations in Theoretical Computer Science Conference, 2017.
[35] H. Lakkaraju and O. Bastani. "How do I fool you?" Manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, pages 79-85, 2020.
[36] T. LLugel. M.-J. Lesot, C. Marsala, X. Renard, and M. Detyniecki. The dangers of post-hoc interpretability: unjustified counterfactual explanations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2801-2807, 2019.
[37] E. Merrer and G. Trédan. Remote explainability faces the bouncer problem. Nature Machine Intelligence, 209):529-539, 2020.
12

---

[38] Z. C. Lipton. The mythos of model interpretability. Communications of the ACM, 61(10):36 43, 2018.
[39] S. M. Lundberg and S.-I. Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, pages 4765-4774, 2017.
[40] D. Mandal, S. Deng, S. Jana, J. Wing, and D. J. Hsu. Ensuring faimess beyond the training data.
In Advances in Neural Information Processing Systems, 2020.
[41] C. Marx, F. Calmon, and B. Ustun. Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning, pages 6765-6774, 2020.
[42] S. Milli, L. Schmidt, A. D. Dragan, and M. Hardt. Model reconstruction from model explana- tions. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 1-9, 2019.
[43] S. Moro, P. Cortez, and P. Rita. A data-driven approach to predict the success of bank telemar- keting. Decision Support Systems, 62:22-31, 2014.
[44] G. Pleiss, M. Raghavan, F. Wu, Kleinberg, and K. Q. Weinberger. On fairness and calibration. Advances in Neural Information Processing Systems, 1115680-5689, 2017.
[45] M. T. Ribeiro, S. Singh, and C. Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[46] C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-25 255:2,0-211,, 2019.
[47] C. Rudin, C. Chen, Z. Chen, H. Huang, L. Semenova, and C. Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. arXiv preprint arXiv:2 1251, 2021.
[48] L. Semenova, C. Rudin, and R. Parr. A study in rashomon curves and volumes: A new perspective on generalization and model simplicity machine learning. arXiv preprint 2019.
[49] D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju. Fooling LIME and SHAP: Adversarial attacks post hoc explanation methods. pages 180-186, 2020.
[50] M. B. Zafar, I. Valera, M. Gomez Rodriguez, and K. P. Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference World Wide Web, pages 1171-1180, 2017. 2017.
[51] I. Zliobaite. On the relation between accuracy and faimess in binary classification. arXiv preprint arXiv: 2015.
13


Reviews:
[{"review_id": "nux7Y0xaAb5", "reviewer": "Reviewer_Ga8q", "summary": "This paper attempts to empirically characterize the risk of \"fair washing\" attacks, in which an unfair model can be described with high fidelity by a global explainer that exhibits substantially less unfairness. The authors determine the fidelity-unfairness Pareto curve for several black box models, explanation models, datasets, and fairness criteria and find that across all tested groups, it is generally possible to reduce apparent unfairness by 50% with a minimal reduction in fidelity to the black box model. The authors additionally show that fair washing attacks trained on a specific subgroup (the suing group) extrapolate well to an unseen test group and that fairwashed explanations trained on one black box model may still have high fidelity on other black box models. Finally, the authors determined the range of unfairness metric values that can be achieved at various levels of fidelity to the black box in order to quantify how susceptible different black box-dataset-explainer combinations may be to fair washing. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and begins to address the important issue of identifying and preventing fair washing attacks. My main concern is that the analysis involved mostly involves repeating tasks in the original fair washing paper [Aivodji 2019] on a wider range of black box/explainer/dataset/metric combinations, without attempting to rigorously quantify the results and how they may relate to the characteristics of the black box/explainer/dataset/metric combination. \n\nIn particular, experiment 1 generally seems to repeat the primary result of [Aivodji 2019], but on a larger combination of black box/explainer/dataset/metrics. I would find this more interesting (and more reflective of the title \"characterizing\" the risk of fair washing) if it were possible to determine some properties or trends across these results: e.g. how and why are the results for COMPAS different than those for default credit? How and why are the results for LogReg different than those for rule lists?  \n\nAgain, experiment 2 repeats a result from [Aivodji 2019], but on a larger combination of black box/explainer/dataset/metrics.\n\nExperiment 3 is new relative to [Aivodji 2019], but I find the motivation rather unconvincing, and perhaps the result is unsurprising - if the signal in the dataset is simple, or if black box accuracy is high across several black boxes, it mostly seems natural that the explanation for one will be a reasonably good match for another, with some variation around decision boundaries.\n\nExperiment 4 again uses a method from another work [Coston 2021] in a very straightforward way, without attempting to characterize the variation in results.\n\nOverall, I think the work is well-presented, but I find that it is lacking with respect to some significant contribution toward the understanding of when and why fair washing attacks are possible. \n\nClarity-wise, my only complaint is that the variation in black box/explainer/dataset/metric combinations presented in the main paper images is somewhat hard to follow, e.g. Figures 1, 2, and 3 concern Decision Tree, Figure 4 concerns Logistic Regression. Perhaps either remain consistent or provide some rationale for the switch.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't have any outstanding concerns about negative social impact after reading this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "f5xJqAg6DkQ", "reviewer": "Reviewer_KFAW", "summary": "This paper studies \"fairwashed\" explanation of black-box ML models -- interpretable models that achieve high fidelity with respect to the black-box predictions but substantially better scores on various fairness metrics. The authors empirically demonstrate how fairwashing can be achieved with very small decreases in fidelity, and how a fairwashed explanation of one type of model (say, random forests) can be successfully repurposed to explain a different model (eg AdaBoost). Because of this, they argue, auditors need to be very careful about how they use interpretable models in fairness assessments.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written, interesting paper about an area of growing concern for the field. Its goals -- to characterize what is possible with fairwashing attacks -- are reasonably modest, but it lands its point clearly. There's room for expanding on this work, but I think it would be a worthy addition to NeurIPS. \n\n### Questions\n\nWhy not also test calibration on the continuous predictions, since this is very often what is used to measure fairness in real-world settings?\n\nIntuitively, it seems like these attacks would be easy to identify because to decrease some unfairness measure by 50% while only changing <1% of predictions would require obvious structure in the changed predictions (eg all the changes affect the same group). Is this the case?\n\n### Notes\n\nThe x axes in Fig 1 would be more readable if they didn't have the same scale.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is focused on the bad outcomes that could occur when interpretable models are used to assess fairness claims. It is clearly concerned with averting the potential negative societal impact of such assessments.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "R6UR5ma1xIr", "reviewer": "Reviewer_qEv3", "summary": "This paper addresses the problem of fairwashing, i.e., the process in which an adversary builds a manipulated explanation of the unfair black-box model. The manipulated explanations should ideally hide the fact that the block-box model is unfair. The authors conduct a series of experimental evaluations and conclude:\n-\tManipulated explanations can be up to 99.2 % as accurate as the original model, while being much less (up to 50% less) unfair\n-\tFairwashed explanations can generalize beyond the data points they were trained on.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper addresses a timely and relevant topic in the area of model accountability. However, there are several concerns regarding this work:\n1. The authors conduct their experimental evaluation for one fair-washing attack; yet the title suggests that a general approach has been identified.\n2. The experimental results in the main paper are shown for one data set only. Yet, there is plenty of space dedicated to a table summarizing related work. I suggest presenting some of the experimental results from the appendix in the main paper.\n3. I do not believe that the way fidelity is described in the paper provides a useful measure to understand how faithful the explanation model is to the black-box model. As an example, an explanation model with a probability of 0.51 would receive the same fidelity score as a model with a probability of 1. Moreover, even when the probability is 1 (or 0), the pre-sigmoid scores (i.e., logits) can still be substantially different, essentially indicating that the explanation model reasons differently. In principle one could imagine that the former explanation model, when outputting a constant class, would be much fairer than a more discriminative model. \n3. At times, the paper is not clear! I will give examples below.\n\nClarifications & Questions:\n-\tThe authors have not really made clear what their paper is trying to achieve: sometimes it sounds as if developing more stable methods is a concern?\n-\tQuestion: Figure 1 - EOpp is essentially constant across different unfairness thresholds. In contrast to that, the fidelity under PE and SP constraints are more responsive to the fairness thresholds. Why is that? Has it to do with EOpp not considering individuals with true label y=0?\n-\tHow does the proposed method generalize for the attacks by Dombrowski et al (2020, ICML) and Slack et al (2020, AIES)? Can the authors comment on this?\n-\tAt no point, the suing group X_sg was formally defined. What kind of data points are in there? Are those random samples from the data set?\n-\tEquation 1 contains very little to no intuition. It would have been nice to include some sentences describing the objective and why we should care about it.\n-\tThere are many claims being made which I feel are unsubstantiated. At least, there is no theory supporting such strong claims, e.g.,\n in the Abstract: ‘[…] show that fairwashed explanation models can generalize beyond the suing group, which will only worsen as more stable fairness methods get developed.’ To me this seems like speculation.\n-\tIn the figures, the vertical lines are thin and hard to distinguish.\n-\tIn Section 3.1, you first describe your experimental results in Figures 1 and 2. Then, in a separate paragraph, you describe the conclusions without referring to the figures these conclusions could be based on. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed fidelity measure could be better discussed and evaluated (see comments above).", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "L-2ldIOkHq", "reviewer": "Reviewer_KC7h", "summary": "The paper presents an empirical study to better understand the ability to fairwash unfair black-box models. They illustrate trade-offs between fidelity and fairness in explanation models. They also demonstrate that fairwashing can generalize beyond the suing group and transfer across models.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper explores a topic that is broadly relevant and significant. The approach is a natural and useful extension of prior work. The methods are not novel, but they produce interesting results and a better understanding of fairwashing. I like the clarity and presentation of key takeaways.\n\nSome kind of theoretical analysis would strengthen the paper. Since the paper lacks novel methods (for understandable and appropriate reasons) as well as theoretical analysis, the main contributions are the questions it chooses to address and the experiments. I like the questions a lot. The experiments are good, but could be improved.\n\nThe datasets used are far from ideal and this undermines the results, but they are the standards in the field. So it could be argued that these datasets are important due to the likelihood of their use in fairness auditing. The fact that experiments are repeated on different samples of the data helps too. However, I do have some questions about the experiments below and additional datasets would strengthen the paper.\n\n\n\nMajor questions and comments:\n\nIt is difficult to tell from the experiment descriptions whether the experiments have been set up correctly or have some common pitfalls. For example, it appears that the equal opportunity model is backwards for some datasets, Default Credit and COMPAS. Are your fair models giving men and women equal opportunity to be labeled as a default risk and people of different races equal opportunity to be label high-risk of reoffending?\n\nI like the takeaway that fidelity is a poor proxy for quality of explanation. However, I wonder if the following more nuanced takeaway is also valid. Since there does seem to be a relationship between fidelity and fairness, do these results imply that fidelity could be used, but only with strict thresholds? In other words, does this work show evidence that very minor discrepancies in fidelity could still be used as a filtering step to identify potential fairwashing instances that should be given higher scrutiny?\n\nSome results for the black box models are left to Table 4 in the appendix as mentioned in the quote below. Can you add some additional discussion of these to the main paper to make it clear that the baseline performance of your black box models is reasonable?\n“The performances (i.e., accuracy and unfairness) of the four black-box models on the training set, the suing group set as well as the test set of the four datasets are provided in Table 4 in the appendix“\n\n\n\nMinor comments/typos:\n\nThe term “suing group” should be defined early in the introduction at least informally. It would also help to have a more prominent formal definition than what is on lines 131-132.\n\nLine 20: high stake -> high stakes\nLine 177: one-hot -> shot\n\nPlease fix all instances of commas in latex math mode. \nE.g., line 155: $48,842$ should be $48{,}842$.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I’m not sure this was adequately addressed, but maybe I’ve missed something. Can the authors point to where they believe they’ve addressed these points?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an experimental study on the phenomenon of "fairwashing", namely an explanation model making an unfair blackbox model more fair than it is. In particular, the authors show that fidelity of an explanation model is not necessarily an indication of "fairness-fidelity", that is an explanation model may be misleading with respect to a given fairness measure, while scoring high no a fidelity metric. The study demonstrates this phenomenon with respect to various fairness metrics and explanation models. 

The reviewers appreciated the clarity of the experimental setup and presentation. Moreover, the phenomenon studied here is clearly important and of interest to the ML community.

On the negative side, the submission does not attempt a formal/theoretical analysis of the the phenomenon. Section 4 seems of speculative nature, empirically exploring the range of unfairness among explanation models of a set fidelity. It is shown that the relationship can differ among different datasets, but no attempt of formally analyzing these seems to have been made.


Decision: Accept (Poster)