Washing The Unwashable : On The (Im)possibility of Fairwashing Detection
Authors: Ali Shahin Shamsabadi
Mohammad Yaghini
Natalie Dullerud
Sierra Wyllie
Ulrich Aïvodji
Aisha Alaagib Alryeh Mkean
Sébastien Gambs
Nicolas Papernot
Venue: NeurIPS 2022 Accept (2022)

Abstract: The use of black-box models (e.g., deep neural networks) in high-stakes decision-making systems, whose internal logic is complex, raises the need for providing explanations about their decisions. Model explanation techniques mitigate this problem by generating an interpretable and high-fidelity surrogate model (e.g., a logistic regressor or decision tree) to explain the logic of black-box models. 
In this work, we investigate the issue of fairwashing, in which model explanation techniques are manipulated to rationalize decisions taken by an unfair black-box model using deceptive surrogate models. More precisely, we theoretically characterize and analyze fairwashing, proving that this phenomenon is difficult to avoid due to an irreducible factor---the unfairness of the black-box model. 
Based on the theory developed, we propose a novel technique, called FRAUD-Detect (FaiRness AUDit Detection), to detect fairwashed models by measuring a divergence over subpopulation-wise fidelity measures of the interpretable model. 
We empirically demonstrate that this divergence is significantly larger in purposefully fairwashed interpretable models than in honest ones. 
Furthermore, we show that our detector is robust to an informed adversary trying to bypass our detector. The code implementing FRAUD-Detect is available at https://github.com/cleverhans-lab/FRAUD-Detect.

Washing The Unwashable: On The (Im)possibility of
Fairwashing Detection
Ali Shahin Shamsabadi' The Alan Turing Institute Vector Institute
Mohammad Yaghini* University of Toronto Vector Institute
Natalie Dullerud" University of Toronto Vector Institute
Sierra Wyllie
UIrich Aïvodji ÉTS Montréal
Aisha Alaagib
University of Toronto
University of Toronto
Vector Institute
Vector Institute
Sébastien Gambs Université du Québec Montréal
Nicolas Papernot University of Toronto Vector Institute
Abstract
The use of black-box models (e.g., deep neural networks) in high-stakes decision- making systems, whose internal logic is complex, raises the need for providing explanations about their decisions. Model explanation techniques mitigate this prob- lem by generating an interpretable and high-fidelity surrogate model (e.g., a logistic regressor or decision tree) to explain the logic of black-box models. In this work, investigate the issue of fairwashing, in which model explanation techniques manipulated to rationalize decisions taken by an unfair black-box model using de- ceptive surrogate models. More precisely, we theoretically characterize and analyze fairwashing, proving that this phenomenon is difficult to avoid due to an irreducible factor-the the unfairness the black-box model. Based on the theory developed, we propose a novel technique, called FRAUD-Detect (FaiRness AUDit Detection), detect fairwashed models by measuring divergence over subpopulation wise fidelity measures of the interpretable model. We empirically demonstrate that this divergence is significantly larger in purposefully fairwashed interpretable models than in honest ones. Furthermore, we show that our detector is robust to an informed adversary trying to bypass our detector. The code implementing FRAUD-Detect is available at htttps://githi www.cleverhaaseeeeeeeoomemmm haw.eeeeeeeaaaaaaaaaaaaaaaaa 1ab/FRAUD Detect
Introduction
The wide applicability of machine learning models has recently increased their usage in high-stakes decision systems such as credit scoring [33]. insurance risk (TO) and predictive justice (30). The consequences of erroncous decisions that are based on predictions of machine learning models (e.g., people being wrongly denied parole (46)) have increased the demand-from both the public and govermment-to provide an explanation to humans about model decisions. For instance, this appears as an explanation requirement in the European General Data Protection Regulation [26). However, various widely-used model architectures, such as deep neural networks, are considered black-boxes due to their complex and hidden internal logic, impeding the ability to explain their decisions in terms that are understandable by a human. To address this issue, black-box model
"Contributed equally.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Honest interpretable model
Company
Austior
.8
Interpetibble
Rate
Diu
Falmers
Interpretable model
Fidelit
Fahwashing?
Black-boo model
Sung set
T
F+
P
Exslanation requeett
User query
Company decision
Suing set (user data, company decisions)
Fairwashed interpretable model
Rate
T
F+
F
Figure 1: A dishonest company provides a high-fidelity but fairwashed interpretable model to rationalize decisions made by their unfair black-box model. FRAUD-Detect can be exploited by the auditor to distinguish between the fairwashed and honest interpretable model in a non-cooperative way with only query access models. FRAUD-Detect detects fairwashing based on the agreement and disagreement of the interpretable model with the black-bor -bx model among subpopulations O
and 1 quantified by the subpopulation-wis wise confusion matrix including the True Positive ). False Positive (F*),, False Negative (F and True Negative (T") rate of the interpretable model w.r.t. the black-box predictions. The top (bottom) plot shows that the confusion matrix of subpopulations are similar (different) in the honest (fairwashed) interpretable model. Fairwashing changes T* or F+ in the opposite direction across subpopulations to equalize the probability of predicting positive label across subpopulations and 1. For example, the fairwashed interpretable model decreases 7++ of subpopulation 0 while increases 7 of subpopulation 1 with respect to their T in the honest one.
explanation techniques 27 6 aim to provide reasons for predictions of black-box models in human- understandable terms. One possible way realize this is approximate the black-box model using an inherently interpretable model (e.g., logistic regression or decision tree) whose decisions can be easily explained by design (27]. For instance, in logistic regression, the coefficients of the model directly represent the importance of each input feature, while decision tree is composed of a set of explainable if-then decision rules related to the input features.
In this black-box model explanation setting, the interpretable model is trained with the objective that its predictions agree with the predictions of the black-box model (i.e., high fidelity (20). However, black-box model explanations can be manipulated to provide deceiving explamations, even if these ex- planations display high fidelity with respect to the original black-box model. An example of such ma- nipulation occurs when the explanations provided cover up the unfairness of the underlying black-box model, resulting in fairwashing 2 3 As concrete example, dishonest bank could use black- box model that denies a loan to customer in discriminative manner based on a sensitive attribute such as race or gender identity-witn while providing an interpretable surrogate model promoting the false impression that the decisions are fair and based solely non-sensitive features. These manipulations are made possible in part by the ambiguity in current regulations (26) when comes to describing what constitutes a valid explanation [45 Fairwashing is clearly an ethical issue as individuals who chance to contest the decision
have unfairly received unfavorable decisions are deprived
Due to the growing importance of fairness in machine learning 77 we focus on a scenario in which dishonest entity attempts to perform fairwashing [2 3 when providing model explanations to an auditor (e.g., an external dedicated party). For example, if company's 's users expect that the decisions they received were a result of model bias, they may send their data and received decisions to an auditor. The auditor may make legal demand [8] on the company to provide an interpretable model to explain these users' received decisions (see Figure M. In this case, dishonest company has an incentive to manipulate the black-box model explanation technique, to under-report and occlude the unfairness of the black-box model under scrutiny to evade the potential consequences of unfair decision- decision-making. making. The dishonest entity could thus mislead the auditor by rationalizing its decisions

---

using the corresponding fairwashed interpretable model. In this paper, we are the first to propose method for detecting such dishonest model explanations. This is challenging problem. Fairwashing cannot be detected through computing the explanation violation of the explainable model, such as fidelity of explainable models, as we demonstrate that the explainable model cannot achieve perfect fidelity with respect to the black-box model. In addition to this, measuring the differences between the fairness of the black-box model and the explainable model cannot help to detect fairwashing as there are several design choices (including fairwashing) that could lead to different fairness, thus non-intentional fairwashing.
Our primary contributions include (1) an extensive theoretical analysis of fairwashing and the faimess limitations in explanation techniques and (2) a novel approach for detecting, and thus deterring, fairwashing. Our proposed method, FRAUD-Detect (FaiRness AUDit Detection), formulates the problem of detecting fairwashing as a non-cooperative test; both to overcome possible dishonest behaviors in entities that are actively being audited and for communication efficiency. FRAUD- BRAUD-Detedd does not require access to the black-box
Detect operates in a realistic scenario: model provided the users calling for the audit have provided their data and received decisions, which is a realistic setting as often the model is not provided by the entity due to intellectual property and trade- secret concerns 39 37]. Thus, FRAUD-Detect only relies on the predictions of the interpretable and black-box models. Both our theoretical and empirical analysis quantify the per-subpopulation, per-label fidelity of the interpretable model with respect to the black-box model.
Figure I shows confusion matrix distributions for two sensitive subpopulations before and after fairwashing. These matrices are constructed by comparing the interpretable to the black-box model predictions. Fairwashing induces a divergence between subpopulation values to conceal the un- fairnsss of the black-box model. FRAUD-Detect leverages the Kullback-Leibler (KL. divergence between subpopulation confusion matrices to distinguish between honest and fairwashed interpretable models. For comprehensive empirical study, we examine logistic regressors and decision trees as interpretable models with respect to black-box Deep Neural Networks, AdaBoost (24), Gradient
Boosted Decision Trees [17 and Random Forests [13 on three benchmark datasets: COMPAS (5)
Adult Income
22 and Bank Marketing 34
We evaluate the strength of our detector in the presence of an informed adversary representing an informed dishonest entity with knowledge of FRAUD-Detect that seeks to fairwash and bypass detection by constraining the divergence over subpopulation-wiss confusion matrices. Our empirical experiments quantify the capacity of this informed adversary by computing the achievable fairness gap and fidelity under the additional detection constraint. We provide additional theoretical support for the challenge of jointly satisfying fairwashing and evasion in appendices.
In summary, our main contributions are as follows:
We characterize fairwashing via a theoretical analysis on the difference in subpopulation gap between the black-box model and the interpretable
violation (Section B Based on our fairwashing theory, we establish requirements for sufficient fairwashing detection method and prove that fairwashing impossible to avoid completely due to an irreducible component in the violation term.
We introduce the first method for fairwashing detection (Section Informed by our theo- retical results, we observe that fairwashing causes disparate effect on the subpopulation fidelity distributions of the interpretable model with respect to the black-box predictions. We leverage this observation to propose a non-cooperative black-box access method, FRAUD- Detect, which utilizes KL. divergence between fairwashed and honest interpretable models.
per-subpopulation confusion matrices to distinguish
Our empirical results demonstrate that FRAUD-Detect successfully detects fairwashing based on the KL divergence between subpopulation-win wise confusion matrices (Section 6). We show that the divergence over subpopulation confusion matrices can vary by over 0.6 between an honest and a fairwashed interpretable models.
We illustrate the robustness of FRAUD-Deteds against an informed adversary (i.e., a dishonest entity who attempts to fairwash while evading detection). Our empirical results show that evading our detector comes at the cost of significant increase in subpopulation gap, negating fairwashing. Specifically, dishonest entity that jointly attempts to fairwash while evading detection only achieves parity gaps greater than 10%.
3

---

2 Preliminaries and Problem Formulation
2.1 Related Work
Fairness. Many different formal definitions of algorithmic fairness have been proposed in the machine learning community [35]. One of the main challenges of algorithmic faireess is the lack of consensus on a universally applicable fairness definition. More precisely, it has been formally demonstrated that many such measures are incompatible with each other (i.e., they cannot be achieved jointly in some situations) [3I. In addition, their differences are also rooted in their underlying philosophical and moral assumptions 28 Thus, the choice of a particular faimess metric is usually context dependent. Nonetheless broadly speaking, there are two main families of fairnsss definitions: individual fairness and group fairness [23] Individual fairness posits that "similar individuals should be treated similarly".
In contrast, group fairness strives to equalize statistical properties of classification outcomes across subpopulations created by partitioning the population based on sensitive attribute A such gender. In our work, we focus on a particular notion of group fairness, demographic parity, also called statistical group parity, which refers to observing equal probability of positive label prediction over subpopulations. One of the reasons we rely on demographic parity as the fairness metric is that true labels will not be available in a realistic scenario (especially in fairwashing scenario described in Section 2.3). In general, the existence of true labels conflicts with the assumption that when requesting decisions (or explanations), users (or auditors) do not have access to these true labels [42]. Furthermore, using the predictions of a black-box model as the true labels is paradoxical in the sense that the black-box model is assumed to be unfair. Demographic parity enforces independence between the class predicted by the model and inclusion in particular subpopulation.
Definition 1 (Demographic parity 15 ). A classifier Y satisfies demographic parity with respect to sensitive attribute if
pr|y 1|A = a| = -Pr(L- = 1|A bb 6] Va,bEA
Global black-box explanation methods focus on explaining the whole logic of black-box models by training an inherently interpretable surrogate model. We refer the interested reader to Appendix A for the description of other explanation methods. In terms of abusing explanations via fairwashing, Fukuchi et al., similarly to us, try to detect fairwashing but with a different setting [25]. Slack et al.
shows alternate method of fairwashing that evades local model explanation techniques in setting that is comparable to ours but they do not attempt to detect fairwashing
44]
We explore these
additional fairwashing settings in Appendix
2.2 Desiderata for Global Black-box Explanations
We consider the setting in which an entity learns a complex black-box model B(-) on training set (Xn, Yn. A), in which A represents a sensitive attribute. We maintain the setup from recent
fairwashing literature (3, and consider binary classifiers mapping M features into binary label B RM + (0. 1}. We also consider binary-valued sensitive attributes, A E {o, 1}. Note that we do not assume that necessarily used during the training of B--). Given user queries, the entity uses the predictions of (-) part of a high-stakes decision system. Later, a group of these users may request an explicit explanation from the entity due to concerns over improper use of the sensitive attribute. Such scenario frequently arises in high-impact domains, such as bank loans and credit scoring. Let Xsg be a suing set comprising the unlabeled data of users demanding an explanation for their particular outcomes (see Figure D. Consequently, external auditing entity requires the company to provide explanations in terms that are understandable to humans about the predictions of the black-box. Here, we will assume that the explanation will be provided in the form of global explanation. This means that a simple interpretable model I(-) will be trained on a dataset X labeled by querying the black-box model B-) trained on XIe such that I(-) accurately reflects and explains the logic of B(-) in terms that are understandable to humans.
Formalizing further the auditing desiderata for black-box model explanations, the interpretable model T(-) must accurately mirror: 1) the output predictions of the black-bor model (i.e., fidelity) and 2) the fairness (according to pre-defined metric) of the black-box model.

---

Fidelity. The interpretable model should satisfy the fidelity criterion for any X ~ D. Fidelity (defined in Appendix D is difficult perfectly achieve and challenging to measure for an auditor over all X ~ D. Thus, we additionally introduce the notion of empirical fidelity.
Definition (Empirical Fidelity). The empirical fidelity on dataset = {xi) J. including data points 24 is defined the relative accuracy of I(.) with respect to B(-) X:
N
1
EmptricalFiiellttgi B) = Σ (T(())= = B(x))),
(1)
N
where the indicator 1 1(5.) outpiis 1 the output label of T()) and B()) are the same and otherwise.
Fairness. As the audit of B-)) is expressly requested due to fairesss concerns with respect to A, the interpretable model should reflect the fairness violations or adherences of the black-box model.
2.3 Fairwashing Definitions
To evade legal consequences of decision-making based on improper use of sensitive attribute, dishonest company may perform fairwashing. Fairwashing with respect pre-defined fairness metric permits the dishonest company to learn an interpretable model that hides the unfair behaviour of their black-box model. In practice, fairwashing could also occur due the fact that current regulations (26) do not define what constitutes valid explanation, thus leaving the possibility for the model provider choose the interpretable model that meets their needs [45] Fairwashing can be quantified with respect to particular fairness measure.
Definition 3 (Fairwashing). Let F and I n define the fairness gaps of the interpretable model and black-box model, respectively, with respect to pre-defined fairness metric. For example, in terms of demographic parity (Definition) 7
F1 = Pr
Y1 = A
Pr Y, = 1| -1
(2)
BB = Pr -n= =
= A-
Pr V
A 1
We define the fairwashing violation y 0 as the difference between D3 and T
? = I1
(3)
The fairwashing violation 7 non-negative. Remark that the case of < the opposite of fairwashing since the interpretable model is displaying bigger fairness gap than that of the black-box model, thus a model owner has no incentive to employ such an interpretable model.
Assumption 1.
Fairness Gap of the Black-Box Breeds Fairwashing
We analyze whether it is possible to eliminate the risk of fairwashing altogether. We characterize the fairwashing violation y and show that completely eliminating fairwashing impossible: an interpretable model explaining an unfair black-box model always has non-zero fairwashing violation.
Theorem 1. Assume YB the black-box model B(-) predictions, Y, are the predictions of surrogate (interpretable) model I(.) trained on B-)) outputs and A is a sensitive attribute: 1) If B(-) does not satisfy demographic parity, completely eliminating fairwashing is impossible (i.e., > 0); 2) A detector for fairwashing measuring false salse-positive rates and true-posst positivee rates of I(-) with respect
(-) ) is sufficient.
We provide proof sketch here (see Appendix D for full proof).
Proof Sketch. Define T the true- rrue-positive rate of I(-) with respect B(-) on X Da. in which D. denotes the distribution over data with attribute a e A. Define F+ the false-positive rate of I()) with respect to B(-) on X D.. Denote the differences between Ty and T and Ft and Ft as 5, 8', respectively. We can express Fy in terms 8, 8, Tst, F+t. FB and Yg by expanding terms and using Bayes" formula:
T = PB (T RT) + (8 8¹) Pr Yg A 0l +
(4)
5

---

Now that we have derived ry in terms of our desired terms, we can eliminate ryy from the equation for The resulting formula for in terms of 8, 8' TT. FT. I'B and YB/ A allows us to demonstrate correctness, irreducibility and sufficiency.
r= + F T - 5' Pr Vo - 01 A= 0| - Pr Vv = A =
(5)
Correctness, Note that if B 0 (i.e., black-bor model is fair) then no- no-fairwashing is present, and
no detector should mistakenly report fairwashing. We can verify this with Equation (L5). Under
we know that 7 2 0 and as below terms are. by definition non non-negenn non-negative, thus: Assumption |
7 Pr YB 0 A 0] 8r Y 114= 0 = ='
Irreducibility. If B * 0, even i = = 0, i.e. when true-positive and false-positive rates are equal
over subpopulations (Ti = ' and Fo = Fit)) there exists an irreducible fairwashing violation:
7 = n Ft- 7,t) 0,
since 1 + FT - > except in trivial cases (see Appendix D for the proof). We further point out that 1 F1 -T not function of disparity in the interpretable model as the value does not measure disparity between subpopulations. Rather, 1+ Fi - -TT is function of the difference in true-positive and false-positive rates (Ti+ and Ft, respectively) within a single subpopulation. The sole measure of disparity in the gap above is black-box TB. which is a given constant.
Sufticiency. The remaining factors of the gap are only function of and 8' as Pr YB = 1 A 0f
and Pr Vv = | A = o] are also given constants. Therefore, a detector that measures 8 and 8' is sufficient to detect fairwashing.
Detecting Fairwashing
4.1 Fairwashing In Practice
Eliminating the fairness gap of the black-box model would be the ideal outcome as it would also remove the possibility for a dishonest model provider to perform fairwashing. However, in practice absolute fairness is often unrealistic and even regulatory bodies tolerate small violations of fairness laws 18] In Section B we showed that fairness violations result in fairwashing violations unless the interpretable model has higher fairness gap than the black-box model, which would not be desirable from the point of view of the model provider. Therefore, not all fairwashing violations may have been intended by the model owners. These involuntary violations are in fact side-effects of the imperfect fidelity, which is inherent optimizing the empirical fidelity (Definition 2 of the interpretable model on a training set X ~ D,
max
EmpirricalFideltty(.... Bx)
In contrast, deliberate fairwashing is the result of a similar optimization problem purposefully constrained to provide better fairness target suing set Xsg D:
Definition 4 (Fairwashing Optimization (2). Given a black-box model B(-) and a suing set Xag. fairwashing optimization is defined as learning an interpretable model 1(-) from B(-) on X such that the interpretable model has 1) high fidelity with respect to the black-box model and 2) is less unfair than this black-box model:
max EmpiricaalFddllttyll. Xxg)
subject to FasrnessGap(I; X Se,
in which e. is an upper bound on the fairness gap of the interpretable model on the suing set FairnessCap(I: Xsz), and FairnessGap(
constitutes a measure of fairness gap according to some fairness metric generically measured by the subpopulation gap in fairness metric. Note that 6 < FairnessCapl B; Xxg) must be less than the fairness gap of the black-box model on the suing set FairnessGap B; Xag) (g) to cause fairwashing violation 0 (see Definition 3..

---

4.2 Proposed method: FRAUD-Detect
As demonstrated in Section S fairwashing detection method that relies on the difference over subpopulations in true-positive and false-positive rates of the interpretable model w.r. the black- box model (Ō and 8' respectively) is sufficient. Inspired by this theoretical analysis. we propose our fairwashing detector, FRAUD- which locates fairwashing violations. FRAUD-Detect aims to distinguish between honest and fairwashed interpretable models approximating the unfair black-box model B-)) as described Section 2.2 under the additional following constraints: I) without white-box access to the black-box and interpretable models as well as; 2) independently of the cooperation the model provider being audited. These constraints are introduced to conform to realistic scenarios in which entities desire to retain the confidentiality of their black-box model due to intellectual property rights and because they considered it as a company's 's asset.
Similarly to definitions of truc- prue-posttive T and false-positive Ftt rates of I()) w.r. B(-) on data with attribute a A {0, 1}, we define the true-negative rate, T-- and false-negative rate, Fa as:
Tg = Pr(V, = 0 YB = 0, A a), F. = Pr 0 Vn = 1,A A=
(8)
for a E A = [0, Note that T~ = 1 111-F Fo and F = 1
FRAUD-Detect computes the T Fxt TT= For for each subpopulation a E A - 10, 1] using the predictions of the interpretable model and black-box model on the suing set Xag Let
Co = - Fot To 1, C C-
be the (flattened) confusion matrices of the interpretable model w.r.t the black-box model on Xag with attribute 0 and 1, respectively. Then, FRAUD- -Dtectt computes the divergence between Co and Ci using Kullback-Leibler (KL) as:
CKL. KL(Co,
(10)
Recall that we demonstrated in Section f that the divergence in T and T (6) and Fo and F+ (8') were sufficient for detection method. We can complemen our detection method with additional divergences in T-- and F- and measure the dissimilarity via KI. divergence. This choice natural as the KI. divergence is commonly used to quantify the divergence over probability distributions- C2 functions as a simplified representation of the probability distribution V/ = a. For an
honest interpretable model, CKL is generally relatively low (6,8' 8' = 0) as the only divergence arises from general error in fidelity optimization and the faimess gap of the black-box model (as shown in Section 3) In other words, the honest interpretable model approximates the black-box model equivalently across subpopulations. However, for fairwashed interpretable model, CKI. grows significantly, as the interpretable model is explicitly manipulated to optimize the fairness across subpopulations and improve over the black-box model. We quantify the distinction between honest and fairwashed interpretable models via a threshold A > 0 on CKL such that if CxL > A, the interpretable model is considered fairwashed. Note here that due the irreducibility result stated in Section 3 A must be chosen such that A is tightly greater than the irreducible term in order to properly distinguish between accidental fairwashing that occurs as a result of the irreducibility and malicious fairwashing. This procedure is detailed in Algorithm
5 Evading FRAUD-Detect
In this section, we investigate on whether a dishonest entity could evade FRAUD-Detect while performing fairwashing. We assume that this dishonest entity, which we call the adversary, is informed about FRAUD-Detect. Namely, the adversary is aware the fairwashing detection method and desires to fairwash while evading detection by the auditor. To achieve this goal, the informed adversary has the objective of finding a fairwashed interpretable model that satisfies an additional constraint on the Kullback-Leible (KL) divergence of the confusion matrices among subpopulations, CxL in the previous section. To probe the robustness of our detector to an informed adversary empirically, explore the range of faimess gap given fixed value of fidelity and fixed value of CKL via solving the informed adversary optimization problem.
We performed experiments in two settings: 1) considering all related true-positive, false-positive true- negative and false-negatee negative rates; 2) considering only independent true posiiiiii and false positivv rates.
7

---

Algorithm FRAUD-Detedd {predicate) {(a, a, CB. 01) (X.g. A, A, fi) predicate]
Input: Query access the interpretable model I(-), suing dataset Xg. Black-box model pre- dictions on the suing set B(x.)), sensitive attribute a E A and threshold A >> O.
5:
F Hi
FNR
T; +- [1
TNR
7: Co f [T Fo.Fo
Fo
[T fairwashing is detected
8: C. F -T,, T FT
Output:
{F
fairwashing is not detected
9: CKT. + KL(CO, Ci)
& Kullback-eeibler
1: Y1 < [(X.g) > -Query interpretable model
10: if > A then
11: return T
2: for + 0, do
3: T (ū==1_,/=1,===)
12: else
> TPR
[0B = |-==|
13: return F
4: F.- + 1( 0 b-
FPR
0,a=i]
Definition 5 (Informed Adversary Optimization). Given black-box model B(-), suing set X a sensitive attribute A, a loss threshold and a fairwashing detection threshold A an informed adversary aims to learn interpretable I(-) such that
FoérnessGap(L; Xea) subject X(I: Su and CKL S.
minimize
(1I)
where CKL the KL divergence between the subpopulation confusion matrices of Xsg.
The differences between the fairwashing optimization in Definition 4 and the informed adversary optimization of Definition 5 are two fold. First, the order of the maximand and the constraint is valid since each indicates priorities of the adversary, namely, whether
reversed. Both formulations to put hard limit on the fidelity loss or on the fairness gap of the interpretable model. Second, in Definition 5 we replaced the non-differentiable measure of Loss L[I; X.g). In our empirical evaluations, we use the logistic regression loss.
FeeFFFiddeil.,,.. B; Xx) with differentiable
To solve the above optimisation problem, we consider the Rashomon set of high-fidelity interpretable models and compute the range of the fairness gap interpretable models. short, given a classifica- models (interpretable
tion task, the Rashomon Set [41] is defined as the set of models in our case). More precisely, given a model class F, loss function (.p(.) over dataset 'D of interest, reference model f (e.g.. optimal model) and performance threshold E [0, 1]. the Rashomon set RF = {f E LD(f) S LD(f") +T).
We extend the Fairness in the Rashomon Set (FaiRS) algorithm 19 to compute the range of the fairness gap of interpretable models that can be generated over the of high-fidelity models satisfying a constraint on the KL divergence. FaiRS exploits the so-called Rashomon Effect (14). which is an empirical phenomenon resulting in multiple models displaying the same performance overall (e.g., w.r.t their global accuracy) but have significant differences in terms of their individual predictions [19]. our setting, the Rashomon effect implies that several interpretable models can achieve the same high fidelity w.r.t the black-box model while displaying different values of fairness gap. Thus, computing the range of the fairness gap of high-fidelity interpretable models that satisfy a constraint on CKL quantifies the robustness of FRAUD-Detect to the informed adversary.
To the best of our knowledge, the FaiRS algorithm of [9 the only work proposing practical solution to efficiently characterize the range of the fairness gap over the Rashomon set. Alternative approaches would require measuring the range of the fairness gap over approximations of the Rashomon set obtained by either generating models using several hyperparameter values (e.g.. seeds 33 21) or brute-forcing over a particular class of model (e.g., depth seven decision trees This is not practical due to the high computational cost and may lead to sub-optimal results.
6 Empirical validation
FRAUD-Detedt aims detect fairwashing by distinguishing between a fairwashed interpretable model and honest interpretable one during audit. Additionally, TRAUD-Detect seeks to be robust

---

AB DT
XGB DT
DNN DT
RF & DT
AB &e LR
XGB LR
DNN & LR
RE LR
Partty gap of T:
D 16
D
34
8
1
Figure 2: KL divergence between the confusion matrices of subpopulations, CKL. and parity gap as function of e fairwashing in interpretable models (Logistic Regression (LR) and Decision Tree (DT)) explaining black-box models (AB, Deep Neural Network (DNN), Random Forest (RF) and Gradient Boosted Decision Trees (XgBoost)) using COMPAS. Values before 1 .6 are constant and not shown. See Appendix F for the results of other two datasets.
even against an informed adversary, i.e., demonstrate the ability to detect fairwashing even when a dishonest entity is aware FRAUD-Detect and attempts evasion. Therefore, we empirically validate the performance of FRAUD-Detect in: 1) capturing relationship between CKI. and demographic parity gap of the interpretable model r(:): CKI. increases as the demographic parity gap of the interpretable model decreases (i.e. as fairwashing becomes more severe); 2) restricting the fairwashing capabilities of the informed adversary: incorporation of CKL constraint into fairwashing problem prevents favorable demographic parity gaps for the interpretable model, precluding fairwashing.
We assess the performance of FRAUD-Detect using diverse set of black-box architectures, inter- pretable models and datasets. More precisely, we consider four architectures of black-box models: Deep Neural Networks (DNN), AdaBoost (AB) [24]. Gradient Boosted Decision Trees (XGBs) [77] and Random Forests (RFs) [3]. We evaluate the approach on three real-world datasets corresponding
to critical decision systems: Adult Income (22). Bank Marketing (34) and COMPAS (5). We rely
on Logistic Regression (LR) and Decision Trees (DTs) as our interpretable models due to their
high performance and ease of explainability. We refer to Appendix E for details on these datasets, black-box models, interpretable models and implementation of FRAUD-Deted
FRAUD-Detect successfully detects fairwashing, Figure 2 illustrates both the demographic parity gap and CKI. of interpretable models as a function of e. The bigger the value of e, the stronger the fairwashing (see Definition 4. Any graph within the figures may be read from left to right as the degree of fairwashing increases. As fairwashing becomes more severe, the demographic parity gap decreases while CKL increases such that we observe a relationship between the demographic parity gap of interpretable models and the associated CKL values. For example, decreasing the demographic parity gap of Decision Trees trained on a COMPAS dataset labelled by RF black-box model by half, increases CKL. by a factor of 4. The amount of change in CKI. and in the demographic parity gap of fairwashed interpretable models differ across the dataset and black-box models due to the different original fairness of the black-box models. As CKL. is sensitive changes in fairwashing, FRAUD-Detect can be used to detect fairwashing by not permitting CKL to pass certain threshold value. The CKI. thresholds corresponding to 5% and 50% fairwashing are reported in Appendix G Per-seed results are provided in Appendix H
FRAUD-Detect is robust to the informed adversary. Figure F shows the range of the demographic parity gap of high-fidelity fairwashed interpretable models subjected to constraint the CRI. by solving the optimization problem of Equation (TI). We use four types of black-box models (AB, XGB, DNN and RF). However, for simplicity, we performed the experiment using logistic regression as the interpretable model. The fairwashing detection thresholds were chosen from a wide range, i.e. A {0.01, 0.03, 0.05, 0.07, 0.10, 0.20) (the constraint on CKI in Equation (T)). A direct implication for FRAUD-Detect is that using low fairwashing detection thresholds makes fairwashing difficult. Consistently over all these results, imposing constraint on the CxL significantly narrows the range of demographic parity gap of fairwashed interpretable models. For instance, on COMPAS with A = 0.2, for an AB model that has demographic parity gap of 0.214, fairwashing can produce completely fair (i.e., demographic parity gap of 0.0) high-fidelity interpretable model while remaining undetected (i.e., CKI. < 0.2). However, when CKI. decreases to 0.1, the fairest high-fidelity interpretable model obtained by the adversary exhibits a demographic parity gap of 0.107.

---

Parity gap of I+))
AB
XGB
DNN
RF
22
10
11
-77
-70
.55
.65
-77
-7
-73
-81
-78 -9
Fidelity
A:
.01 .03 .05
.07
.2
Figure 3: Range of the demographic parity gap of fairwashed logistic regression interpretable models subjected to a constraint (A) on the CKI. (A E {0.01, 0.03, 0.05, 0.07, 0.10, 0.20)). explaining AB, Deep Neural Network (DNN). RF (RF) and Gradient Boosted Decision Trees (XgBoost) black-box models trained on using COMPAS. Horizontal lines denote the parity gap of the black-box models. See Appendix for the results of other two datasets.
7 Conclusion and Future Work
Incorporating fairness and explainability in decision systems can help foster trust (2 them by characterizing failure modes (e.g., unfairness) and providing assurances (e.g., fairness constraints) [T6). Auditing provides an oversight component and helps avoid first-order failures such as unfairness. In this sense, fairwashing can be in both fairness and explainability. Second-order failures help characterize the limits of techniques designed to reduce the risks of first-order failures. Our work is the first to delineate the theoretical practice, limits of what is possible in auditing fairness and in containing the risk of fairwashing. demonstrated the possibility of performing fairwashing through joint optimization of fidelity and fairness constraints when providing explanations. This could thwart auditing processes that are solely based analysis of the delivered interpretable model. We address this issue by proposing an additional auditing protocol that queries the interpretable model. Finally, through both a theoretical demonstrate that fairwashing detector cannot be
failure of auditing second-order failure that reduces trust
framework and experimental evaluation, evaded by an attacker, informed of our detector, without significantly degrading the fidelity of model explanations. Future directions include the extension of the detector other fairness measures (e.g.. measures incorporating ground -ruun information) as well as extending the analysis to multi-class setups. However, these would both require additional assumptions on the data manifold (to account for class correlations) to establish similar results as in Section B
Acknowledgments
This work was supported by CIFAR (through Canada CIFAR AI Chair and Catalyst grant), by NSERC (under the Discovery Program, and COHESA strategic research network), by the Ontario Early Researcher Award, and by gifts from Intel and Meta. We also thank the Vector Institute's sponsors. Ali Shahin Shamsabadi Wyllie was also partially supported by Engineering Science at University of Toronto through ESROP grant. Ulrich Aïvodji
also partially supported by The Alan Turing Institute. Sierra
supported by the NSERC Discovery Grants program (2022-04006). Sébastien Gambs supported by the Canada Research Chair program, a Discovery Grant from NSERC as well as the NSERC-RDO DEEL project.
References
[1] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. 2018. A reductions approach to fair classification. In Proceedings of the 35th International Conference on Machine Learning (ICML). Stockholm, Sweden, (July 2018).
[2] UIrich Aïvodji, Hiromi Arai, Olivier Fortineau, Sébastien Gambs, Satoshi Hara, and Alain Tapp. 2019. Fairwashing: the risk of rationalization. In Proceedings of the 36th International Conference on Machine Learning (ICML) Long Beach, California, USA, (June 2019).
10

---

[3] Ulrich Aïvodji, Hiromi Arai, Sébastien Gambs, and Satoshi Hara. 2021. Characterizing the risk of fairwashing. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS). Sydney, Australia, (December 2021).
[4] Christopher Anders, Plamen Pasliev, aan-Kathrin Dombrowski, Klaus- Robert Müller, and Pan Kessel. 2020. Fairwashing explanations with off manifold detergent. In Proceedings of the 37th International Conference on Machine Learning (ICML). Online, (July 2020).
[5]
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine bias. ProPublica, (May 2016). Retrieved January 15, 2022.
[6] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Si- ham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020. Explainable artificial intelligence (XAI): concepts, taxonomies, oppor- tunities and challenges toward responsible AI. Information Fusion, 58, 82-115.
[7] Solon Barocas and Andrew D. Selbst. 2016. Big data's disparate impact. California Law Review, 104, 671.
[8] Adrien Bibal, Michael Lognoul, Alexandre De Streel, and Benoît Frenay. 2021. Legal re- quirements on explainability in machine learning. Artificial Intelligence and Law, 29, 2, 149- 169.
[9] Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020. Fairlearn: A toolkit for assessing and improving fairness in AI. Technical report MSR- Microsoft, (May 2020). http://ppp/.//..ioooootttttooo wwtp::/wwwwoooooooooooo
a-
and oco.l:1----...eeisiiaaaaaaiiaiir
forkiit-for-assesssiie
[10] Noorhannah Boodhun and Manoj Jayabalan. 2018. Risk prediction in life insurance industry
using supervised learning algorithms. Complex Intelligent Systems, 4, 2, 145-154.
[11] Olcay Boz. 2002. Extracting decision trees from trained neural networks. In Proceedings of the 8th International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD). Edmonton, Canada, (July 2002).
[12] Leo Breiman. 2017. Classification and regression trees. Routledge.
[13] Leo Breiman. 2001. Random forests. Machine Learning, 45, 1, 5-32.
[14]
Leo Breiman. 2001. Statistical modeling: the cultures (with comments and rejoinder by the author). Statistical Science, 16, 3, 199-231.
[15] Toon Calders. Faisal Kamiran, and Mykola Pechenizkiy. 2009. Building classifiers with independency constraints. In 2009 JEEE International Conference on Data Mining Workshops (ICDM). Miami, Florida, USA, (December 2009).
[16] Varun Chandrasekaran, Hengrui Jia, Anvith Thudi, Adelin Travers, Mohammad Yaghini, and Nicolas Papernot. 2021. Sok: machine learning governnnee. arXiv preprint ar 1xxiv:2199.1087
[17] Tianqi Chen and Carlos Guestrin. 2016. XgBoost: a scalable tree boosting system. In Proceed- ings of the 22nd International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD). San Francisco, California, USA, (August 2016).
[18] U.S. Equal Employment Opportunity Commission. 1979. Questions and answers to clarify and provide a common interpretation of the uniform guidelines on employee selection procedures. (March 1979). Retrieved 01/15/2021 from https /wwww eeoc.. gov/ laws /ggidncce questions and answers clarify and provide common interpretation umiform-guti gwwddeiinnnee
[19] Amanda Coston, Ashesh Rambachan, and Alexandra Chouldechova. 2021. Characterizing fair- ness over the set of good models under selective labels. In Proceedings of the 38th International Conference on Machine Learning (ICML). Online, (July 2021).
[20] Mark Craven and Jude Shavlik. 1995. Extracting tree-structured representations of trained networks. Advances in Neural Information Processing Systems (NIPS), 8, 24-30.
[21] Alexander D Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, et al. 2020. Underspecification presents challenges for credibility in modern machine learning. arXiv preprint arXiv:2011.03300
[22] Dheeru Dua and Casey Graff. 2017. UCI machine learning repository. http://archive archive
ics. www.uci.ee. udi.edu/ml
11

---

[23] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (TTCS). Cambridge, Massachusetts, USA, (August 2012).
[24] Yoav Freund and Robert E. Schapire. 1997. A decision-theorett generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55, 1,
139. ISSN: 0022-0000 DOL: hott:tttt:://o.i.oooo org/ 10. 1006 jss. 1997 1504 https / w/www.sciencedi.e..
com www.ccccciicccccc.c cttp:///ssiiieeecttt.cccceo/////
[25] Kazuto Fukuchi, Satoshi Hara, and Takanori Machara. 2020. Faking faimess via stealthily
biased sampling. In Proceedings of the 34th AAAI Conference on Artificial Intelligence (AAAI)
number 01. Volume 34. New York, New York, USA, (February 2020).
[26] Bryce Goodman and Seth Flaxman. 2017. European union regulations algorithmic decision- making and a "right to explanation", Al Magazine, 38, 3, 50-57.
[27] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM Computing Surveys (CSUR), 51, 5, 1-42.
[28) Hoda Heidari, Michele Loi, Krishna P. Gummadi, and Andreas Krause. 2019. A moral frame- work for understanding of fair ML through economic models of equality of opportunity. In Proceedings of the 2019 Conference
Fairness, Accountability, and Transparency (ACM F) Atlanta, Georgia, USA, (January 2019).
[29] Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. 2021. Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in ai. In Proceedings of the 2021 Conference on Fairness, Accountability, and Transparency (ACM FAccT). Online, (March 2021). ISBN: 9781450383097.
[30] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2018. Human decisions and machine predictions. The Quarterly Journal of Economics, 133, 1, 237-293.
[31]
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, 2017. Inherent trade-offs in the fair determination of risk scores. In 8th Innovations in Theoretical Computer Science Conference (ITCS 2017) (Leibniz International Proceedings in Informatics (LIPlcs)). Christos fuer Informatik, Dagstuhl, Germany. DO1: 110.114.2800 4230/LIPICS. ITCS. 2017 (3) Retrieved 07/16/2019 from.
H. Papadimitriou, editor. Volume 67. Schloss Dagstuhi--LecinnizzZZer
[32] Zachary C. Lipton. 2018. The mythos of model interpretabillity: in machine learning, the concept of interpretability is both important and slippery. ACM Queue, 16, 3, 31-57.
[33] Charles Marx, Flavio Calmon, and Berk Ustun. 2020. Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning (ICML). Online, (July 2020).
[34] Sérgio Moro, Paulo Cortez, and Paulo Rita. 2014. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62, 22-31.
[35] Arvind Narayanan. 2018 Tutorial: 21 fairness definitions and their politics. FAT* 2018, (March 2018). https://wwe youtube............oo..m....m cow wwtch?/==jIXuYYaaa
[36] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830.
[37] Inioluwa Deborah Raji. Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020. Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (ACM FAT*). Association for Computing Machinery, New York, New York, USA, (January 2020). DOI: 10. 10. 10e11111333110 000511095.3332773 Retrieved 01/19/2022 from.
[38] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should i trust you?" ex- plaining the predictions of any classifier. In Proceedings of the 22nd International Conference on Knowledge Discovery and Data Mining (ACM SIGKDD). San Francisco, California, USA, (August 2016).
12

---

[39] Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: research methods for detecting discrimination on internet platforms. In Data and Discrimination: Converting Critical Concerns into Productive Inquiry, preconference at the 64th Annual Meeting of the International Communication Association. Seattle, Washington,
USA, (May 2014).
[40] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: visual explanations from deep networks via gradient- based localization. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy, (October 2017).
[41] Lesia Semenova, Cynthia Rudin, and Ronald Parr. 2019. A study in rashomon curves and volumes: a new perspective on generalization and model simplicity in machine learning. arXiv preprini arXiv: /ttv:9908.000 01500.
[42] Reza Shokri, Martin Strobel, and Yair Zick. 2021. On the privacy risks of model explanations. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (A/ES). Online, (May 2021).
[43] Nacem Siddiqi. 2012. Credit risk scorecards: developing and implementing intelligent credit scoring. Volume 3. John Wiley & Sons.
[44] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020. Fooling lime and shap: adversarial attacks on post hoc explanation methods. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES). New York, New York, USA. (February 2020).
[45] Sandra Wachter, Brent Mittelstadt, and Luciano Floridi. 2017. Why a right to explanation of au- tomated decision-making does exist in the general data protection regulation. International
Data Privacy Law, 7, 2, 76-99.
[46] Rebecca Wexler. 2017. When computer program keeps you in jail. New York Times. Retrieved January 15, 2022.
13


Reviews:
[{"review_id": "eeK51LFm3H", "reviewer": "Reviewer_53g2", "summary": "This paper presents a theoretical analysis of fair washing and demonstrates that completely reducing fair washing in certain cases is not possible. In addition, they present a simple technique that looks at the KL between surrogate model confusing matrices to detect fairwashing. They evaluate their technique and find it helps detect fairwashing issues.", "questions": "- Figure 3 is a bit hard to follow. Could you help clarify this figure? Where does the dotted line come from? Why are there multiple fidelity values for every $\\Delta$?\n- Could you help clarify the problem setup a bit more, per the weaknesses section?\n- Could you clarify what you mean by interpretable model?\n", "limitations": "The authors have done a sufficient job.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Interesting problem \n- The empirical evaluation of fraud detect is comprehensive, considering many different models and settings.\n\nWeaknesses:\nThe main weakness is that the theory is missing explicitness in several places and there are gaps in what the exact setting is. Specifically, here are the main gaps I see:\n- The initial problem setup is not defined explicitly enough. It seems like the setting is a classifier with a sensitive attribute, but the specifics here are lacking. For example, in equation (1), is this a multiclass or binary class setting? Is the protected attributed discrete or continuous? If it's continuous, is it allowed to take on multiple values? These details are a bit hazy right now making it hard to figure out where the claims apply exactly.\n- What are the specifications of the interpretable model? Does this model mirror the predictions across multiple classes or just a single class?\n- The construction of interpretable model $I$ is a bit hard to follow as well, and $I$ is introduced as just \"a simple interpretable model I\". I think what is exactly meant by the interpretable model needs to be stated more explicitly, because right now, it is a bit hazy and hard to follow. This makes it difficult to follow the statement of Theorem 1, where the theorem specifices and interpretable model, without this really being defined. \n\nWhile the empirical results about the detection algorithm are pretty complete, it seems tricky to choose this threshold $\\Delta$ that determines whether fair washing is going on. It would useful to provide more guidance here.\n\nOverall, I think this paper needs to much more rigorously provide the problem setting and explicit definitions of what is going on.\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zh34qCft8o2", "reviewer": "Reviewer_bYAU", "summary": "The authors deal with the problem of detecting fairwashing, a maliciously generated explanation such that the model looks fair even if it is actually unfair. They investigated fairwashing in the global explanation and found that elimination of fairwashing is difficult, and fairwashing can be detected using false- and true- positive rates. Then, their proposed detection algorithm for fairwashing is to check the deviation of the false/true positive/negative rates between the original and explanation models using KL divergence. To evaluate the robustness of their algorithm, they introduce a fairwashing algorithm with a mechanism evading the proposed detection method. The empirical evaluations demonstrate the high detectability and robustness of their proposed detection algorithm.", "questions": "See strengths and weaknesses.", "limitations": "The accessibility to $B$ makes the situation no fairwashing risk. Hence, I recommend the authors reconsider the situation to be meaningful.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(Strengths)\n- The fairwashing is an urgent risk. Hence, the detection of the fairwashing is well motivated.\n- To the best of my knowledge, this is the first to propose a detection method for fairwashing.\n\n(Weakness)\n- The authors make an impractical assumption about the defender's knowledge.\n- Lacks some important references.\n\nThe authors deal with the interesting and demanding task of detecting fairwashing, and most of the results are technically sound. However, I found a fundamental flaw in the problem setting, by which they deal with the situation where there is no risk of fairwashing. Due to such a considerable flaw, I recommend the rejection of this paper.\n\nThe problem setting looks weird. The authors assume the detector can query the original model $B$ in a black-box way. However, if we can query $B$, we can also calculate the unfairness score of $B$, such as the difference of the conditional probabilities in the demographic parity, by which we can confirm the fairness of the original model $B$. Hence, under the assumption of having accessibility to $B$, the defender easily detects the unfairness of the original model $B$. In other words, there is no risk of fairwashing in this context. While this paper adequately validates the fact that their method can detect fairwashing in this context,  the detectability is obvious due to accessibility to $B$.\n\nThe statement of Theorem 1 is not rigorously defined. What mean by \"completely eliminating fairwashing\"? Also, what mean by \"sufficient\"? \n\nThis paper lacks the comparison with the following studies:\n- K. Fukuchi, et al. \"Faking fairness via stealthily biased sampling.\"  AAAI2020.\n- D. Slack, et al. \"Fooling lime and shap: Adversarial attacks on post hoc explanation methods.\" AIES2020.\n\nBoth papers investigate the risk of deceiving fairness. Remarkably, the first paper discusses the detectability of malicious modification for deceiving unfairness under situations where the detector can access the labeled benchmark dataset. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EvJ4GagR51c", "reviewer": "Reviewer_WPPT", "summary": "The paper characterizes the problem of fairwashing for interpretable models. It proposes a fairwashed model detection method based on measuring the difference over subpopulations in true-positive and false-positive rates of the interpretable model w.r.t. the black-box model. It shows the method’s sufficiency theoretically and effectiveness empirically even under the assumption of an informed attacker. The evaluation is conducted on two interpretable models of four classification models on three widely used fairness datasets.", "questions": "1.For other fairness criteria such as equality of opportunity, how easy is it to adapt the current method?", "limitations": "They are well addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1.The problem is well-motivated. Fairwashing can be a practical threat in reality.\n\n2.The theoretical analyses look correct.\n\n3.The proposed method is comprehensively evaluated and shows decent performance for practical usage. In particular, the setting of an informed adversary strengthens the generalizability of the proposed method.\n\n4.The paper is well-written, well-organized, and easy to follow.\n\n## Weaknesses\n1.The method currently only applies to one fairness definition. However, in practice, other definitions like equality of opportunity are also commonly used.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9tqLesugn61", "reviewer": "Reviewer_vBkK", "summary": "This paper focuses on the problem of fairwashing detection in black-box models. To be specific, fairwashing is a technique used to fool model explanation methods to produce deceptive outputs for unfair models. In this paper, the authors argue the importance of studying the faiwashing issue, and propose a novel framework FRAUD-Detect to detect fairwashed models, and present empirical results. In addition, theoretical analysis is provided to show that fairwashing is unlikely to be avoided. And the robustness of the proposed detection method towards an informed adversary is also discussed.", "questions": "1. How should we choose the hyperparameter \\delta in different scenarios (e.g. different tasks, different data)?", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper investigates a novel and important problem. Model explanation is a popular and promising approach for auditing a model with critical usage to avoid unfairness in automated decision-making. But such model explanation approaches are also vulnerable to being attacked, which has been overlooked. The problem is also challenging, a thorough study on it is needed.\n\n2. The paper provides a simple but effective method for detecting dishonest interpretation models that take advantage of the fairwashing technique.\n\n3. Theoretical conclusion is provided to show that fairwashing is unlikely to be completely avoided. Empirical results are provided to validate the effectiveness of the proposed detection method and also its robustness towards an informed attack.\n\nWeaknesses:\n1. The logic of the paper is clear but the expressions in the paper are kind of obscure and not easy for readers to follow. The motivation and the algorithm in the proposed detection model are simple and clear, but the authors use a lot of long sentences to describe them, which makes the simple things complicated. It would be better if the authors could use more short and succinct sentences, which will improve the presentation and readability of the paper.\n\n2. The hyperparameter \\delta (the fairwashing detection threshold) is a key parameter in this proposed detection algorithm. More discussions on how to choose this hyperparameter in practical use are needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept