Fooling SHAP with Stealthily Biased Sampling
Authors: gabriel laberge
Ulrich Aïvodji
Satoshi Hara
Mario Marchand
Foutse Khomh
Venue: ICLR 2023 poster (2023)
Keywords: Explainability
Robustness
SHAP
Stealthily Sampling

Abstract: SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus 
a background distribution. Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired 
explanations. However, existing attacks focus solely on altering the black-box model itself. In this paper, we propose a complementary family 
of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution. In the context of fairness audit, we show that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups while remaining undetected. More precisely, experiments performed on real-world datasets showed that our attack could yield up to a 90\% relative decrease in amplitude of the sensitive feature attribution. These results highlight the manipulability of SHAP explanations and encourage auditors to treat them with skepticism.

Published as a conference paper at ICLR 2023
FOOL SHAP WITH STEALTHILY BIASED SAMPLING.
Gabriel Laberge', Ulrich Aivodji", Satoshi Hara', Mario Marchand", Foutse Khomh* Polytechhiqqee Montréal, Québec "École de technologie supérieure, Québec $Osaka University, Japan *Universitié de Laval Québec
(gabriel. foutse. khomh @po ymt
ulrich. www.aaieeeiiieeell...... mario. www.mmmarcaaeeieeemmmm
ac. ip
ABSTRACT
SHAP explanations aim identifying which features contribute the most the difference in model prediction at specific input versus background distribution. Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired explanations. However, existing attacks focus solely on altering the black-box model itself. In this paper, we propose complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expecta- tions w.r.t the background distribution. In the context of fairness audit, we show that our attack can reduce the importance the difference in outcomes between groups while remaining undetected. More precisely, experiments performed on real-world datasets showed that our attack could yield up 90% relative decrease in amplitude of the sensitive feature attribution. These results highlight the manipulability of SHAP explanations and encourage auditors treat them with skepticism.
sensitive feature when explaining
INTRODUCTION
As Machine Learning (ML) gets more and more ubiquitous in high-stake decision contexts (e.g. healthcare, finance, and justice), concerns about its potential to lead to discriminatory models are becoming prominent. The use of auditing toolkits (Adebayo et al. 2016 Saleiro al.) 2018 |Bellamy et al. 2018) is getting popular to circumvent the use of unfair models. However, although auditing toolkits can help model designers in promoting fairness, they can also be manipulated to mislead both the end-users and external auditors. For instance, recent study of Fukuci al. (2020) has shown that malicious model designers can produce a benchmark dataset as fake "evvdence" of the fairness of the model even though the model itself is unfair.
Another approach to assess the fairness of ML systems is to explain their outcome in post hoc manner (Guidotti et al. 2018). For instance, SHAP (Lundberg Lee 2017) has risen in popularity as a means to extract model aagnosic local feature attributions. Feature attributions are meant to convey how much the model relies on certain features make a decision some specific input. The use of feature attributions for fairness auditing is desirable for cases where the interest is on the direct impact of the sensitive attributes the output of the model. One such situation is in the context of causal fairness [Chikahara et al. 2021). In some practical cases, the outputs cannot be independent from the sensitive attribute unless we sacrifice much of prediction accuracy. For example, any decisions based on physical strength are statistically correlated to gender due to biological nature. The problem in such a situation not the statistical bias (such as demographic parity), but whether the decision based physical strength gender, i.e. the attributions of each feature.
The focus of this study is on manipulating the feature attributions so that the dependence on sensitive features is hidden and the audits are misled as if the model is fair even if it is not the case. Recently, several studies reported that such a manipulation possible, e.g. by modifying the black-box model to be explained (Slack al. 2020 Begley et al., 2020; Dimanov et al. 2020), by manipulating the computation algorithms of feature attributions (Aīvodji et 2019), and by poisoning the data distribution (Baniecki et al. 2021 Baniecki & Biecek] (022)). With these findings in mind, the current

---

Published as a conference paper at ICLR 2023
possible advice to the auditors is not to rely solely on the reported feature attributions for fairness auditing. A question then arises about what "evidence" we can expect in addition to the feature attributions, and whether they can be valid "evidence" of fairness.
In this study, we show that we can craft fake "evidence" of fairness for SHAP explanations, which provides the first negative answer to the last question. In particular, we show that can produce not only manipulated feature attributions but also a benchmark dataset as the fake "evidence" of fairness. The benchmark dataset ensures the external auditors reproduce the reported feature attributions using the existing SHAP library. In our study, we leverage the idea of stealthily biased sampling introduced by Fukuchi et al. (2020) to cherry-pick which data points be included in the benchmark. Moreover, the use of stealthily biased sampling allows us to keep the manipulation undetected by making the distribution of the benchmark sufficiently close to the true data distribution. Figure] iillustrates the impact of our attack in an explanation scenario with the Adult Income dataset.
Our contributions can be summarized as follows:
Theoretically, we formalize a notion of foreground dis- tribution that can be used to extend Local Shapley Val- ues (LSV) to Global Shapley Values (GSV), which can be used to decompose fairness metrics among the fea- tures (Section 2.2). Moreover, we formalize the task of manipulating the GSV as a Minimum Cost Flow (MCF) problem (Section 4
accipaaion
Marboltted
wendais
Experimentally (Section 5). we illustrate the impact of the proposed manipulation attack synthetic dataset and four popular datasets, namely Adult Income, COM- PAS, Marketing, and Communities. We observed that the proposed attack can reduce the importance of a sensitive feature while keeping the data manipulation undetected by the audit.
-320
-2.00
-2.00
-0.32
3.0
Figure 1: Example of our attack on the Adult Income dataset. After the at- tack, the feature gender moved from the most negative attribution to the 6th, hence hiding some of the model bias.
Our results indicate that SHAP explanations are not robust and can be manipulated when it comes to explaining the difference in outcomes between groups. Even worse, our results confirm we can craft benchmark dataset so that the manipulated feature attributions are reproducible by external audits. Henceforth, we alert auditors to treat post-hoc explanation methods with skepticism even if it is accompanied by some additional evidence.
SHAPLEY VALUES
2.1 LOCAL SHAPLEY VALUES
Shapley values are omnipresent in post-hoc explainability because of their fundamental mathematical properties (Shapley 1953) and their implementation in the popular SHAP Python library (Lundberg Lee 2017). SHAP provides local explanations in the form of feature attributions i.e. given an input of interest ar, SHAP returns score di R for each feature i 1, . d. These scores are meant to convey how much the model f relies on feature i to make its decision f fxx). Shapley values have a long background in coalitional game theory, where multiple players collaborate toward common outcome. In the context of explaining model decisions, the players are the input features and the common outcome is the model output fx)). In coalitional games, players (features) are either present or absent. Since one cannot physically remove an input feature once the model has already been fitted, SHAP removes features by replacing them with a baseline value z. This leads to the Local Shapley Value (LSV) di(J, æ. 2) which respect the so-called efficiency axiom (Lundberg & Lee 2017
d Σ
$1 = -
Simply put, the difference between the model prediction at x and the baseline z is shared among the different features. Additional details the computation of L.SV are presented in Appendix B.1
2

---

Published as a conference paper at ICLR 2023
2.2 GLOBAL SHAPLEY VALUES
LSV are local because they explain the prediction at specific æ and rely on a single baseline input 2. Since model auditing requires more global analysis of model behavior, we must understand the predictions at multiple inputs & J sampled from distribution F. called the foreground. Moreover, because the choice of baseline somewhat ambiguous, the baselines are sampled a distribution B colloquially referred to as the background. Taking inspiration from Begley et al. (2020), we can compute Global Shapley Values (GSV) by averaging L.V over both foreground and background distributions.
B from
Definition 2.1.
(J,F,B) E E. 2)], i= 1,2,
w- ---
Proposition 2.1. The GSV have the following property
Σ (. Tt==,__,K [_]f(a)]-
E [(())))
(3)
=-B
2.3 MONTE-CARL.O ESTIMATES
In practice, computing expectations w.r.t the whole background and foreground distributions may be prohibitive and hence Corlo estimates are used. For instance, when a dataset is used to represent background distribution, explainers in the SHAP library such the Exact Exp 1 ner and TreeExple Exelaa a ner will subsample this dataset by selecting 100 instances uniformly at random when the size of the dataset exceeds 100. More formally, let
C(S, w) = Σ w)) alies
(4)
represent a categorical distribution over a finite set of input examples S, where 8(.) is the Dirac probability measure, W/ 0 Vj. and Σ, wj 1. Estimating expectations with Monte-Carlo amounts sampling M instances
So F.
M
and computing the plug-in estimate
$ So, S1) : +JJ, C(So, 1/M), 1/M))
1
Σ
Σ
Φ(f (1) 20)).
(6)
M
When set of samples a singleton (e.g. S = (2())), we shall use the convention (f,, So, {2(0)) ( So, 2(ا) to improve readability. In Appendix B.2 ( So, Si) is shown be consistent and asymptotically normal estimate of (f B) meaning that one can compute approximate confidence intervals around & to capture & with high probability. In practice, the estimates O are employed as the model explanation which we see as a vulnerability. As discussed in Section 4 the Monte-Carlo estimation the key ingredient that allows manipulate the GSV in favor of a dishonest entity.
3 AUDIT SCENARIO
This section introduces an audit scenario to which the proposed attack of SHAP can apply. This see- nario involves two parties: a company and an audit. The company has a dataset = ((x(0), ( with (6) E Rd and (8) E {0, 1} that contains N input-target tuples and also has a model X' [0. 1] that is meant to be deployed in society. The binary feature with index s (i.e.
T E {0, 1)) represents a sensitive feature with respect to which the model should not explicitly
https://githun
tabuler.

---

Published as a conference paper at ICLR 2023
f1
2 1 [=) [|
E 8
B
f2
f(()
(a) The data initially provided by the company the andit is (Do) and f(;;) f.e the model predictions for
(b) Two models fi and fz (decision boundaries in dashed lines) with perfect accuracy exhibit a dispar- ity in outcomes w.r.t groups with Ia < 0 and x. > 0. Here, b. (J. Hence, is indirectly unfair toward x, because of correlations in the data.
all instances in the private dataset for different values
of r.. This dataset can later be used by the audit to assess whether not the subsets Sp. Si provided by the company where cherry-picked.
-1 1 while (J2 F, B) 0.
Figure 2: Illustrations of the audit scenario.
discriminate. Both the data D and the model f are highly private so the company is very careful when providing information about them to the audit. Hence, f is a black box from the point of view of the audit. At first, the audit asks the company for the necessary data to compute fairness metrics e.g. the Demographic Parity (Dwork et al. 2012), the Predictive Equality
Corbett-Daviee et al.) 2017), or the Equal Opportunity (Hardt et al, 2016). Note that our attack would apply as long as the fairness metric is difference in model expectations over subgroups. For simplicity, the audit decides to compute the Demographic Parity
[[f(=) 01 - B[f(=)/2, = 1].
and therefore demands access to the model outputs for all inputs with different values of the sensitive
feature: ((Do) and f(Di), where Do (x) x (3) 0} and Di (x)(f) x 1} are subsets of the input data of sizes No and N, respectively. Doing so does not force the company to share values of features other than F, nor does it requires direct access to the inner workings of the proprietary model. Hence, this demand respects privacy requirements and the company will accept to share the model outputs across all instances, see Figure 2a At this point, the audit confirms that the model is indeed biased in favor of v, 1 and puts in question the ability of the company to deploy such a model. Now, the company argues that, although the model exhibits a disparity in outcomes, does not mean that the model explicitly uses the feature I, to make its decision. If such is the case, then the disparity could be explained by other features statistically associated with V... Some of these other features may be acceptable grounds for decisions. To verify such claim, the audit decides to employ post-hoc techniques to explain the disparity. Since the model is a black-box, the audits shall compute the GSV. The foreground F and background B are chosen to be the data distributions conditioned on x, = O and v. = 1 respectively
= C(Do, 1/No)
B: (-=C(D,1/NN))
(8)
According to Equation 3 the resulting GSV will sum up to the demographic parity (cf. Equation 7). If the sensitive feature has a large negative GSV 4., then this would mean that the model is explicitly relying on 2, to make its decisions and the company would be forbidden from deploying the model. If the GSV has a small amplitude, however, the company could still argue in favor of deploying the model in spite of having disparate outcomes. Indeed, the difference in outcomes by the model could be attributed to more acceptable features. See Figure 2b for a toy example illustrating this reasoning.
To compute the GSV, the audit demands the two datasets of inputs Du and D1. well the ability to query the black box f at arbitrary points. Because of privacy concerns on sharing values of æ across the whole dataset, and because GSV must be estimated with Monte-Carlo, both parties agree that the company shall only provide subsets So = Do and S₁ c D2 of size M to the audit so they can compute a Monte-Carke estimate ( So S1). The company first estimate GSV on their own by choosing So, S₁ uniformly at random from F and B (cf. Equation 5) and observe that $ indeed has
4

---

Published as a conference paper at ICLR 2023
large negative value. They realize they must carefully select which data points will be sent, otherwise, the audit may observe the bias toward I, = 1 and the model will not be deployed. Moreover, the company understands that the audit currently has access to the data f(D)) and f(D,) representing the model predictions on the whole dataset (see Figure 2a). Therefore, if the company does not share subsets So S1 that were chosen uniformly random from Do, D1, it is possible for the audit to detect this fraud by doing a statistical test comparing f(Sn) to f(Do) and f(Si) to f(D)). The company needs method to select misleading subsets Sy Sy whose GSV is manipulated in their favor while remaining undetected by the audit. Such method is the subject of the next section.
4 FOOL. SHAP WITH STEALTHILY BIASED SAMPLING
4.1 MANIPULATION
To fool the audit, the company can decide to indeed sub-sample S% uniformly at random Sy FMM Then, given this choice of foreground data, they can repeatedly sub-sample Si ~ Bha ~5* and choose the set S leading to the smallest [ Sy S.) We shall call this method "brute-force.. Its issue that, by sub-sampling S from B, it will take an enormous number of repetitions reduce the attribution since the GSV (f, S1) is concentrated on the population GSV F...).
A more clever method is to re-weight the background distribution before sampling from it i.e. define 8", := C(D1,w) with w * 1/N and then sub-sample Sj ~ BtM To make the model look fairer, the company needs the क. computed with these therry-picked points to have small magnitude.
Proposition 4.1. Let Sy be fixed, and let Is represent convergence in probability as the size M of the sei S ~ B.M increases, we have
(., 81) P.
Σ gGEDD
w (6.
We note that the coefficients (f, 2()) in Equation 9 are tractable and can be computed and stored by the company. We discuss in more detail how to compute them in Appendix [B.]] An additional requirement is that the non-uniform distribution K", remains similar to the original B. Otherwise, the fraud could be detected by the audit. In this work, the notion of similarity between distributions will be captured by the Wasserstein distance in output space.
Definition 4.1 (Wassertein Distance). Any probability measure π over D1 D2 called a coupling measure between B and B',, denoted т E A(B, B.)), if i/N1 = E> Tig and wj - Fag. The Wassertein distance between B and B' mapped to the output-space is defined
W(B,B'))
Σi((20) - min (EA(B, bi
(1(((0)-((zb))) {(=(0) (m:j)
(10)
a.k.a the cost of the optimal transport plan that distributes the mass from one distribution to the other.
We propose Algorithm to compute the weights w by minimizing the magnitude of the GSV while maintaining small Wasserstein distance. The trade-off between attribution manipulation and proximity to the data is tuned via a hyper-parameter > >> O. We show in the Appendix A.2 that the optimization problem at line 5 of Algorithm can be reformulated as Minimum Cost Flow (MCF) and hence can be solved in polynomial time (more precisely Ö(N2-5) as in Fukuchi al. (0200).
4.2 DETECTION
We now discuss ways the audit can detect manipulation of the sampling procedure. Recall that the audit has previously been given access to f (D))) f(D)) representing the model outputs across all instances in the private dataset. The audit will then be given sub-samples S% S of Do, Dj on which they can compute the output of the model and compare with (Dii. To assess whether or not the aub-samples provided by the company were sampled uniformly at random, the audit has to conduct statistical tests. The null hypothesis of these tests will be that S6. Si were sampled uniformly at random from D,, Dr. The detection Algorithm with significance uses both the Kolmogorov- Smirnov and Wald tests with Bonferonni corrections (i.e. the a/4 terms in the Algorithm). The Kolmogorov v and Wald tests are discussed in more detail in Appendix C
5

---

Published as a conference paper at ICLR 2023
Algorithm I Compute non-uniform weights
1: procedure COMPUTE, WEIGHTS(D1, 1b, (f, So x =01));
2:
B sign ( ((,
B
C(D1,1/N1)
Unmanipulated background
4: B == ()
t- Manipulated background as function of w
5: arg min $ E Wy S z11) AW(B, B²))
Optimization Problem
6: return w;
Algorithm 2 Detection with significance a
1: procedure DETECT F.A
M)
2:
for 0, do
3:
f(S:) 1/NN) M
Subsample without cheating. KS test comparing ((S)) and ((S) t> Wald test
4:
p-value-KS KS( (S.)) ((S)
5:
p-value-Wale Wald = Wald( (()),, ((.))
6:
ifp-value-KS < a/4 orp-value- Wald < a/4 then
t Reject the null hypothesis
7:
return
8:
return 0;
4.3 WHOLE PROCEDURE
The procedure returning the subsets Sy Si is presented in Algorithm 3 It conducts a log-space search between Amin and Amax for the À hyper-parameter (line 6) in order to explore the possible attacks. For each value of X, the attacker runs Algorithm I to obtain B' (line 7), then repeatedly samples S ~ BB (line 10) and attempts to detect the fraud (line 11). The attacker will choose B that minimizes the magnitude of C. while having detection rate below some threshold (line 12). An example of search over on a real-world dataset presented in Figure
Algorithm 3 Fool SHAP
procedure FOOL SHAP(F. Do. D1, M, Amin Amaxs T, a.)
2: 5% C(D,, 1/No)
Sg is sampled without cheating
3: Compute (.. St 20) Vz(i) E D1
p> cf. Section B.3
4:
=C(D,, 1/N1)
5:
Φ = 1/N1 ExmeD, Sf,
F- Initialize the solution
6: for Amax ..... Amin do
7:
w = COMPUTE_ SEIPUTT_WIIGTS((D) ( S/: 2()))}
8:
Detection 0 for rep 1 1...., 100 do
9:
Detect the manipulation
10:
S BB
Detection DETECT_ DDTTT
M)
12:
EgDDDD Wj &. 2()) [ and Detection < 100r then
13:
B = B
14:
$ Σ
LDDeD, Wj (f. 20))
> Update the solution
15:
S B
t Chherry-pick by sampling from the non-uniform background
return S6 Si
One limitation of Fool SHAP is that it manipulates a single sensitive feature. In Appendix E.4 we present possible extension of Algorithm to handle multiple sensitive features and present preliminary results of its effectiveness. A second limitation is that it only applies to "interventional" Shapley values which break feature correlations. This choice was made because most methods the SHAP library? "interventional". Future work should port Fool SHAP "observational" Shapley values that use conditional expectations to remove features (Frye et al. 2020
except the kxpe her when background data is provided

---

Published as a conference paper at ICLR 2023
100
0.00
Deloction Rate (%)
0.02
60
GSV
-0.04
40
-0.06
20
Sensitive feature
00.08
Other features
10
10
10a
10P
10
10
Figure 3: Example of log-space search over values of À using an XGBoost classifier fitted on Adults. (a) The detection rate as a function of the parameter À of the attack. The attacker uses a detection rate threshold 5 10%. (b) For each value of the vertical slice of the curves is the GSV obtained with the resulting B' The goal here is to reduce the amplitude of the sensitive feature (red curve).
4.4 CONTRIBUTIONS
The first technique to fool SHAP with perturbations of the background distribution was genetic algorithm Baniecki Biecek (022). Although promising. the cross-over and mutation operations it employs to perturb data do not take into account feature correlations and can therefore generate unrealistic data. Moreover, the objective to minimize does not enforce similarity between the original and manipulated backgrounds. We show in Appendix E.3 that these limitations lead to systematic fraud detections. Hence, our contributions are two-fold. First, by perturbing the background via non- uniform weights pre-existing instances (i.e. B C(D1, w) ) rather than genetic algorithm, we avoid the issue of non-realistic data. Second, by considering the Wasserstein distance, we can control the similarity between the original and fake backgrounds.
Since the Stealthity Biased Sampling technique introduced in Fukuchi et al. (2020) also leverages a non-uniform distribution over data points and the Wasserstein distance, makes sense adapt it to fool SHAP. Still, the approach of Fukuchi et al. is different from ours. Indeed, in their work, they minimize the Wasserstein distance while enforcing a hard constraint the number of instances that land on the different bins for the target and sensitive feature, That way, they can set the Demographic Parity any given value while staying close the original data, In our setting of manipulating the model explanation, we leave the Demographic Parity intact and instead manipulate its feature attribution. terms of the optimization objective, we now minimize Shapley value with a soft constraint the Wasserstein distance.
EXPERIMENTS
5.1 TOY EXPERIMENT
The task is predicting which individual will be hired for a job that requires carrying heavy objects. The causal graph for this toy data is presented in Figure (left). We observe that sex (S) influences height (H), and that both these features influence the Muscular Mass (M). In the end, the hiring decisions (Y) are only based on the two attributes relevant the job: H and M. Also. two noise features N1, N2 were added. More details and justifications for this causal graph are discussed in Appendix D.I Since strength and height (two important qualifications for applicants) are correlated with sex, any model ff that fits the data will exhibit some disparity in hiring rates between sexes. Although, if the model decisions do not rely strongly on feature S, the company can argue favor of deployment. GSV are used by the audit to measure the amount by which the model relies on the sex feature, see Figure 4 (Middle). By employing Fool SHAP with M = 100, the company can reduce the GSV of feature S considerably compared to the brute-force and genetic algorithms. More importantly, the audit is not able to detect that the provided samples S, S were cherry- picked, see Figure (Right). More results are presented in Appendix E.
7

---

Published as a conference paper at ICLR 2023
Origial
1.0
Genetic
0.8
Fool SHAP
((D.)
0.6
((S))
Muscle
((Dg)
0.4
((S))
Height
0.2
0.0
Output
Figure 4: Toy example. Left: Causal graph. Middle: GSV for the different attacks with M 100. Right: Comparison of the CDF of the Fool SHAP subsets f(S)). f(S) and the CDF over the whole data ((D)). Here the audit cannot detect the fraud using their detection algorithm.
5.2 DATASETS
We consider four standard datasets from the FAccT literature, namely COMPAS, Adult-Income, Marketing, and Communities.
COMPAS regroups 6, 6,150 records from criminal offenders in Florida collected from 2013-2014 This binary classification task consists in predicting who will re-offend within two years. The and I, = 1 for Caucasian. Adult Income contains demographic attributes of 48,842 individuals from the 1994 U.S. census. It is binary classification problem with the goal of predicting whether or not a particular person makes more than 50K USD per year. The sensitive feature in this dataset is gender, which took values
sensitive feature is race with values
for
for female, and X. = for male.
Marketing involves information on 41,175 customers of Portuguese bank and the binary classifi- cation task is to predict who will subscribe to a term deposit. The sensitive attribute is age and took values x, 0 for age 30-60, and 2, for age no:30-60
Communities & Crime contains per-capita violent crimes for 1994 different communities in the US. The binary classification task is to predict which communities have crimes below the median rate. The sensitive attribute is PercentWhite and took values 2, = for PercentWhite<33 and
for PercentWhite>>9005
Three models were considered for the two datasets: Multi-Layered Perceptrons (MLP), Random Forests (RF), and eXtreme Gradient Boosted trees (XGB). One model of each type was fitted on each dataset for different train/test splits seeds, resulting in models total. Values the test set accuracy and demographic parity for each model type and dataset are presented in Appendix D.2
5.3 DETECTOR CALIBRATION
Detector calibration refers to the assessment that, assuming the null hypothesis to be true, the prob- ability of rejecting it (i.e. false positive) should be bounded by the significance level a. Remem- ber that the null hypothesis of the audit detec- tor is that the sets Sy Si provided by the com- pany are sampled uniformly from Do. Dj. Hence, to test the detector, the audit can sample their own subsets f(So), uniformly from at ran- dom from f( run the detection algo- rithm, and count the number of detection over 1000 repeats. Table [ shows the false positive rates the five train-test splits using signifi- cance level a 5%. We observe that the false positive rates are indeed bounded by a for all model types and datasets implying that the detector employed by the audit is calibrated.
Table 1:
False Positive Rates (%) of the de- tector i.e. the frequency at which So. S are considered cherry-picked when they are not. No rate should be above 5%.
mlp
rf
xgb
COMPAS
4.0
4.6
4.0
Adult Marketing
4.3
4.3
4.2
4.9
5.0
Communities
3.8
4.2
8

---

Published as a conference paper at ICLR 2023
5.4 ATTACK RESULTS AND DISCUSSION
The first step of the attack (line of Algorithm 3 requires that the company run SHAP their own and compute the necessary coefficients to run Algorithm [ For the COMPAS and Adults datasets, the Exact Explaaeeeeieeeeee Explainer of SEAP was used.
Relative Amplitude Decresse (%)
100
Brute
Genetic
Since Marketing and Communities contain more than 15 features, and since the ExpctExplalle lainer scales exponentially with the number of features, we were restricted to using the TreeExpla ner (Lundberg et al., 2020) on these datasets. The
60
Ours
20-
TreeExplainer avoids the exponential cost of
Shapley values but is only applicable to tree-based models such RFs and XGBs. Therefore, we could not conduct the attack on MLPs fitted on Marketing and Communities.
COMPAS
Adult Dalases
Markoting
C&&C
Figure 5: Relative decrease in amplitude of the sensitive feature attribution induced by the various attacks on SHAP.
The following step is to solve the MCF for various
values of X (line of Algorithm 3) As stated previ- ously, solving the MCF can be done in polynomial time in terms of N1, which was tractable for a small dataset like COMPAS and Communities, but for larger datasets like Adult and Marketing. To solve this issue, as was done in Fukuchi et al. (2020), we compute the manipulated weights multiple times using 5 bootstrap sub-samples Dj of size 2000 to obtain set of weights wا=l, w1e),, ..... [15)] which we average obtain the final weights w.
Results of 46 attacks with M 200 are shown in Figure Specific examples of the conducted attacks presented Appendix E.2 As point of reference, we also show results for the brute-force and genetic algorithms. To make comparisons to our attack more meaningful, the brute-force method was only allowed to run for the same amount of time it took search for the non aon-uniform weights w (about 30- 90-180 seconds). Also, the genetic algorithm ran for 400 iterations and was stopped early there were 10 consecutive detections. We note that, across all datasets, Fool SHAP leads to greater reductions of the sensitive feature attribution compared to brute-force search and the genetic perturbations of the background.
Now focusing on Fool SHAP, for the datasets COMPAS and Marketing, we observe median reductions in amplitudes of about 90%. This means that our attack can considerably reduce the apparent importance of the sensitive attribute. For the Adult and Communities datasets, the median reduction in amplitude is about 50% meaning that we typically reduce by half the importance of the sensitive feature. Still, looking the maximum reduction in amplitude for Adult-Income and Communities, we note that one attack managed to reduce the amplitude by 90%. Therefore, luck can play a part in the degree of success of Fool SHAP, which is to be expected from data-driven attacks.
Finally, the audit was consistently unable to detect the fraud using statistical tests. This observation raises concerns about the risk that SHAP explanations can be attacked to return only manipulated attributions but also non-detectable fake evidence of fairness.
CONCLUSION
To conclude, we proposed a novel attack on Shapley values that does not require modifying the model but rather manipulates the sampling procedure that estimates expectations w.r.t the background distribution. We show on a toy example and four fairness datasets that our attack can reduce the importance of sensitive feature when explaining the difference in outcomes between groups using SHAP. Crucially, the sampling manipulation is hard to detect by an audit that is given limited access to the data and model. These results raise concerns about the viability of using Shapley values to assess model fairess.. We leave as future work the use of Shapley values to decompose other faimess metrics such as predictive equality and equal opportunity. Moreover, we wish to move cases beyond fairness, as we believe that the vulnerability of Shapley values that was demonstrated can apply many other properties such safety and security.

---

Published as a conference paper at ICLR 2023
7 ETHICS STATEMENT
The main objective of this work is to raise awareness about the risk of manipulation of SHAP explanations and their undetectability. As such, it aims exposing the potential negative societal impacts of relying on such explanations. It remains however possible that malicious model producers could use this attack mislead end users or cheat during an audit. However, we believe this paper makes significant step toward increasing the vigilance of the community and fostering the development of trustworthy explanations methods. Furthermore, by showing how faimess can be manipulated in explanation contexts, this work contributes to the research on the certification of the fairness of automated decision making systems.
REPRODUCIBILITY STATEMENT
The source code of all experiments is available online Moreover, experimental details provided in appendix D.2 for the interested reader.
REFERENCES
Julius A Adebayo et al. Fairml: Toolbox for diagnosing bias predictive modeling. Master's thesis,
Massachusetts Institute of Technology, 2016. [
Ulrich Alvodji, Hiromi Arai, Olivier Fortineau, Sébastien Gambs, Satoshi Hara, and Alain Tapp- Fairwashing: the risk of rationalization. In International Conference on Machine Learning, pp. 161-170. PMLR, 2019. !
Hubert Baniecki and Przemyslaw Biecek. Manipulating shap via adversarial data perturbations (student abstract). 2022.
Hubert Baniecki, Wojciech Kretowicz, and Przemyslaw Biecek. Fooling partial dependence via data poisoning. arXiv preprint arXiv:2105.12837, 2021.
[
25
Tom Begley, Tobias Schwedes, Christopher Frye, and llya Feige. Explainability for fair machine
learning. arXiv preprint arXiv:20100.77389, 2020.
Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, al. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint
2018.
Yoichi Chikahara, Shinsaku Sakaue, Akinori Fujino, and Hisashi Kashima. Learning individually fair classifier with path-specific causal- efffect constraint. In International Conference on Artificial Intelligence and Statistics, pp. 145-153. PMLR, 2021. [
Sam Corbett-Davies Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 797-806, 2017.
Botty Dimanow, Umang Bhatt, Mateja Jamnik, and Adrian Weller. You shouldn't 't trust me: Learning models which conceal unfairness from multiple explanation methods. In SafeAI@ AAAI, 2020.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference, pp. 214-226, 2012.
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. arXiv preprint XXix:2006.01272. 2020. 6
Kazuto Fukuchi, Satoshi Hara, and Takanori Maehara. Faking fairness via stealthily biased sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 412-412444 412-419, 2020.
7
9
https: twwithhubbc.m htt.bb.cco//bbblblooaaooo abcp//ac///ooo
10

---

Published as a conference paper at ICLR 2023
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):1-42, 2018. [
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning, Advances in neural information processing systems, 29, 2016.
Ian Janssen, Steven B Heymsfield, ZiMian Wang, and Robert Ross. Skeletal muscle mass and distribution 468 men and women aged 18-88 88 yr. Journal of applied physiology, 2000. 20
A J Lee. U-statistics: Theory and Practice. Routledge, 2019. 18
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017. 2
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable
trees. Nature machine intelligence, 2(1):56-67, 2020.
17
Frank Massey Jr. The kolmogorov-smmrnov test for goodness of fit. Journal of the American
statistical Association, 46(253):68-78 1951. 19
Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. Aequitas: A bias and fairness audit toolkit. arXiv preprint arXiv:1811.05577, 2018. [
Lloyd Shapley. value for n-person games. Contributions to the Theory of Games, pp. 307-317, 1953. 2
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference AI, Ethics, and Society, pp. 180-186, 2020. T
Larry Wasserman. All of Statistics: concise course statistical inference. Springer, 2004. 13

---

Published as a conference paper at ICLR 2023
A PROOFS
A.1 PROOFS FOR GLOBAL SHAPLEY VALUES (GSV)
Proposition A.I (Proposition 2.1 The GSV have the following property
(f [_[(()]]
E [f(x))
(1I)
Proof. As a reminder, we have defined the vector
DJ.F,B)
E E =)]
(12)
a=t B
whose components sum up
d
d Σ
F,
Tf x,
(13)
R~ T
E Σ (J),
(14)
2-8
E Elf(() f(z)]
(15)
B 2-B
E [[(() E [[(()
(16)
E a [[([] E [.()()))
(17)
where at the last step we have simply renamed a dummy variable.
Proposition A.2 (Proposition
4.1
Let Sy be fixed, and let Es represent convergence in probability increases, then we have
as the size M of the set S ~ B T
N Σ
(J, S1 P.
Wy
(18)
Proof.
$().
Σ
Σ
M2
(f)ee ziiES
Σ
Σ
Φ( 20!) 2(0))
(19)
M
M
20EES
Σ g
(f, S z0))
M
Since Sy is assumed to be fixed, then the only random variable in @. S. 200 is 20ح which represents instance sampled from the B'. Therefore, define (12) $. S 2) and get
$.(1,S),
1
Σ
$. 2011
M 20)es
(20)
1
Σ
(2(1))
with S ~ AM
M
12

---

Published as a conference paper at ICLR 2023
By the weak law of large number, the following holds M goes infinity (Wasserman 2004 Theorem 5.6)
Σ (( B E M x
1
(21)
Now, as reminder, the manipulated background distribution is B² = C(D1, with w * 1/M1. Therefore
(f. 8i) E E [x))]]
2 2-B z-B
=- [bx
w)b((111
(22)
wy
concluding the proof.
13

---

Published as a conference paper at ICLR 2023
A.2 PROOFS FOR OPTIMIZATION PROBLEM
A.2.1 TECHIICAL LEMMAS
We provide some technical lemmas that will be essential when proving Theorem A.1 These are presented for completeness and are not intended contributions by the authors. Let us first write the formal definition of the minimum of function.
Definition A.1 (Minimum). Given some function f D - R, the minimum of f over D (denoted f*) is defined as follows:
min f(x) 3x E D s.t. f(x') < f(x) E D.
Basically, the notion of minimum coincides with the infimum inf ((D) (highest lower bound) when this lower bound is attained for some x² E D. By the Extreme Values Theorem, the minimum always exists when D compact and f is continuous. For the rest of this appendix, we shall only study optimization problems where points the domain set D {(z, y) E X, V's ) can be selected by the following procedure
1. Choose some x X
2. Given the selected æ, choose some y E V c: V, where the set 32 is non-empty and depends the value of x.
When optimizing functions these domains, following lemma.
optimize in two steps highlighted in the
Lemma A.1. Given a compuct domain D of the form described above and a continuous objective function f D R, the minimum f is attained for some (x* y²) and the following holds
min f(x, v) min min f(x, v). (a.a)eD
aex ye
Proof. Let f(xx) == infyer, fa, v). which is well defined function on X. We can then take its infimum f infaex f(x). But is f an infimum of f(D)? By the definition of infimum
f F(x)
VzE
inf f(z,y)
vel.
f(z v)
Vye V,.
that f" is lower bound of J(D). In fact, is the highest lower bound possible
inf f(x,z) inf inf f(z, v). (x.))e
(23)
yE
By the Extreme Value Theorem, since D is compact and f is continuous, there exists (x", yy) D s.t. f* inf (z.w)ED f(x y) (x,g)e) f(x y) = f(x" ,("""""). Since the infimum is attained on the left-hand-sick of Equation 23 then it must also attained we can replace all inf with min in Equation 23 leading to the desired result.
the right-hand-side and therefore
Lemma A.2. Given a compact domain D of the form described above and two continuous functions h: X I. and
IR, then
((xz) ((e++s())
min (x.0)eD
min ZER'
[(xx)) + min g(v)
yEy'
Proof. Applying Lemma
A.1 with the function f(x,y) u) = h(x)) g(y) proves the Lemma.
14

---

Published as a conference paper at ICLR 2023
a(e) Al5(2 - 1(00))
ej
<Cc) s
Ti
f(e) = Rid
ale) (f, S6: 2 c(e)
afe) 0 dle)
f(e) 3j
f(e)
8
t
Figure 6: Graph G on which we solve the MCF, Note that the total amount of flow is d = N1 and there are N₁ left and right nodes l,, Ti.
A.2.2 MINIMUM COST FLOWS
Let G (V.E) be graph with vertices v with directed edges e & V v. R be capacity and a - R be a cost. Moreover, let E & be two special vertices called the source and the sink respectivelly, and de I be total flow. The Minimum-Coss Flow (MCF) problem of G consists of finding the flow function f R+ that minimizes the total cost
min f
Σ ((e)
f(e) c(l)) VeE
(24)
We VEV (s,t) u ==8
S(e) s(e) Σ ((e) ceut
d
EEG
d
=
where ut {u, v) E and -- ((u, u) E) are the outgoing and incoming edges from u. The terminology of flow arises from the constraint that, for vertices that are not the source nor the sink, the outgoing flow must equal the incoming one, which is reminiscent of conservation laws in fluidic. We shall refer to f(u, v)) the flow from to
Now that we have introduced minimum cost flows, let us specify the graph that will be employed to manipulate GSV, see Figure 6 We label the flow going from the sink to of the left vertices as W Wi x N1, and the flow going from & to Tú as 7., = Tef x N1. The required flow is fixed at d N1.
15

---

Published as a conference paper at ICLR 2023
Theorem A.1. Solving the MCF of Figure O solution of the linear program in Algorithu !
Proof. We begin by showing that the flow conservation constraints in the MCF imply that ι is coupling measure (i.e. ा E ABB, )), and w is constrained to the probability simplex A(N1). Applying the conservation law on the left-side of the graph leads to the conclusion that the flows entering vertices Ej must sum up to Ni
Σ W N1.
This implies that w is must be part of the probability simplex. By conservation, the amount of flow that leaves specific vertex l must also be WJ hence
Σ Mie Wy-
For any edge outgoing from r; to the sink i, the flow must be exactly This because we have N₁ edges with capacity e(e) = 1 going into the sink and the sink must receive an incoming flow of N₁. As consequence of the conservation law on a specific vertex rs, the amount of flow that goes into each is also
Σ R 1.
Putting everything together, from the conservation laws on G, we have that w E ( and E A(B, B²,). Now, make the parallel between the MCF and Algorithm I must use Lemma A.2 Note that w is restricted to the probability simplex, while π is restricted to be a coupling measure. Importantly, the set of all possible coupling measures A (B, B8,) is different for each w because B depends on w. Hence, the domain has the same structure as the ones tackled in Lemma A.2 (where
A(N))) and y E V'2 becomes E A(B, B3))). Also, the set of possible w and bounded simplex in (N)++1) so it is compact, and the objective function of the MCF is linear, thus continuous. Hence, we can apply the Lemma A.2 to the MCF.
. É X becomes
N
min Σ f(e)a(e)= = min WAA
Σ
B D. (fr S. 2 20) + A
Bes
>
M -
330
min
N1
3
2 (J, 2 20)) + A
म {t)))((00) - s(z61)))
N1
N
N min
W a s (J, S/ 21
Tij
[[(=(0) ,(=(0))
N1
N₂
N1
min
3
Wy w To (f, Sy z 20) + x Σ
x -)) - -(((())))
(EA(
==1
2,j
min
h(w) g(x))
N₁ min wE)(N:)
h((w)
rE)(B,B)) g(TT)
min
(cf. Lemma A.2
N₁ min wE)(N))
3
w/ $
(F. U6, 56, x()) +
min
[xssl(((()) -(((00)))
mEA(B,
Ni min
3
Σ
w/ $ (f. 2(01 AW(B BL)
WEA(N))
which (up multiplicative constant N1) is solution of the linear program of Algorithm
16

---

Published as a conference paper at ICLR 2023
B SHAPLEY VALUES
B.1 LOCAL SHAPLEY VALUES (LSV)
We introduce Local Shapley Values (LSV) more formally. First, as explained earlier, Shapley values are based on coalitional game theory where the different features work together toward a common outcome f f() game, the features can either be present or absent, which is simulated by replacing some features with baseline value 2.
Definition B.1 (The Replace Function). Let x be an input of interest æ, S {1, (1.2...., d} be a subset of input features that considered active, and be baseline input, then the replace-function rs R. x R - IR is defined as
rs(2, 2: = ai if
(25)
otherwise.
We note that this function is meant to "activate" the features in S.
Now, if let be random permutation of d features, and n, denote all features that appear before i in R,, the LSV are computed via
E [f(r [((m)(1)22, T~
f( (2, 1,2, 1,2,...,d4
(26)
where 52 is the uniform distribution over d! permutations. Observe that the computation of LSV is scales poorly with the number of features hence model-agnostic computations are only possible with datasets with few features such COMPAS and Adult-Income. For datasets with larger amounts of features the TreeExplaine
algorithm (Lundberg et al. (2020) can be used to compute the LSV (cf. Equation 26 polynomial time given that one explaining tree-based model.
B.2 CONVERGENCE
As reminder, we are interested in estimating the GSV + (JJFF which requires estimating expectations w.r.t the foreground and background distributions. Said estimations can conducted with Monte-Carlo where we sample M instances
So FM
S ~ MMM
(27)
and compute the plug-in estimates
$ (f. S1) == ΦfJ, C(S,, 1/M), C(S,, 1/M))
1 Σ Σ
Φlf ((fxx0) 20)).
(28)
M
w z
We now show that, Sc, S1) is consistent and asymptotically normal estimate of $(f.F, B) Proposition B.1. Let f X - [0, 1] be a black box, F and be distributions on X, and & = (f, S2) be the plug-in estimate of s = (f, F, B), the following holds for any E Jo, 1[ and =1,2...,
F FOOD (1 - 8/2)
lim P @6x-
Del
V Voo
o0 =6,
2 M
where FN is the inverse Cumulative Distribution Function (CDF) of the standard normal distribution, o = V x))]] and o V V.-
Proof. The proof consists simply in noting that LSV de(.,, (1)) -200))0)) 2(1) are function of two indepen- dent samples (+() ~ F and 2() B. The model f is assumed fixed and hence for any feature k we can define 1) x()) = dx ( 21)). Now, the estimates of GSV can be rewritten
( Si)
Σ
Σ
2())),
(29)
|Su S
aite 2i)sS
17

---

Published as a conference paper at ICLR 2023
which recognize well-known class of statistics called two-samples U-statistics. Such statistics are unbiased and asymptotically normal estimates of
4 F, E [1(x,)))
(30)
Φー 2-8
The asymptotic normality of two-samples U-statistics is characterized by the following Theorem (Lee 2019, Section 3.7.1).
Theorem B.1. Let T. = ( So, S1) be a two-samples U-statistic with |So] N, 1S = M, moreover let h(x, 2) have finite first and second moments, then the following holds for any d Jo, 1L
Fx (1 8/2)
2
2
lim N+M NLNNN++M)--(0..
P 10x
4x
8,
VM N
p
p
where 20 V Va-= s[h(2, 2]] and of V. Eg h((:; z)]].
Proposition B.1 follows from this Theorem by choosing N = M. = 0.5 and noticing that having
a model with bounded outputs (ʃ X [0, 1]) implies that |M(,r, =) < Var E X which means that ((x, z) has bounded first and second moments.
B.3 COMPUTE THE LSV
Running Algorithm requires computing the coefficients (f Sy 00) for j = .,,,. To compute them, first note that they can be written in terms of LSV for all instances in Si
Σ 20E
$.l. ( z())
(31)
M
The LSV ax 200) computed deeply in the SHAP code and are directly accessible using the current API. Hence, we had to access them using Monkey-Patchingg i.e. we modified the ExactExplaines class so that stores the LSV one of its attributes. The attribute can then be accessed seen in Figure 7 The code is provided as fork the SHAP repository. For the TreeExpla ner, because its source code C++ and wrapped in Python, we found it simpler to simply rewrite our own version of the algorithm in C++ that it directly returns the LSV, instead of Monkey-Patching the TreeExplainer.
# Mask features using the whole background distribution Independent max samp sa 1)) explainer ssap.
predict predict_proba,, mask)
# Explain all instances sampled fron the foreground
explainer(S 8)
# The LSV are extracted with Monkey-Patchine
LSV explainer.LSH
..V.ehapp- (n features, 05_0], [D
Phi S LSV.mean (1).T
Phi 5 shape (ID_1l, n features)
Figure 7: How we extract the LSV from the Exact Explainen ainer via Monkey-Patchine
18

---

Published as a conference paper at ICLR 2023
C STATISTICAL TESTS
C.1 KS TEST
first test that can be conducted is two-samples Kolmogorov-Smirno (KS) Masssyy Jr If we let
1951
Fs(())
1
Σ
1( & x)
(32)
[
be the empirical CDF observations in the set S. Given two sets and S the KS statistic is
KS(S, Supleste (x)(
(33)
FeR
Under the null-hypothesis Ho ~ DI ~ D for some univariate distribution D, this statistic is expected to not be too large with high probability. Hence, when the company provides the subsets Sg, S the audit can sample their own two subsets [(S) uniformly at random from f(Do), (1) and compute the statisties KS(f(So), f(S)) and f(S) to detect a fraud.
C.2 WALD TEST
An alternative is the Wald test, which is based on the central limit theorem. If S1 BM then the empirical average of the model output over S₁ is asymptotically normally distributed as increases
Exe((s))> -- 1
Wald(
37
NN 1),
(34)
O/VM
where := [2) and := V2- are the expected value and variance of the model output across the whole background. The same reasoning holds for Sg and the foreground F. Applying the Wald test with significance would detect fraud when
((5,, > F a/2),
(35)
where (o(0..) is the inverse of the CDF of a standard normal variable.
19

---

Published as a conference paper at ICLR 2023
D METHODOLOGICAL DETAILS
D.1 TOY EXAMPLE
The toy dataset was constructed to closely match the results of the following empirical study compar- ing skeletal mass distributions between men and women (Janssen et al. 2000). Firstly, the sex feature was sampled from Bernoulli
S ~~ Bernoulli(0.5).
(36)
According to Table of Janssen et al. (2000), the average height of women participants was 163 em while it was 177cm for men. Both height distributions had the same standard deviation of 7cm. Hence we sampled height via
H = man (177, H woman NV663, 49)
(37)
It was noted in Janssen et al. (2000) that there was approximately a linear relationship between height and skeletal muscle mass for both sexes. Therefore, we computed the muscle mass M as
M H h, S man) 0.186h 5e
M| H = h, S= woman) = 0.128h + 4e
(38)
with 1)
The values of coefficients 0. 0. 128 and noise levels 5 and 4 were chosen so the distributions of MJS would approximately match that of Table in Janssen et al. (2000). Finally the target was chosen following
YKH h1 M = m) Bernoulli P(H, M))
(39)
with P( = [1 + 100 x 1(H < 160) - 0.3 28)] 1-t
Simply put, the chances of being hired the past (Y) were impossible for individuals with smaller height than 160cm. Moreover, individuals with higher mass skeletal mass were given more chances to be admitted. Yet, individuals with less muscle mass could still be given the job if they displayed sufficient determination. In the end, generated 6000 samples leading to the following disparity in
Y.
P(Y 1|S man) 0.733
IP 1|5= = woman) 0.110.
(40)
20

---

Published as a conference paper at ICLR 2023
Table 2: Models Test Accuracy % (mean stddev).
mlp
rf
xgb
COMPAS
68.2 + 0.9
67.7 = 0.8
68.6 + 0.8
Adult
85.6 + 0.3
86.3 - 0.2
87.1 + 0.1
Marketing
91.1 0.1
91.4 + 0.3
Communities
83 2
82 + 2
Table 3: Models Demographic Parity (mean stddev).
mip
xgb
COMPAS
-0.12 0.01
-0.11 - 0.01
-0.11 0.02
Adult
-0.20 + 0.01
-0.19 0.01
-0.192 + 0.004
Marketing
-0.104
-0.11 0.01
Communities
-0.50 0.01
-0.54 0.02
D.2 REAL DATA
The datasets were first divided into train/test subsets with ratio 5 The models were trained on the training set and evaluated on the test set. All categorical features for COMPAS, Adult, and Marketing were one-hot-encoded which resulted in a total of 11, 40, and columns for each dataset respectively. A simple 50 steps random search was conducted to fine-tune the hyper-parameters with cross-validation on the training set. The resulting test set performance and demographie parities for all models and datasets, aggregated over 5 random data splits, reported in Tables 2 and respectively.
21

---

Published as a conference paper at ICLR 2023
1.0
0.075
0.070
0.8
((Di)
Anplitdde
0.066
Detretion
CDF
0.6
F(SD
0.060
((Db)
0.085
0.4
((S)
0.050
0.2
0.2
0.046
100
120
0.0
Iterationss
0.0
0.2
0.4
0.6
0.8
1.0
Output
(a) Iterations of genetic algorithm.
(b) CDFs for genetic algorithm.
1.0
0.8
0.8
f(D.)
F(D))
CDF
0.6
JUD
0.6-
JSS
f(Db)
f(Ds)
0.4
0.4
((S)
0.2
0.2-
0.0
0.8
1.0
0.0
0.2
1.0
Output
Output
(c) CDFs for brute-force
(d) CDFs for Fool SHAP
Figure 8
E ADDITIONAL RESULTS
E. TOY EXAMPLE
Figure presents additional results for the toy example. More specifically, Figure (a) illustrates the evolution of the detection and amplitude of the sensitive feature during the genetic algorithm. We note that beyond 90 iterations, the detector is systematically able to assert that the dataset is manipulated. The smallest value of amplitude that can reached via the genetic algorithm without being detected is around 0.05. Figures 8 (b) (c) and (d) show the CDFs of f(S)) where S is chosen via the genetic algorithm, brute-force, and Fool SHAP respectively. We observe that Fool SHAP is the method where the resulting CDF for S(S)) is closest to the CDF for f(D,). This is why the audit is not able to detect fraud using statistical tests. The fact that Fool SHAP generates fake CDFs that are close to the data CDFs is consequence of minimizing the Wasserstein distance. These results highlight the superiority Fool SHAP compared to the brute-force force approach and the genetic algorithm.
22

---

Published as a conference paper at ICLR 2023
E.2 EXAMPLES ATTACKS
In this section, we present specific examples the attacks that were conducted on COMPAS, Adult, Marketing, and Communities.
Mr
Original Manioulated
1.0
nO.)
ont
Joy
charge depres
0.5
mied count
0.4
0.2
pfors 00amt
-0.57
-0.06
-0.05
-0.09
-0.02
0.0
Shap
Figure 9: Attack of RF fitted on COMPAS. Left: GSV before and after the attack with M 200. As a reminder, the sensitive attribute race. Right: Comparison of the CDF of the misleading subsets f(S)), f(S) and the CDF over the whole data. f(Di).
Ccharge_degres
Original Manipdated
1.0
Jou jwumiestoom count
0.8
My mthere zomm
0.6
0.4
AD
10.
0.2
A5
800
65
0.0
0.7
Outout
Figure 10: Attack of XGB fitted on COMPAS. Left: GSV before and after the attack with M = 200. As reminder, the sensitive attribute is race. Right: Comparison of the CDF of the misleading subsets f(S)). f(S)) and the CDF over the whole data. fDi)..
ccopettnn
Originsl
educaton-
workcioo
captal Joss
resiorship
weak
castal_gain
(0)
0.2
(5%)
(2,)
gender
0.3
0.6
0.8
1.0
Figure 11: Attack of XGB fitted Adults. Left: GSV before and after the attack with M = 200. As reminder, the sensitive attribute gender. Right: Comparison of the CDF of the misleading subsets f(S).. ((() and the CDF over the whole data. ((Do),((D))
23

---

Published as a conference paper at ICLR 2023
coupatin
Orginal Nanpulited
1.0
18ce
0.8
capial Jos-
eduaton
0.6
gatn
age
toun, per
gender
(S
-0.10
-0.08
-0
05
-0.05
-0.02
2.50
0.02
0.0
0.2
0.4
0.8
1.0
Shop value
OuEbuc
Figure 12: Attack of RF fitted on Adults. Left: GSV before and after the attack with M 200. reminder, the sensitive attribute is gender. Right: Comparison of the CDF of the misleading subsets ((S) and the CDF over the whole data. [
durasin
compaign
morttl-
van
hausing
day week
educctin
cons, price
poutcame
contect
cmmbor3m
age
emp rate
nr_empoyedd
puloted
-0.05
-0.04
-003
-0.02
-0.01
0.00
0.01
3.0
02
0.6
0.6
0.8
Shap
Dutour
Figure 13: Attack of RF fitted on Marketing. Left: GSV before and after the attack with 200. As a reminder, the sensitive attribute is age. Right: Comparison of the CDF of the misleading subsets ((S)).(() and the CDF over the whole data. f(D)).
graboou
campiig
Minatleted
marbal
cons jofooo_oj
day
bm
poutcame
comtet
plays
cort
durstion
employed
eno_vir_rtee
-0.6
Shap
DuEDut
Figure 14: Autack of XGB fitted on Marketing. Left: GSV before and after the attack with M 200. As reminder, the sensitive attribute is age. Right: Comparison of the CDF of the misleading subsets f(S).. ((S)) and the CDF over the whole data. ((Do),((D.)
24

---

Published as a conference paper at ICLR 2023
Pallhurp wyed
onggial pulated
ratroungg d52
Faa
ari Prtlargliuunnt Jm
scordrrip
tamr coh
Toka poip
Num mnderhov
Pertup Pertup Indethor
rcillig
koWiikce-0 50
0.2
AS
Muriiing
DiPenDDensHoous 7clK osthar
0.0 -10
OUtguE
Figure 15: Attack of XGB fitted on Communities. Left: GSV before and after the attack with M 200. As reminder, the sensitive attribute is PctWhi wetthite>900 Right: Comparison of the CDF of the misleading subsets ((S),, f(S)) and the CDF over the whole data. f(Do),
ProtouskePhore tet Dir
Numimmmi
recoPcatis
Peteoplindee floy
Troalhet
Potroungkis:22
FomoloP-iEhe
pettiisiin
ADU
Pctwhitz--0
02-
AR1
Pctllog
PeskidcPPer
0.0
A5
-0.07
-0.05
-0.06
-0.03
00.03
0.00
Shop value
Figure 16: Attack of RF fitted Communities. Left: GSV before and after the attack with M 200. As a reminder, the sensitive attribute is Pct Wetth>it>@0 Right: Comparison of the CDF of the misleading subsets f(S), ((() and the CDF the whole data. f(Di).
E.3 GENETIC ALGORITHM
This section motivates the of stealthily biased sampling to perturb Shapley Values in place of the method of [aanicci et al. (002)). which fools SHAP by perturbing the background dataset S via genetic algorithm. In said genetic algorithm, population of 1 fake background datasets (5(()) Jk-1 evolves iteratively following three biological mechanisms
Cross-Over: Two parents produce two children by switching some of their feature values.
Mutation: Some individuals perturbed with small Gaussian noise.
Selection:
The individuals 5((h) with the smallest amplitudes ( s((k)) are selected for the next generation.
Although the use of a genetic algorithm makes the method of Baniecki et al. 2202 very versatile, its main drawback is that there no constraint the similarity between the perturbed background and the original one. Moreover, the mutation and cross-over operations ignore the correlations between features and hence the perturbed dataset can contain unrealistic instances. To highlight these issues, Figure 17 presents the first two principal components of Di and S for the XGB models used in Section 5.4 On COMPAS and Marketing especially, we see that the fake samples S lie in regions outside of the data manifold. For Adult-Income and Marketing, the fake data overlaps more with the original one, but this could be an artifact of only visualizing 2 dimensions.
For more rigorous analysis of "similarity" between Sy and D1, must study the detection rate of the audit detector. To this end, Figures [nd 19] present the amplitude reduction and the detection rate after given number of iterations of the genetic algorithm. These curves show the average and
25

---

Published as a conference paper at ICLR 2023
7.5
5.0
5.0
Compotent 2
2.5
Component
2.5
0.0
0.0
-2.5
2.5
-5.0
-5.0
-2.5
0.0
2.5
5.0
7.5
-25
0.0
2.5 Component
5.0
7.5
Component
(a) COMPAS
(b) Adult-Income
7.5-
20-
Component 2
5.0
Component 2
10-
2.5
0.0
-2.5
-10
-10
Component
Component
(c) Marketing
(d) Communities
Figure 17: First two principal components of D₃ (Blue) and S (Red) returned by the genetic
algorithm XGB models.
standard deviation across the train/test splits employed in our main experiments. Moreover, window 20 convolutions were used smooth the curves and make them more readable. On the Marketing and Communities datasets, we see that for both XGB and Random Forests models, the detector is quickly able assert that the data was manipulated. We suspect the genetic algorithm cannot fool the detector on these two datasets because they contain large number of features (Marketing has 20, Communities has 98). Such large number of features could make harder to perturbate samples while staying close to the original data manifold. Since the model behavior is unpredictable outside of the data manifold, is impossible for the genetic algorithm to guarantee that the CDF of f(S) will be close the CDF of f (D:). For adult-income, the detection rate appears be lower but still, the largest reductions in amplitude the sensitive feature were about 15%, even after 2.5 hours of run-time.
Contrary to the genetic algorithm, our method Fool SHAP addresses both constraints of making the fake data realistic and keeping it close the original dataset. Indeed, our objective is tuned make sure that the Wasserstein distance between the original and perturbed data is small. Moreover, since we do not generate new samples but rather apply non-uniform weights pre-existing ones, do not run into the risk of generating unrealistic data.
26

---

Published as a conference paper at ICLR 2023
Rolative Amplineh Docruaee
30
Rolative Amplitnde Decrouse
15
Detextion Eiaa Rate
-10
Deletiin Rate
20
20
30
1001
2000
300
400
101
2010 Iterations
300
400
(a) COMPAS
(b) Adult-Income
15
100
Belative Ampli atle Deriaan
Relative Anplttade Deceunse
80
Dotecton Rabe
Delociin Date
60
10
60
40
20
100
2000
300
400
0
25
SU
75
100
Iterations
[terations
(c) Marketing
(d) Communities
Figure 18: Iterations the genetic algorithm applied to 5 XGB models per dataset.
Relaiive Amplicade Decrosse
Bolaiive Anguätude Decraee
10
10.0
Deseciou tia Rate
30
Rude
10-
10
5.0
20
Desofiion
51
5
2.5
0-
150
20
40
60
80
100
Iterations
Hierations
(a) COMPAS
(b) Adult-Income
Relative Ampli ude Decresse
25
1000
Relative Amplitude Decroose
10
100
20-
80
Desection Hate
Dolection Rate
16-
10.
64
0
0.0
2.5
5.0
7.5
10.0
0
10
20
Iterations
Iteratious
(c) Marketing
(d) Communities
Figure 19: Iterations of the genetic algorithm applied RF models per dataset.
27

---

Published as a conference paper at ICLR 2023
E.4 MULTIPLE SENSTTIVE ATTRIBUTES
We present preliminary results for settings where one wishes to manipulate the Shapley values of multiple sensitive features 8 each part of a set S. For example, in our experiments, considered gender as a sensitive attribute for the Adult-Income dataset and we showed that one can diminish the attribution of this feature. Nonetheless, there are two other features in Adult- Income that share information with gender: relationshie and marital all-status. In- deed, relatioinnn onship can take the value widowed and marital-statuss can take the value wi fe, which are both proxies of gender=female For this reason, these two other features may be considered sensitive and decision-making that relies strongly on them may not be ac- ceptable. Hence, we must derive a method that reduces the total attributions of the features in marital-statuus
= (gendez, relationitttiiiiieeoeeaeooeo.
We first let 3, == sign| Lg D, L. St [0))]
J for any s S. In our experiments, all these signs will typically be negative. The proposed approach is to minimize the & norm
( Sy S))) == Σ
(T, S) 1,
(41)
seS
which we interpret as the total amount of disparity we can attribute to the sensitive attributes. Re- member that (J, S. Si) converges in probability to Lgone wy (J, 2()) (ef. Proposition Therefore minimizing the norm will require minimizing
4.1b
Σ SES
Σ x
Wj (f, S.,20)10) x0) = Σ
w B. 20)
(42)
2DDED
SES
which is again linear function of the weights. We present Algorithm 4 as overload of Algorithm that now supports taking multiple sensitive attributes as inputs.
Algorithm 4 Compute non non-uniform weights for multiple sensitive attributes s
S
1: procedure COMPUTE_ WEIGHTS(D { ( S.. =())) }ag>
2:
B, sign EoOVED, (., 201] V
3: B = C(D1,1/N1)
Unmanipulated background
4: B", = w(D
t Manipulated background function of w
5: w arg min ΣxmED Wj Eas 3, a S. 200) AW
6: return w;
The only difference in the resulting MCF is that we must use the cost a(e) EES 3o (J, Sy 2(0) for edges (s, Eÿ) ) in the graph G of Figure 6 This new algorithm is guaranteed to diminish the L norm of the attributions of all sensitive features. However, that this does not imply that all sensitive attributes will diminish in amplitude. Indeed, minimizing the sum of multiple quantities does not guarantee that each quantity will diminish. For example, 4 + 7 is smaller than 6 6 although 4 is smaller than 6 and is higher than 6. Still, we see reducing the ε norm as a natural way to hide the total amount of disparity that attributable to the sensitive features. Another important methodological change is the way we select the optimal hyper-parameter À in Algorithm 3 Now at line 12, we use the & norm Exes EgGeED W (f. S. | selection criterion.
Figures 20 and 21 present preliminary results of attacks on three RFs/XGBs fitted on Adults with different train/test splits. We note that in all cases, before the attack, the three sensitive features had large negative attributions. By applying our method, we can considerably reduce the amplitude of the two sensitive attributes. The attribution of the remaining sensitive feature remains approximately constant or slightly becomes more negative. We leave it as future work to run large-scale experiments with multiple sensitive features for various datasets.
28

---

Published as a conference paper at ICLR 2023
100
0.000
Delection (%)
80
0.005
60
GSV
0..55
0.075
20
Seusitive features
0.100
Other features
10a
10a
10
102
10a
10"
100
0.000
Dotection (%)
0.005
60-
-0.050
40
0.075
20
Sensitive features
-0.100
Other features
X
10a
10a
10a
102
10A
10"
A
100
0.000
Detection (%)
0.022
60
0.050
0.075
20
Sensitive features
0.100
Other features
10
10
10
10a
10a
10a
Figure 20: Example of log-space search over values of À using RFs classifier fitted on Adults and three sensitive attributes. Each row is different train/test split seed. (Left) The detection rate as a function of the parameter of the attack. (Right) For each value of the vertical slice of the 11 curves is the GSV obtained with the resulting B The goal here is to reduce the amplitude all sensitive features (red curves) order hide their contribution to disparity model outcomes.
29

---

Published as a conference paper at ICLR 2023
100
0.2
Delection (%)
80
0.0
60-
GSV
-0.4
20
0.6
Sensitive features
Other features
-0.8
10P
10
10
10
10
10
100
0.2
Detadiion (%)
80
0.0
60
00.2
GSV
-0.4
20
Sensitive features
-0.8
Other features
10P
10a
10
A
10
10
10
100
0.2
Detection Rake (%)
80
0.0
60
GSV
00.2
40
0.4
20
Sensitive features
0.6
Other features
A*
10a
10a
Figure 21: Example of log-space search over values of using XGBs classifier fitted on Adults and three sensitive attributes. Each row is different train/test split seed. (Left) The detection rate as function of the parameter À of the attack. (Right) For each value of X, the vertical slice of the 11 curves is the GSV obtained with the resulting B" The goal here is to reduce the amplitude all sensitive features (red curves) in order hide their contribution the disparity in model outcomes.
30


Reviews:
[{"review_id": "ywLSu95SslG", "reviewer": "Reviewer_L3wE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses the use of contrastive Shapley values as a way of post-hoc auditing of prediction models. \n\nThe scenario that an auditor asks for samples from two populations according to a sensitive demographic feature, which should not appear in the model. Then, the auditor checks the distribution of Shapley values to see if there are positive or negative differences across these populations. \n\nThe paper argues that by making small changes to the sampling weights, the company being audited can attack the audit and alter it's results. The paper provides an optimization score that weighs the changes in the distribution (in Wasserstein distace) against the change in the global Shapley values, and an algorithm for minimizing this optimization problem. \nThey show that cosiderable changes to the audit score can be achieved while still being hard to detect.\n\nIn terms of the main cotribution, as outlined in section 4.4,  it is moving the idea of stealth sampling from the original important feature to the SHAP value computed for the feature. \n\n\nUpdate: I'm changing my evaluation to marginally accept. Again, I think this paper is written clearly and is therefore useful in presenting the ideas. My main concern is the amount of novelty required for ICML. The answers to my questions have been clear about the nature of the  novel contributions. I think at this point this an editorial call that I am happy to leave to the area-chairs etc.  \n\n", "summary_of_the_review": " \nPaper is written very nicely.\nIt's not clear to me that the changes from previous methods are sufficient for acceptance.\n\n\nPS. I appologize for this very late review. ", "strength_and_weaknesses": "Strength: \nThe paper is well organized, very clear in its communication, and displays well the results.\nI found it to be a very nice read. \n\nWeakness:\nThe main weakness is that it is not clear how different this paper is from Fukuchi et al.\nIt seems that both the attack, and more generally almost all the methodology (Wasserstein distance) can already be found there. \nMoving this idea to SHAP is nice, but the question is whether there was anything interesting to say about this move. \nIn other words:\n- I do not understand whether there are any substantial changes that the method  discussed in   Fukuchi et al.\n(2020) had to undergo to work for the SHAP values. \nIn a sense, the  SHAP values themselves are almost not discussed at all  in the paper; do they provide different bounds?\ndid adapting Fukuchi et al.\n(2020) required any extra work, or indeed if the method works differently in some sense when adapting it to SHAP. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "n_yHEUeskky", "reviewer": "Reviewer_yVbY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is a carefully prepared, complete, and convincing paper in computer science about Shapley Values. It challenged the established statistics tool of SHAP that is yet to emerge as a widely adopted Interpretable AI method in evaluating machine/deep learning methods. Namely, the challenge was implemented by proposing \"a complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution.\" Its results illustrated that SHAP can be fooled, thereby begging the further research questions on Shapley Values as a fair and reliable tool in evaluating, or even auditing, features in machine/deep learning applications.", "summary_of_the_review": "This is a timely and good paper and hence, I recommend its acceptation to ICLR 2023.", "strength_and_weaknesses": "This is a carefully prepared, complete, and convincing paper in computer science about Shapley Values. It challenged the established statistics tool of SHAP that is yet to emerge as a widely adopted Interpretable AI method in evaluating machine/deep learning methods. Namely, the challenge was implemented by proposing \"a complementary family of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution.\" Its results illustrated that SHAP can be fooled, thereby begging the further research questions on Shapley Values as a fair and reliable tool in evaluating, or even auditing, features in machine/deep learning applications. \n\nSee some minor comments related to improving the clarity of this paper below in the next section.\n\nI would also encourage the authors to engage with a statistician (e.g., through research project support, statistical consultation unit, statistics services, or similar of their university) to ensure that their study design follows scholarly conventions in statistics. The paper excelled as a computer science study but I noticed that, for instance, the chosen significance level of 1% was somewhat unusual as typically \\alpha = 0.005 or 0.05 is selected.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4O4lKLLQt0g", "reviewer": "Reviewer_fFZy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the manupulation of the feature attributions method, so that the attributino to sensitive features are hidden. More concretely, this paper studies the audit scenario, where the auditors are given a full set of all predictions, and a small sample of features; the authors then propose an algorithm (called stealthily biased sampling) to get the small samples to submit to audit, so that 1. the auditors cannot detect the manipulation by studying the prediction distribution, and 2. the attribution to some sensitive feature given by SHAP is decreased.", "summary_of_the_review": "Overall I think the paper is very well written, and studies an interesting auditing scenario in details. However, this vulnerability to the manipulation is not entirely new. I give 5 for now, but will reconsider if my concerns are addressed.", "strength_and_weaknesses": "Pros:\nThe paper's presentation is very clear, the introduction of the auditing scenario is well motivated and interesting. The paper, as well as the supplemental code, is well organized and easy to follow.\n\nCons:\nThe vulnerability of the interventional SHAP (more details below) is not entirely new. While the experiments do show the effectiveness of the proposed manipulation algorithm, its superiority over brute-force/baseline method is not provided theoretically or experimentally.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster