Better sampling in explanation methods can prevent dieselgate-like deception
Authors: Domen Vreš
Marko Robnik Šikonja
Venue:  (2021)
Keywords: Explaniable AI
explanation methods
robust explanations

Abstract: Machine learning models are used in many sensitive areas where besides predictive accuracy their comprehensibility is also important. Interpretability of prediction models is necessary to determine their biases and causes of errors, and is a necessary prerequisite for users' confidence. For complex state-of-the-art black-box models post-hoc model-independent explanation techniques are an established solution. Popular and effective techniques, such as IME, LIME, and SHAP, use perturbation of instance features to explain individual predictions.  Recently, Slack et al. (2020) put their robustness into question by showing that their outcomes can be manipulated due to poor perturbation sampling employed. This weakness would allow dieselgate type cheating of owners of sensitive models who could deceive inspection and hide potentially unethical or illegal biases existing in their predictive models. This could undermine public trust in machine learning models and give rise to legal restrictions on their use.  

We show that better sampling in these explanation methods prevents malicious manipulations. The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set. We show that the improved sampling increases the robustness of the LIME and SHAP, while previously untested method IME is already the most robust of all. 

Under review as a conference paper at ICLR 2021
BETTER SAMPLING IN EXPLANATION METHODS CAN PREVENT DIESELGATE-LIKE DECEPTION
Anonymous authors Paper under double-blind review
ABSTRACT
Machine learning models are used in many sensitive areas where, besides predic- tive accuracy, their comprehensibility diction models is necessary to determine their biases and causes of errors and is prerequisite for users' confidence. For complex state-of-the-arg black-bo models, post-hoc model-independent explanation techniques are established solution. Popular and effective techniques, such as IME, LIME, and SHAP, use perturba- tion of instance features to explain individual predictions. Recently, (Slack al.. (2020) put their robustness into question by showing that their outcomes can be manipulated due poor perturbation sampling employed. This weakness would allow dieselgate type cheating of owners of sensitive models who could deceive inspection and hide potentially unethical or illegal biases existing in their predic- tive models. This could undermine public trust in machine learning models and give rise to legal restrictions on their use.
also important. Interpretability of pre-
We show that better sampling in these explanation methods prevents malicious manipulations. The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set. We show that the improved sampling increases the LIME and SHAP's robustness, while the previously untested method IME is already the most robust of all.
INTRODUCTION
Machine learning models are used in many areas where besides predictive performance, their com- prehensibility is also important, c.g., in healthcare, legal domain, banking, insurance, consultancy, etc. Users in those areas often do not trust machine learning model if they do not understand why made a given decision. Some models, such as decision trees, linear regression, and naïve Bayes, are intrinsically easier understand due to the simple representation used. However, complex models, mostly used in practice due to better accuracy, are incomprehensible and behave like black boxes, e.g., neural networks, support vector machines, random forests, and boosting. For these models, the area of explainable artificial intelligence (XAI) has developed post-hoe explanation methods that are -ndeeeeeenndnt and determine the importance of each feature for the predicted outcome. Fre- quently used methods of this type are IME (Strumbelj & Kononenko 2013). LIME (Ribeiro al. 2016), and SHAP (Lundberg & Lee 2017).
To determine the features' importance, these methods use perturbation sampling. Slack et al. (2020) recently noticed that the data distribution obtained in this way is significantly different from the original distribution of the training data as we illustrate in Figure [.. They showed that this can be serious weakness of these methods. The possibility to manipulate the post-hoc explanation methods a critical problem for the MI. community, as the reliability and robustness of explana- tion methods are essential for their use and public acceptance. These methods are used to interpret otherwise black-box models, help in debugging models, and reveal models' biases, thereby estab- lishing trust in their behavior. Non-robust explanation methods that can be manipulated can lead to catastrophic consequences, as explanations do not detect racist, sexist, or otherwise biased models the model owner wants to hide these biases. This would enable dieselgate- like cheating where own- ers of sensitive prediction models could hide the socially, morally, or legally unacceptable biases present in their models. As the schema of the attack explanation methods Figure Tpp shows,

---

Under review as a conference paper at ICLR 2021
Advarirtial mocel
Fioed
10
a)
b)
Figure 1: a) PCA based visualization of part of the COMPAS dataset. The blue points show the original instances, and the red points represent instances generated with the perturbation sampling used in the LIME method. The distributions are notably different. b) The idea of the attack on explanation methods based on the difference of distributions. The attacker's adversarial model con- tains both the biased and unbiased model. The decision function that is part of the cheating model decides if the instance outside the original distribution (i.e. used only for explanation) or an actual instance. If the case of an actual instance, the result of the adversarial model is equal to the result of the biased model, otherwise it is equal to the result of the unbiased model.
owners of prediction models could detect when their models are examined and return unbiased pre- dictions in this case and biased predictions in normal use. This could have serious consequences in areas where predictive models' reliability and fairness are essential, e.g., in healthcare or banking- Such weaknesses can undermine users' trust in machine learning models in general and slow down technological progress.
In this work, we propose to change the main perturbation- bbaed explanation methods and make them more resistant to manipulation attempts. In our solution, the problematic perturbation- sampling is replaced with more advanced sampling, which uses modern data generators that better capture the distribution of the training dataset. We test three generators, the RBF network based gen- erator (Robnik -Skuikjjj 2016), random forest-based generator, available in R library semiArtificial (Robnik- Siikkaajjjjja, ) as well the generator using variational autoencoders (Miok al. 2019). We show that the modified gLIME and gSHAP methods are much more robust than their original versions. For the IME method, which previously was not analyzed, we show that it is already quite robust. We release the modified explanation methods under the open-source license]
In this work, we use the term robustness the explanation method as a notion of resilience against adversarial attacks, i.e. the ability of an explanation method to recognize the biased classifier in an adversary environment. This type of robustness could be more formally defined as the number of instances where the adversarial model's 's bias concerning the attacks described in Slack et al. (2020). There are other notions of robustness in ex- planation methods; e.g., (Alvarez-Mei) in the sense that similar inputs should give rise to similar explanations.
correctly recognized. We focus the robustness
Jaakkola 2018) define the robustness of the explanations
The remainder of the paper organized follows. In Section 2 we present the necessary back- ground and related work on explanation methods, attacks on them, and data generators. In Section 3 propose a defense against the described weaknesses of explanation methods, and in Section we empirically evaluate the proposed solution. In Section we draw conclusions and present ideas for further work.
BACKGROUND AND RELATED WORK
In this section, we first briefly describe the background on post-hoc explanation methods and attacks on them, followed by data generators and related works the robustness of explanation methods.
2.1 POST-HOC EXPLANATION METHODS
The current state-of tatte-of-the-art perturbation- explanation methods, IME, LIME, and SHAP, ex- plain predictions for individual instances. To form an explanation of a given instance, they measure the difference in prediction between the original instance and its neighboring instances, obtained
2

---

Under review as a conference paper at ICLR 2021
with perturbation sampling. Using the generated instances, the LIME method builds local inter- pretable model, c.g., a linear model. The SHAP and IME methods determine the impact of the features as Shapley values from the coalitional game theory (Shapley) 1988). In this way, they sure that the produced explanations obey the four Shapley fairness axioms (Strumbelj & Kononenko/ 2013). Due to the exponential time complexity of Shapley value calculation, both methods try to approximate them. The three methods are explained in detail in the above references, and formal overview is presented in Appendix[A) [ while below, we present a brief description. In our exposition of the explanation methods, we denote with f the predictive model and with x the instance we are explaining.
Explanations of instances with the LIME method is obtained with an interpretable model g. The model g has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that is interpretable). Therefore in LIME, g is linear regression model trained on the instances sampled in the neighborhood of the explained instance explanation instances are generated by adding Gaussian noise to each feature of independently. Using linear models to determine the contributions of features, LIME is fast but may produce poor explanations for instances close to complex decision boundaries.
which is first normalized. The perturbed
SHAP estimates Shapley values using the approach of LIME (i.e. local linear models computed with the weighted Ieast squares algorithm) but using a different neighborhood and loss function for local accuracy, which ensures that Shapley values are obtained. The sampling of explanation instances in SHAP relies on the distribution set D, obtained from the training set, by computing the centroids of k-means clusters.
IME computes Shapley values using a pure sampling approach (no proxy with an interpretable model). Its sampling replaces part of the explained instance with existing values of selected features and then averages differences in the prediction of explained instance and samples. This makes the method slower but also potentially more robust the method does not assume the shape of the dependency.
2.2 ATTACKS ON EXPLANATIONS
The idea of the attack on explanations presented Sllak et al. (2020) is that the attacker creates an adversarial (biased) model whose behavior it wants to hide from explanation methods, e.g., racist model that does not grant credits to blacks. As Figure [ illustrates, a part of the attacker's adversarial model is an unbiased model, which, e.g., does not take disputed attributes such as race into account. The adversarial model tries to manipulate the explanation method by behaving like an unbiased model perturbed instances used for the explanation in this case, it uses the unbi- ased model's output. On instances from the original distributions that are a part of normal model use, the adversarial model uses the biased model's output. To distinguish between the original and cituurattion-sappple data, the adversarial model contains a decision model that selects the right type of model (biased or unbiased) based on the predicted type of instances.
The reason that the decision function can distinguish between the original data and sampled in- stances are the shortcomings of explanation methods that use the perturbation sampling (see Fig- ure TT). For that graph, we created new instances based on the COMPAS dataset (see Section 4 with the Gaussian noise as in the LIME explanation method. We mapped the original and sampled instances to 2D space using the PCA transformation. We can see that the sampled data is distributed differently than the original data. As the difference is revealed with PCA, other machine learning algorithms can also effectively distinguish between sampled and original data.
The attack can be formally defined in the following way. The attacker has a biased model b and a sample S from the distribution Xaist- The attacker has no additional information about the distribu- tion Xdist- The attacker's 's goal is to produce an adversarial model e that behaves like on the data from Xdist but does not reveal b's bias to the explanation methods. We assume that the attacker also has an unbiased model & that hides the f's bias. The attacker creates a decision model d, that should output if the input instance is from Xaist and 0 otherwise. The model d is trained on S and generated perturbation samples. The attacker creates the adversarial model e with its output defined by the following equation:
(bxx), (x)))
d(x) 1
e(x)
(1)
d(x) = 0
3

---

Under review as a conference paper at ICLR 2021
2.3 DATA GENERATORS
We used three different data generators based on different algorithms, modeling the distribution of the training set; variational autoencoder with Monte Carlo dropout (Miok et al. 2019). RBF network (Robnik- (Robnik-Sikonja -iikojja 2016), and random forest ensemble -Robnik-Sikonja 2019). ). In the re- mainder of the paper, we refer to the listed generators consecutively as MCD-VAE, rbiDataGen, and TreeEnsemble.
Autoencoder (AE) consists of two neural networks called encoder and decoder. It aims to compress the input instances by passing them through the encoder and then reconstructing them to the original values with the decoder. Once the AE is trained, it can be used generate new instances. Variational autoencoder (Doersch, 2016) is a special type of autoencoder, where the vectors in the latent dimension (output of the encoder and input of the decoder) are normally distributed. Encoder is therefore approximating the posterior distribution p(z(z)) where assume p p(z|zz ~ (Na En). The generator proposed by Miok et al. (219)) uses the Monte Carlo dropout (Gal Ghahramani, 2016) on the trained decoder. The idea of this generator is to propagate the instance through the encoder to obtain its latent encoding 2. This can be propagated many times through the decoder, obtaining every time different result due to the Monte Carlo dropout but preserving similarity to the original instance x.
The RBF network (Moody & Darken 1989) uses Gaussian kernels as hidden layer units in neural network. Once the network's parameters learned, the rbfDataGen generator (Robnik- 2016) can sample from the normal distributions, defined with obtained Gaussian kernels, to generate new instances.
(Robnik-Sikonjaa Sikonja
The TreeEnsemble generator (Robnik-Sikon)) 2019) builds a set of random trees (forest) that de- scribe the data. When generating new instances, the generator traverses from the root the leaves of a randomly chosen tree, setting values of features in the decision nodes on the way. When reaching leaf, it assumes that it has captured the dependencies between features. Therefore, the remaining features can be generated independently according to the observed empirical distribution in this leaf. For each generated instance, all attributes can be generated one leaf, or another tree can be ran- domly selected where unassigned feature values are filled in. By selecting different trees, different features are set in the interior nodes and leaves.
2.4 RELATED WORK ON ROBUSTNESS OF EXPLANATIONS
The adversarial attacks perturbation based explanation methods were proposed by Slack et al. 2020), who show that LIME and SHAP are vulnerable due to the perturbation based sampling used. We propose the solution to the exposed weakness in SHAP and IME based on better sampling using data generators adapted to the training set distribution.
In general, the robustness of explanation methods has been so far poorly researched. There claims that post-hoc explanation methods shall not be blindly trusted, as they can mislead users (deliberately or not) and disguise gender and racial discrimination (Lipton) 2016). [Selbst & Barocas) 20188) and Kroll et al.) (2017) showed that even if a model is completely transparent, it is hard to detect and prevent bias due the existence of correlated variables.
Specifically, for deep neural networks and images, there exist adversarial attacks on saliency map based interpretation of predictions, which can hide the model's bias (Dombrowski et al. 2019 Heo et al. 2019; Ghorbani et al.] 2019). Dimanov et al. (2020) showed that a bias of a neural network could be hidden from post-hoc explanation methods by training a modified classifier that has similar performance to the original one, but the importance of the chosen feature is significantly lower.
The kNN-based explanation method, proposed by Chakraborty et al. (2020), tries to overcomes the inadequate perturbation based sampling used in explanation methods by finding similar instances to the explained one in the training set instead of generating new samples. This solution inadequate for realistic problems as the problem space is not dense enough to get reliable explanations. Our defense of current post-hoc methods is based on superior sampling, which has not yet been tried. Saito et al. (2020) use the neural CT-GAN model to generate more realistic samples for LIME and prevent the attacks described in Slack et al. (2020). We are not aware of any other defenses against the adversarial attacks post-hoc explanations.
4

---

Under review as a conference paper at ICLR 2021
3 ROBUSTNESS THROUGH BETTER SAMPLING
We propose the defense against the adversarial attacks on explanation methods that replaces the problematic perturbation sampling with a better one, thereby making the explanation methods more robust. We want to generate the explanation data in such a way that the attacker cannot determine whether instance is sampled obtained from the original data. With improved sampling, the adversarial model shown in Figure D shall not determine whether the instance x was generated by the explanation method, generator, the adversarial model cannot adjust its output properly and the deception, described in Section 2.2 becomes ineffective.
the original instance the model has to label. With a better data
The reason for the described weakness of LIME and SHAP is inadequate sampling used in these methods. Recall that LIME samples new instances by adding Gaussian noise the normalized feature values. SHAP samples new instances from clusters obtained in advance with the k-means algorithm from the training data.
Instead of using the Gaussian noise with LIME, we generate explanation samples for each instance with one of the three better data generators, MCD- VAE, rbfDataGen, or TreeEnsemble (see Sec- tion 2.3) We call the improved explanation methods gLIME and gSHAP (g stands for generator- based). Using better generators in the explanation methods, the decision function in the adversarial model will less likely determine which predicted instances are original and which are generated for the explanation purposes.
Concerning gLIME, we generate data in the vicinity of the given instance using MCD-VAE, as the LIME method builds a local model. Using the TreeEnsemble and rbfDataGen generators, we do not generate data in the neighborhood of the given instance but leave it to the proximity measure of the LIME method to give higher weights to instances closer to the explained one.
In SHAP, the perturbation sampling replaces the values of hidden features in the explained instance with the values from the distribution set D. The problem with this approach is that it ignores the dependencies between features. For example, in a simple dataset with two features, house size, and house price, let us assume that we hide the house price, but not the house size. These two features not independent because the price of a house increases with its size. Suppose we are explaining an instance that represents a large house. Disregarding the dependency, using the sampled set D, SHAP creates several instances with low price, as such instances appeared in the training set. In this way, the sampled set contains many instances with small price assigned to large house, from which the attacker can determine that these instances were created perturbation sampling and serve only for the explanation.
In the proposed gSHAP, using the MCD-VAE and TreeEnsemble generators, the distribution set D is generated in the vicinity of the explained instance. In the sampled instances, some feature values of the explained instance are replaced with the generated values, but the well-informed generators consider dependencies between features detected in the original distribution. This will make the distribution of the generated instances very similar to the original distribution. In our example, the proposed approach generates new instances around the instance representing large house, and most of these houses will be large. As the trained generators capture the original dataset's dependencies, these instances will also have higher prices. This will make it difficult for the attacker to recognize the generated instances used in explanations. The advantage of generating the distribution set close to the observed instance is demonstrated in Appendix B
The rbfDataGen generator does not support the generation of instances around a given instance. Therefore, we generate the sampled set based on the whole training set and not for each instance separately (we have this option also for TreeEnsemble). This is worse than generating the distribu- tion set around the explained instance but still better than generating it using the k-means sampling in SHAP. There are at least three advantages. First, the generated distribution set D can be larger. The size of the k-means distribution set cannot be arbitrarily large because it is limited by the number of clusters in the training set. Second, the centroids of the clusters obtained by the k-means algorithm do not necessarily provide a good summary of the training set and may not be valid instances from training set distribution. They also do not capture well the density of instances (e.g., most of the data from the training set could be grouped one cluster). Third, using the proposed generators, SHAP becomes easier to compared to the k-means generator, where users have to determine
5

---

Under review as a conference paper at ICLR 2021
Data set COMPAS
# inst.
# features
# categorical
sensitive
unrelated 1
unrelated 2
6172
7
4
race gender perOflncome
random1
random2 / random2
German
1000
25
15
CC
1994
100
0
racePctWhite
random!
Table 1: Basic information and sensitive features in the the used data sets. The target variable is not included in the number of features and is binary for all data sets. The pctOflncome full name is (oonRateAsPereenOffccmm
the number of clusters, while the data generators we tested can be used in the default mode without parameters,
EVALUATION
To evaluate the proposed improvements in the explanation methods, we first present the used datasets in Section 4.1 followed by the experiments. In Section 4.2 we test the robustness of gLIME, gSHAP, and gIME against the adversarial models. To be more realistic, we equipped the adversarial models with the same improved data generators we used in the explanation methods. is reasonable to assume that attackers could also use better data generators when preparing their decision function, making their attacks much stronger. As the evaluation metric for the success of deception, we use the proportion of instances where the adversarial model deceived the explanation methods so that they did not detect sensitive attributes as being important in the prediction models. Section [433] we test if enhanced generators produce different explanations than the original ones. As the attacker might decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method, we test different thresholds of the decision function d from Equation (1) (currently set to 0.5). We report on this analysis in Section 4.4
4.1 SESSITIVE DATASETS PRONE TO DECEPTION
Following Sllak et al. 2020), we conducted our experiments on three data sets from domains where a biased classifier could pose a critical problem, such as granting a credit, predicting crime recidivism, and predicting the violent crime rate. The basic information on the data sets is presented in Table ! The statistics were collected after removing the missing values from the data sets and before we encoded categorical features as one-hot-encoded vectors.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) a risk assess- ment used by the courts in some US states to determine the crime recurrence risk of a defendant. The dataset (Angwin et al. 2016)) includes criminal history, time in prison, demographic data (age, gender, race), and COMPAS risk assessment of the individual. The dataset contains data of 6.172 defendants from Broward Couty, Florida. The sensitive attribute in this dataset (the one on which the adversarial model will be biased) is race. African Americans, whom biased model associates with a high risk of recidivism, represent 51.4% of instances from the data set. This set's target variable the COMPAS score, which is divided into two classes: high and low risk. The majority class is the high risk, which represents 81.4% of the instances.
The German Credit dataset (German for the rest of the paper) from the UCI repository (Dua & Graff 2019) includes financial (bank account information, loan history, loan application information, etc.) and demographic data (gender, age, marital status, etc.) for 1,000 loan applicants. A sensitive attribute in this data set is gender. Men, whom the biased model associates with a low-risk, represent 69% of instances. The target variable is the loan risk assessment, divided into two classes: good and bad customer. The majority class is good customer, which represents 70% of instances.
Communities and Crime (CC) data set (Redmond Baveja 2002) contains data about the rate of violent crime in US communities in 1994. Each instance represents one community. The features are numerical and represent the percentage of the community's 's population with a certain property or the average of population in the community. Features include socio-economic (e.g., education, house size, etc.) and demographic (race, age) data. The sensitive attribute is the percentage of the white race. The biased model links instances where whites' percentage is above average to low rate of violent crime. The target variable is the rate of violent crime divided into two classes: high and low. Both classes are equally represented in the data set.

---

Under review as a conference paper at ICLR 2021
4.2 ROBUSTNESS OF EXPLANATION METHODS
To evaluate the robustness of explanation methods with added generators (i.e. LLIME, gSHAP, and glME), we split the data into training and evaluation set in the ratio 90% 10%. We used the same training set for the training of adversarial models and explanation methods. We encoded categorical features as one-hot-encoded vectors.
We simulated the adversarial attack for every combination of generators used in explanation methods and adversarial models (except for method IME, where we did not use rbfDataGen, which cannot generate instances in the neighborhood of a given instance). When testing SHAP, we used two vari- ants of the Treeensemble generator for explanation: generating new instances around the explained instance and generating new instances according to the whole training set distribution. In the LIME testing, we used only the whole training set variant of TreeEnsemble inside the explanation. In the IME testing, we only used the variant of Tree Ensemble that fills in the hidden values (called TEns- FillIn variant). For training the adversarial models, we used the whole training set based variant of TreeEnsemble, except for IME, where we used the TEnsFillln variant. These choices reflect the capabilities of different explanation method and attempt to make both defense and attack realistic (as strong possible). More details on training the decision model d inside the adversarial model can be found in Appendix D
In all cases, the biased model b (see Section 2.2) was simple function depending only the value of the sensitive feature. The unbiased model &i depended only on the values of unrelated features. The sensitive and unrelated features are shown on the right-hand side of Table I Features random] and random2 were uniformly randomly generated from the {0, 1} set. On COMPAS and CC, we simulated two attacks, one with single unrelated feature (the result of sb depends only the value of the unrelated feature 1), and another one with two unrelated features.
For every instance in the evaluation set, we recorded the most important feature according to the used explanation method (i.e. the feature with the largest absolute value of the contribution as determined by the explanation method). The results are shown as a heatmap Figure The green color means that the explanation method was deceived in less than 30% of cases (i.e. the sensitive feature was recognized as the most important one in more than 70 % of the cases as the scale on the heatmap suggests), and the red means that it was deceived in more than 70% of cases (the sensitive feature was recognized as the most important one in less than 30 of the cases as the scale on the heatmap suggests). We consider deception successful if the sensitive feature was not recognized as the most important by the explanation method (the sensitive features are the only relevant features in biased models b).
The gLIME method is more robust with the addition of rbiDataGen and TreeEnsemble than LIME and less robust with the addition of MCD-VAE. This suggests that parameters for MCD-VAE were not well-chosen, and this pattern repeats for other explanation methods. Both TreeEnsemble and rbfDataGen make gLIME considerably more robust on COMPAS and German datasets, but not the CC dataset. We believe the reason for that is that features in CC are strongly interdependent, and many represent the same attribute as a fraction of value, e.g., we have the share of the white population, the share of the Asian population, and so on. This interdependence dictates that all fractions have to add up to 1. The data generators are unlikely to capture such strong conditional dependencies, but the adversarial model's decision function is likely to approximate it.
The gSHAP method is most robust when using the Treeensemble generator, but shows less robust behavior than the gLIME method. The reason for that could be in the feature value replacement strategy used by the SHAP and gSHAP methods, which change only some of the feature's values with the help of the distribution set. This can lead to out-of-distribution instances if the instances in the distribution set are not close enough to the explained instance. With ILIME we generate complete instances that are more likely to be in the distribution of the training set.
IME is quite robust even with the perturbation sampling, as we expected. This suggests that IME is the most robust of all three tested explanation methods and shall be considered the chosen method in sensitive situations. The gIME results when using the TreeEnsemble generator (TEnsFillIn variant) comparable to the original IME variant results. This suggests that sampling from smaller data set, which represents the neighborhood of the explained instance, does not decrease the method's s robustness.
7

---

Under review as a conference paper at ICLR 2021
Pattarstion
TreeErn
Compae Pett returatton
Compue IMSTMAAE
compos, TroeEis
Comgasa MODD-WE
Compost RuDzarSen
Soman Returtation
Geraa JIC--AE
Gonus TecE o
DE FELPPtinel motton
MCD VAE
Potulaainn
002 MCD-
Ca RoDasOmn
06
TEvafille
Carpa retuiaation
Campas, Jaroooso
Compae, TrowEe
Compade Nartuaatoo
INCIAVCC
Carpast RMDaas-on
con2662 TrovE
Cerman INDEAWA
Gaman Farrat Ceteen
Trndftt
Putuiisiin
0C PELE
032. Nettatatioon
C22 MM33/W
CGL Tieerr
Compoe JND
000
0.
Perurtatonn
08
MCD
TEnFFth
02
Compe Patuteaann
NCDDWE
Compasd Persrtation
Contadd BEDSWE
Goman Porutattonn
Gomun JMCD WAE
02 RIDDMAE
CO2 NOD
C2L TinFiin
00
Dancan
Composz
02
Figure 2: The robustness results for gLIME (top), gSHAP (middle), and gIME (bottom). The graphs show the proportion of evaluation set instances, where the sensitive feature was recognized as the most important by the used explanation method. Rows represent the generators used for explana- tions. The column labels consist of the name of the data and the name of the generator used for training of the adversarial model. Compas2 and CC2 de- note attack with two independent features. Perturbation represents the original sampling used in LIME, SHAP, and IME, TEnsFillln represents the Tneebnsemble variant where new instances are generated around the given one, and TreeEns represents the generation from the whole distribution.
which the experiment was performed
4.3 COMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHIODS
We checked if improved data generators affect explanations in non-adversary environment. We compared the explanations of original and modified methods for four different classification models on three different datasets. As measure of differences, we use the mean of the absolute difference between the original and enhanced explanations over the whole evaluation set. Details of these ex- periments reported in Appendix below shortly summarize the results, contained in Table
The differences between original LIME and gLIME explanations are considerable (see the top part of Table 4 This is not surprising since LIME fits local linear models in its local explanations, which can strongly differ even for small perturbations of the model's input. The differences in SHAP and IME explanations that explanations of SHAP and gIME are not significantly different from SHAP and IME in the non-adversary environment.
small (the average MAD is almost negligible). We conclude
8

---

Under review as a conference paper at ICLR 2021
4.4 BEHAVIOR ADVERSARIAL CLASSIFIER WITH DIFFERENT PREDICTION THRESHOLDS
The attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method. We express different levels of attacker's conservatism by changing the decision function threshold d from Equation (T). where it currently set 0.5. We tested the robustness of modified explanation methods on the COMPAS dataset with adversarial models that use different certainty thresholds for deploying the biased classifier (simulating different levels of aggressiveness). For results and more detailed description of the experiment, see Appendix G Below, we shortly summarize the results.
Even with different thresholds, gIME is still the most robust of the three explanation methods, and treeEnsemble still gives the best results the data generator. The sensitive feature is recognized as the most important more often when the threshold is lower, which expected as this case the adversarial model behaves more aggressively. While the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher values of the thresholds, remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeEnsemble as data generator). We can con- clude that the enhanced explanation methods remain robust enough, even with more conservative adversarial models.
5 CONCLUSIONS
We presented the defense against adversarial attacks on explanation methods. The attacks exploit the shortcomings of perturbation sampling in post-hoc explanation methods. This sampling used in these methods produces instances too different from the original distribution of the training set. This allows unethical owners of biased prediction models to detect which instances are meant for explanation and label them in an unbiased way. We replaced the perturbation sampling with data generators that better capture the distribution of given data set. This prevents the detection of instances used in explanation and disarms attackers. We have shown that the modified gLIME and gSHAP explanation methods, which use better data generators, are more robust than the original variants, while IME is already quite robust. The difference in explanation values between original and enhanced gSHAP and gIME is negligible, while for gLIME, it is considerable. Our preliminary results in Appendix C show that using the Ensemble generator, the gIME method converges faster and requires from 30-50% fewer samples.
The effectiveness of the proposed defense depends the choice of the data generator and its param- eters. While the Eneebnse generator turned out the most effective in our evaluation, in practice, several variants might need to be tested to get a robust explanation method. Inspecting authorities shall be aware of the need for good data generators and make access to training data of sensitive pre- diction models a legal requirement. Luckily, even few non-deceived instances would be enough to raise the alarm about unethical models.
This work opens a range of possibilities for further research. The proposed defense and attacks shall be tested on other data sets with a different number and types of features, with missing data, and on different types of problems such as text, images, and graphs. The work on useful generators shall be extended to find time-efficient generators with easy to set parameters and the ability to generate new given one. Generative adversarial networks (Goodfellow et al. 2014) may be promising line of research. The TreelEnsemble generator, currently written in pure R, shall be rewritten in a more efficient programming language. The MCD-VAE generator shall be studied to allow automatic selection of reasonable parameters for a given dataset. In SHAP sampling, could first fix the values of the features we want to keep, and the values of the others would be generated using the Treelnsemble generator.
instances the vicinity
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv: /8066 08049, 2018.
Julia A Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May 23, 2016.
Joymallya Chakraborty, Kewen Peng, and Tim Menzies. Making fair ML software using trustworthy explanation. arXiv preprint arXiv:2007.02899; 2020.
Botty Dimanov, Umang Bhatt, Mateja Jamnik, and Adrian Weller. You shouldn't trust me: Learning models which conceal unfaimess from multiple explanation methods. In SafeAl@ AAAI, pp- 63-73, 2020.
Carl Doersch. Tutorial on variational autoencoders. ArXiv, abs/ Abs//600.05508, 2016.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances Neural Information Processing Systems, pp. 13589-13600, 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository. http: ics uci. edu ml 2019. [Accessed: 8. 2020).
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference Machine Learning, volume 48, pp. 1050-1059, 2016.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence, 33:3681-3688, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversar- ial model manipulation. In Advances in Neural Information Processing Systems, pp. 2925-2222222 2936,, 2019.
Joshua A Kroll, Joanna Huey, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robin- son, and Harlan Yu. Accountable algorithms. University of Pennsylvania Law Review, 165(3): 633-705, 2017.
Zachary Chase Lipton. The mythos of model interpretability. CoRR, abs/1606.03490 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30, pp. 3765-4774 Curran Associates, Inc., 2017.
Kristian Miok, Deng Nguyen- Doan, Daniela Zaharie, and Marko Robnik- Sobnik-Sikonja. Generating data using Monte Carlo dropout. In 2019 IEEE 15th International Conference on Intelligent Computer Communication and Processing (ICCP), pp. 509-515, 2019.
Moody and C. Darken. Fast learning in networks of locally-tuned processing units. Neural Computation, 1:281-294, 1989.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Rexearch, 12:2825-2830, 2011.
Michael Redmond and Alok Baveja. A data-driven software tool for enabling cooperative infor- mation sharing among police departments. European Journal of Operational Research, 141: 660-678, 2002.
10

---

Under review as a conference paper at ICLR 2021
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Marko Robnik-Sikonja. semiArtificial:: Generator of Semi-Artificial Data, 2019. URL https:
//eran.r-projje http://ett..ccccccccaaaaaaaaaaaa
R package version 2.3.1.
Marko Robnik-Sikoni Data generators for learning systems based on RBF networks. IEEE Trans-
actions on Neural Networks and Learning Systems, 27(5):926-955) 2016.
Marko Robnik-Sikonja and Igor Kononenko. Explaining classifications for individual instances.
IEEE Transactions on Knowledge and Data Engineering, 20:589-600, 2008
Peter J. Rousseeuw, Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computaaionnal and Applied Mathematics, 20:53 65, 1987.
Sean Saito, Eugene Chua, Nicholas Capel, and Rocco Hu. Improving lime robustness with smarter locality sampling, 2020.
Andrew D Selbst and Solon Barocas, The intuitive appeal of explainable machines. Fordham Law Review, 87:1085, 2018.
Lloyd S. Shapley. A value for n-person games. In Alvin E. Roth (ed.), The Shapley Value: Essays in Honor of Lloyd S. Shapley, pp. 31-40. Cambridge University Press, 1988.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: Adversarial attacks on post-hoc explanation methods. In AAAI/ACM Conference on AI, Ethics, and Society (AIES), 2020.
Erik Štrumbelj and Igor Kononenko. efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11:1-18, 2010.
Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41:647-665, 2013.
A DETAILS ON POST-HOC EXPLANATION METHODS
For the sake of completeness, we present further details on the explanation methods LIME (Ribeiro et al. 2016), SHAP (Lundberg Lee 2017). and IME (Strumbelj Kononenko 2013). Their com- plete description can be found in the above-stated references. In exposition of the explanation methods, we denote with the predictive model, with x the instance we are explaining, and with n the number of features describing r.
A.1 LIME
Explanations of instances with the LIME method is obtained with an interpretable model g. The model has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that it is interpretable).
The explanation of the instance x for the predictive model f. obtained with the LIME method, is defined with the following equation:
{(x) arg min R.) a(g)),
gEG
where Mgg) denotes the measure of complexity for interpretable model, πュ (2) denotes the proximity measure between and generated instances L(f, g, TT)) denotes the measure of local fidelity of interpretable model g to the prediction model f. and G' denotes the set of interpretable models. We the linear version of the LIME method, where G represents the set of linear models. With x' we denote the normalized presentation of instance æ, i.e. the numerical attributes have their mean set to 0 and variance to and the categorical attributes contain value if they have the same value

---

Under review as a conference paper at ICLR 2021
as the exoplained instance and 0 otherwise. The proximity function T. is defined on the normalized instances (hence we the notation Ta'),, and uses the exponential kernel: )z((=') dir' where d(r', z²) denotes a distance measure between x' and z'. The local fidelity measure L from Equation defined as:
C(f.g
Σ ( Z'EZ
where 2 denotes the set of samples. In LIME, each generated sample is obtained by adding Gaussian noise to each feature of x' independently.
Using linear models as the set of interpretable models, LIME relatively fast but may produce poor explanations for instances close to complex decision boundaries.
A.2 SHAP
We refer to SHAP as the method called Kernel SHAP by (Lundberg & Lee, 2017). SHAP essentially estimates Shapley values using LIME's approach, which means that the calculation of explanations is fast due the use of local linear models computed with the weighted least squares algorithm. The explanation of the instance x with the SHAP method are feature contributions éi, = 1, 2..,. n that coefficients of the linear model g(x') Φu + E 6i 2'y, obtained with LIME, where z E {0, 1} for i E [1,2,... n).
As LIME does not compute Shapley values, this property is assured with the proper selection of functions (gg), Tzr'(2), and L( 9, T2) Let us first define the function h='2²) that maps from the (0. 1 the original feature space. The function h, is defined implicitly with the equation fhi, (2')) {=)()) = I; Vi E (j:z}= = 1}}. This is the expected value the model ff when we do not know the values of features at indices where z' equals 0 (these values are hidden). Functions agg), Tee (2') and [(S, g, ) that enforce the computation of Shapley values are:
(gg) -0
Ty (2') (1))) 1//1 (n 12'1)
n
((f.).To))) = Σ (f(h)(z')) 9(21))² Te (2'),
z'eZ
where |2'| denotes the number of nonzero features of z' and 2 C 210.13"
The main purpose of the sampling in this method is to determine the value (ha(2²)) because in general predictive models cannot work with hidden values. To determine (2)))))), SHAP uses the distribution set D which we obtain from the training set. For D, SHAP takes the centroids of clusters obtained by the k-means clustering algorithm on the training set. The number of clusters is set by the user. Value of f(h is determined by the following sampling:
f(hx(z')) = =[f+)) = vi (
Σ f(Z ((
(4)
D
dED
where [[===d;=/=0] denotes instance with features that are 0 in z' being the feature values from d.
A.3 IME
The explanation of with the method IME are feature contributions é,, 1,,, Strumbelj & Kononenko (2013) shoved that Shapley values are the only solution that takes into account contri- butions of interactions for every subset of features in fair way. The i-th feature contribution can be calculated with the following expresiion:
(1(x) -
1 Σ Σp(w) p(w) TI FES, WEX
- -f
(5)
12

---

Under review as a conference paper at ICLR 2021
where S, denotes group of permutations of elements, X' denotes the training set instances, Pre (T) represents the set indices that precedes in the permutation т, ie. Pre'(a) {j:m()) < T()))). Let p(w) denote the probability of the instance w in X' and let Wiformula) denote the instance w with some of its features values changed according to the formula. To calculate (1)), have go through | n! iterations, which be slow. Therefore, the method IME uses the following sampling. The sampling population of i-th feature is V ii
for every combination of the permutation and instance w. IME draws m, samples V1, Vm, at random with repetition. The estimate of ø, (x) defined with the equation:
mi
Σ V₁.
m
Contrary to SHAP, IME does not approximation with linear models, which compute all features' contributions at once but has to compute the Shapley values by averaging over a large enough sample for each feature separately. This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency in the space normalized features.
B DEMONSTRATION OF BETTER SAMPLING IN SHAP
The graphs in Figure3Jshow the PCA based 2D space of the evaluation part of the COMPAS dataset (see Section 4 for the dataset description). The left-hand side shows the SHAP-generated sampled instances using the k-means algorithm (14 clusters determined by the silhouette score (Rousseeuw) (1987)). The sample produced with the MCD-VAL generator in gSHAP is shown on the right-hand side. This sample is much more similar the original distribution compared the SHAP sampling.
Figure 3: Visual comparison of original and sampled distributions for the COMPASS dataset. The SHAP k-means based generator (left) produces instances less similar to the original data, compared the MCD- VAE generator (right).
C IMPROVED IME CONVERGENCE RATE WITH THE TREEENSEMBLE GENERATOR
Preliminary, we tested how better generators affect the convergence rate of the IME explanation method. The preliminary results in Table Sablle]ssin show a significant reduction the number of needed samples and slight increase in the eтror compared to the original perturbation sampling. Note that the error measure deployed is biased favor of the perturbation sampling, which was used to determine the gold standard. This determined with the sampling population's 's variance, as described in Strumbelj & Kononenko (011)).
D TRAINING DISCRIMINATOR FUNCTION OF THE ATTACKER
Details of training attacker's decision models (see Figure Tp) is described in Algorithms/ T and
We used slightly different algorithm for each of the three explanation methods, LIME, SHAP.
13

---

Under review as a conference paper at ICLR 2021
COMPAS dataset
Error
# samples
Classifier
Perturb.
TEnsFilll
Perturb.
TEnsFillln
Reduction
CA %
Naive Bayes
0.0076
0.0217
18571
11278
39
83
Linear SVM
0.0033
0.0080
8244
4423
46
84
Random forest
0.0049
0.0221
45372
26960
41
80
Neural network
0.0057
0.0130
16157
8841
45
84
German dataset
Error
# samples TEnsFillln
Classifier
Perturb.
TEnsFillln
Perturb.
Reduction %
CA %
Naive Bayes
0.0076
0.0201
56052
39257
30
77
Linear SVM
0.0005
0.0013
3877
2157
44
Random forest
0.0046
0.0141
92478
66639
28
74
Neural network
26
CC dataset
Error
# samples
Classifier
Perturb.
TEnsFillin
Perturb.
TEnsFillln
Reduction %
CA %
Naive Bayes
0.0028
0.0046
32910
20117
39
70
Linear SVM
0.0009
0.0048
73324
39098
47
62
Random forest
0.0032
0.0045
109958
58852
46
79
Neural network
0.0012
0.0061
144183
70020
51
Table 2: Comparison of original perturbation sampling and the TreeFnsemble generator with data fill-in inside the IME method. The results show average scores for the evaluation set. The col- umn Perturb. presents the perturbation based sampling and TEnsFillIn presents the sampling using TreeEnsemble generator with missing parts instances filled in. The Reduction column shows reduction in the number of samples using the TEnsFillln method compared to perturbations. The CA stands for the classification accuracy of the explained classifier on the evaluation set.
and IME, as each method uses a different sampling. Algorithms first create out-of-distribttion in- stances by method-specine sampling. The training sets for decision models are created by labeling the created instances with the instances from sample S (to which the attacker has access) from distribution Xaist are labeled with 1. Finally, the machine learning model dModel is trained on this training set and returned d. In our experiments, we used random forest classifier as dModel and the training part of each evaluation dataset as S.
E HEATMAPS AS TABLES
We present the information contained in Figure in more detailed tabular form in Table
F COMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHODS
We check if improved data generators affect explanations non-adversary environment. We split the dataset into training and evaluation set in the ratio 90% 10%, and trained four classifiers from Python scikit-learn (Pedregosa et al. 2011)) library: Gaussian naive Bayes, linear SVC (SVM for classification), random forest, and neural network. We explained the predictions of each classifier on the evaluation set with every combination of explanation methods and generators used in the adversarial attack experiments. For instances in the evaluation set, measured the mean absolute
14

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Training of the decision model d, used by the attacker to distinguish between in- stances from distribution Xdist and samples produced by explanation methods LIME or gLIME.
Input: S [= training set, nSamples: number of generated instances for each instance Iy E D. gen: data generator, dModel: machine learning algorithm
Output: Classifier that outputs if its input æ is from Xdist and otherwise
X + 0 Training set for dModel
gen.fit(S) // Train the data generator on
for i = to m do
X f (zz, 1) If Add instance from distribution G - gen.newdata (nnittaannSSaa nammless, x4) // Generate nSamples new samples around x;
for j = to nSamples do // Add nSamples out distribution instances
X < X (G 0) // Add j-th instance from set G to X
end
end
d t (Model.fit((X) // Fit model dModel to set X and save it in d
return d
Algorithm 2: Training of the decision model d, used by attacker to distinguish between instances from distribution Xdist and samples produced by explanation methods SHAP or gSHAP.
Input: S (=r training set, nSammple: number of generated instances for each instance E D, k: size of the generated distribution set, gen: data generator, dModel: machine learning algorithm
Output: Classifier d that outputs 1 if its input æ is from Xaist and 0 otherwise
X 4-00 0 Training set dModel
gen.fit(S) // Train the data generator on S gen K M eans or gen == rbf DataGen or gen == Ensemble then
D gen.newdata (k) // Generate the distribution set with KMeans,, rbiDataGen treeEnsemble
end
for
i nSamples do Add nSamples of distribution instances I random instance from S
if gen = MCD - AA or gen == tree Ensemble then gen.newdata(11, 2) Generate an instance w the vicinity x
end
else // KMeans, treeFnsemble or rbfDataGen w <- take a random instance from /
end
M + choose a random subset of {1, 2,. len(x)) // Choose random features
2 x{M]] < w| [M] Replace the values of chosen features æ with values from w as in SHAP method X + X U (z, 0) // Add out of distribution instance
end
for m do // Add instances from distribution
X + XXU (zt,
end
+- fildel.fil(X // Fit model dModel to set and save it in d
return d
difference (MAD) of modified explanation methods, defined with the following equation:
MAD gen (x)) = ( - (.(2)).
where 6g (x: and dx(x)) represent the explanations of i-th feature returned by the modified and original explanation method, respectively (recall that n denotes the number of features in the data set).
We experimented on three datasets. The COMPAS dataset is described in Section 4.1 In addition that, used synthetic dataset condInd from Robnik-Sikonja & Kononenko 2008), and lonosphere
15

---

Under review as a conference paper at ICLR 2021
Algorithm 3: Training of the decision model d, used by attacker to distinguish between instances from distribution Xdist and samples produced by explanation methods IME or gIME.
Input: S (= training set, nSamples: number of generated instances for each instance Iy E D, gen: data generator, dModel: machine learning algorithm
Output: Classifier that outputs if its input æ is from Xdist and otherwise
X + 0 Training set for dModel
gen.fit(S) // Train the data generator on set
for i = to m do
X f X U (zz, 1) // Add instance distribution
for j = 1 to nSamples do // Add nSamples out of distribution instances
- gen. gen.newwata 1, 2z.) // Generate an instance the vicinity x; b - w // First out of distribution instance
2 w // Second out of distribution instance
+ choose random permutation from Sten(o,) // i.e. random permutation of s;'s
features
idæ choose random number from {1, 2, .... (en();)}
M1 < {A E {1, 2, 21., len(x;)), =kk < π (idx)) // Features that precede idr in
permutation
M₂ + M1 U {idæ}
bi nz [M] // Vector by in IME method [Strumbelj & Kononenko 2013
by Ha M2 // Vector by in IME method [Strumbelj & Kononenko 2013)
X t X U { (b1, 0), (b2, 0)} // Add out of distribution instances
end
end
d +- fftoode.fit(X)) // Fit model dModel to set X and save it in d
return d
dataset from UCI repository (Dua & Graff (2019)) Both datasets represent binary classification problem. Apart from the target variable, condInd consists of binary features, while lonosphere consists of 34 numerical attributes. The condInd datasets contains 2000 instances and Ionosphere contains 351 instances.
The results shown in Table The differences between original LIME and gLIME explanations are considerable (see the top table). This is not surprising since LIME fits local lincar models in its local explanations, which can strongly differ even for small perturbations of the model's input. SHAP and IME explanations are very similar (the average MAD almost negligible). We can conclude that explanations of gSHAP and gIME are not significantly different from SHAP and IME the non-adversary environment.
16

---

Under review as a conference paper at ICLR 2021
Penaboiio
MCDVAE
RuDuuvden
TroErro
Compas Purtelikkan
MCD-VWF
0.07
0.40
10
10
Compas FörbataCen
0.128
2008
00
10
Compas2_ Pertrbatin
01
2.006
10
1.0
Compar? MCD-WAF
0.392
0.01
0.002
10
Compasz, TreoEro
0.887
0.086
0.381
10
Patuftition
Gemman MCDVAE
1.0
1.0
1.
Serman Geb
0.35
Ganman TrweEn
Cc Partabslior
00
0:05
60
00
CC Car
00
0.30
00
00
GC_ Treetins
00
00
Portrszticn
GC2 MCD-VVV VAE
0.0
0.0
00
075
002 Ineainna
00
0.07
00
00
Portrbotion
MDD VAE
RelDussSonn
TodErs
TEnsF 8n
Compper_ MCD VAE
0.099
0.207
0.877
0.837
0.822
Compas AbrGataceen
0104
051
TreeEns
0019
Compasz Pirtubation
0.327
B417
0003
0.989
0.799
Comsant2
Compas2 RarDataee
0.19
0.154
0.88
0.754
0513
Lompaax Treetine
0222
0.102
0243
Sermen MCD-VAE
027
0.5
C181
0.75
081
Gemman_ Traeline
0.0
0940
010
0.0
0.0
CC_ Peruubtton
2.18
0.115
0.0
0.200
0:46
MCD-WAE
CC_ RMD
0945
0.965
0.0
0.866
0945
Treefrra
0.77
GP2 Perto
0.08
0085
002 MCE-VAE
0115
0.44
00
3.18
0.7
662 TrosEns
3.56
0.325
00
3.04
0.22
Neturbal
D.1
c311
0/2
Compas, MCD WAE
0998
5.463
0,877
Campue Parutatton
091
C487
677
Compon2
0.576
Samqued di
DX0
0.487
0.81
Germnn Persstation
6/7
028
MCD-VAE
0:5
German TI TrraFiir
5:
0757
6164
20, MCDA/AE
0:38
8:1
0.88
20_ TErsFil
811
0:34
2.0
Parlubation
222 MEDAAA
0.455
0106
0.46
The robustness results for gLIME (top table), gSHAP (middle table), and gIME (bottom table). The tables show the proportion of evaluation set instances, where the sensitive feature was recognized the most important by the used explanation method. Columns represent the generators used for explanations. The row labels consist of the name of the dataset on which the experiment was performed and the name the generator used for training the adversarial model. Compas2 and CC2 denote attacks with two independent features. Perturbation represents the original sampling used in LIME, SHAP, and IME, TEnsFillnn represents a variant of the TreeEnsemble generator where new instances are generated around the given one, and TreeEns represents the generation from the whole distribution.
Table 3:
17

---

Under review as a conference paper at ICLR 2021
COMPAS
lonosphere
Avcrage
Avcrage
Averoge
Bayes MCDDVAE
0.1196
0.0029
3,038
0,0002
3.11
2.0
Bayas RSF
0.1361
6,0006
0.0294
3,019
0,0002
SXM_MDD MCD VAE
0.0006
0.0
0.0557
0.0005
0.0591
0,0002
SWM RBF
0.1206
6.0011
0.0126
0.0
0.0309
0,0001
SvMLTEER
0,0744
0.0001
Porwal
0.1901
0.0079
0.0002
Porest Rer
0.1226
0.0023
0,022
0.0
0.0223
0,9001
0.1350
0,0018
0.0247
0,9001
0.0011
NP RBF
0.1523
0.0019
0.0185
0.0
0.0369
0,0001
NHL TEna
2.097
6.0006
0,0200
2,037
0,0001
lonosphere
Average MAE
Variance of MAE
Average KAAL
Variance MAE
Average MAE
varkance of MAL
Bayes MCD-VVA
0.0307
0,0007
0.0000
0.0999
Diyes_Aer
BayessTE TE1s
0.023
0.0003
0.0069
0.0
0,0079
0.0
Bayss, tb shill
0.031
0.001
0.0483
0.0001
0.0001
0.0001
MCD-VAE
0,0007
SVM
0.0101
0.0002
0.6225
S/M TEne
0.0129
0,0001
0.0369
0.0
0.012
0.0
S/M TEnefil
0,0002
0,0001
0.6262
0.0001
est, RBF
0.0343
0,0003
0.0127
0.0132
0.0
tol.
0.0211
0,0022
0.0001
0.022
0.0
Forwet
NR MCD--AR
0,0003
0.0377
6.0000
0.0193
0.0001
MR RDr
0.0262
0,0003
0.0391
0.0
0.0111
0.0
TEno
0,0364
0.0183
0.0001
COMPAS
condind
lonosphere
Avttrags
agu
Bayes
6.0329
0.0005
0.0137
0.0
Bayes__Tee Tinshi
0.0209
0,0003
0.0019
0.0
0 0.0093
0.0
SUM MLD-VAS
0.0112
0.0001
01002
U/2776
0.002
Forest,
6.0333
0.151
0.0
Formsc TErafil
6.0232
0.0002
0.0499
0.0001
0.0095
0.0
KN NCD-VAE
6.0202
6.0002
0.0118
0.0088
0.027
0.0003
Table The differences in explanations between LIME and gLIME (top table), SHAP and gSHAP (middle table), and IME and gIME (bottom table). The tables show MAD and the variance of MAD on evaluation set for every combination of the explanation method and predictive model. Columns represent MAD and the variance of MAD on used datasets. The row labels consist of the used classifier and generator.
18

---

Under review as a conference paper at ICLR 2021
G ADVERSARIAL CLASSIFERS WITH DIFFERENT PREDICTION THRESHOLDS
In a realistic scenario, the attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method. In these circumstances, the number of times when the sensitive variable is used inside the model will also depend on how aggressively the biased classifier is used. This reflection leads to another metric for the defense's 's success, which measures the proportion of instances when the sensitive variable is identified when it being used, i.c. at different levels of conservatism.
We express different levels of attacker's conservatism by changing the threshold of the decision function d. The output of the adversarial model Ct, which uses threshold is now defined with the equation:
(bx)) dprab (ar) 2 (x))),
ex
dprob(x) <
where derob (x) denotes the probability that æ is from distribution Xdise, returned by decision model d. Threshold therefore represents the required certainty of the attacker that given instance is from Xdist before it deploys the biased classifier. Larger values of t will result in biased classifier being deployed fewer times but with larger certainty that it has to be used. If we use t = 0.5 in Equation (8. we get the adversarial model that behaves identically to the one, described with Equation ① in Section 2.2
In this experiment, we used only the COMPAS dataset, which we split into the training and evalua- tion set in ratio 90% : 10%. We performed the same experiment as section 4.2 but with different thresholds for the adversarial models. We used five different thresholds: 0.3, 0.4, 0.5, 0.6 and 0.7.
We first checked how different thresholds affect the deployment of the biased model. For each gen- erator used in the adversarial model's training and the certainty threshold used by the adversarial model, we measured the proportion of instances from the evaluation set, where the biased classifier was deployed. The results are shown in Table 5 As expected, the percentage of instances where the biased classifier is deployed decreases with increasing t. The biased classifier is deployed less frequently when using treeEnsemble and rbfDataGen in its training, which is also expected, as these two generators give the best results inside the explanation methods. This means that for these gen- erators, it is more difficult for an attacker to distinguish between the actual instances and generated instances. The biased classifier is deployed the most often in the case of the LIME adversarial model and least often in the SHAP adversarial model.
Next, we tested the proportion of instances where the sensitive feature (in case of COMPAS dataset this is race) was recognized as the most important feature by the explanation method on two different subsets of instances: instances from the evaluation set on which the biased classifier was deployed and the whole evaluation set. We measured these numbers for all combinations of thresholds and explanation methods. The results are shown in Tables 6 (gLIME), 7 (gSHAP) and g (gIME) for gLIME, gSHAP, and gIME, respectively. We can observe the same pattern as in section 4.2; gIME is still the most robust, and the Eneemble generator still gives the best results. The sensitive attribute is recogmized as the most important feature more often when the decision threshold values are lower, which is expected as in these cases, the adversarial model behaves more aggressively. Whilst the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher thresholds, it remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeensemble data generator). From that, we can conclude that modified explanation methods remain robust enough, even with more conservative adversaries.
19

---

Under review as a conference paper at ICLR 2021
0.3
0.4
0.5
0.0
0.7
Pwrburs
100.0
40.27
RSSP
83.00
79.45
TEns
85.28
74.92
53.88
27.67
14.4
Perturb.
99.84
98.87
98.22
96.76
95.31
MCD-VAE
0.3
0.4
0.5
0.6
0.7
Pwrturh
09 15
06.79
79.77
TEnc
02.3
51.29
43.04
35.76
30.59
Perturh
09 16
69.00
m7 16
75.7
70.55
MCD.VAE
83.01
79.13
TEna
TEnsFill
TDmsill
Table 5: Proportions of instances deployed for adversarial LIME (top table), SHAP (middle table) and IME model (bottom table). Columns represent different threshold used for deploying the biased classifier. The rows represent the generator used in training of the adversarial attack. The labels "One unrelated" and "Two unre- lated" represent the attacks with one or two unrelated features.
the evaluation set on which the biased classifier was
20

---

Under review as a conference paper at ICLR 2021
0.
Blascd pred
AI
Blased prod.
Based prod.
Biased pred.
Al
Sissed pred.
Petatb, NCD-VAE
93.67
59.63
00.6)
92.68
99.67
91.56
39
90.68
90.67
90.48
Peturo,_TEe TErs
29.62
19.63
99.57
98.68
12
1.46
0.0
0.0
0.0
VAD
0.85
MCD-VA, VCD-ME
9.52
9.39
2.7
6.50
24.05
52.43
MCD V4C RBF
45.25
44.5
45.93
44.00
0.5
0.0
0.21
0.30
0.9
0.65
RSP Perarb.
300.0
100.0
103.0
100.0
100.0
300.0
100.0
100.0
100.0
100.0
R3F RBF
0.0
0.9
3.5
0.0
0.3
0.0
0.0
0.0
0.0
0.0
TEix Pertarks
300.
102.0
103.0
100.0
182.5
300.0
100.0
100
300.8
100.0
200.0
100.0
103.0
200.0
100.0
300.0
100.0
100
100.0
100.0
TExx R8F
300.0
100.5
300.0
100.0
IEna, 12na
200.0
100.0
103.0
200.0
$9.95
14.8
0.0
0.0
0.0
0.0
Biosed prod.
A8
Seased pred.
Baascd prod.
Biosed prod.
A1
Blosed pred.
Perurb. Pertutt.
Peturb, MCD-VAL
$9.00
90.00
95.87
59.05
90.57
99.00
29.07
90.03
92.57
90.00
Peturo, TEs6
99.63
93.60
95.79
90.63
99.59
98.68
100.5
99.98
100.0
98.68
NCD-VAE VED-ME
32 14
51.88
28.38
28:16
26 14
23.89
13.53
23.3
21.31
21.2
30.57
20.11
VAE
36.08
16.0
Perturt
102.0
300.0
200.0
100.0
300.0
200.0
100.C
180.0
200.0
200.0
REF R3F
39 34
41.75
36.75
35.44
3.24
2.75
3.30
2.75
5.0
0.0
TEAs
TEns Porturb.
103.5
100.0
100.0
102.0
100.0
350.0
100.
105.0
100.0
100.0
TEr. NCD-VAF
103.
100.0
108.0
300.0
100
Tin RBT
103.
1000
3000
100.0
106.0
200.0
199.
105.0
300.0
1000
TEns
103.0
100.0
100.0
100.0
300.0
200.0
100.0
100.0
3.88
Table 6: Percentages of instances where the sensitive feature (race) was recognized as the most important feature with gLIME for adversarial attacks with one (top table) two unrelated features (bottom table). The columns labeled Biased pred. represent the results on instances on which the biased classifier was deployed, while the columns labeled All represent the results on the whole evaluation sets. The numbers above represent the used threshold. The row labels are in the form where <explainer>> denotes the generator used the explanation method and <adversarial> denotes the generator used in the training of the adversarial model.
21

---

Under review as a conference paper at ICLR 2021
Baced
M
Blascd
Perturh, Pertatt.
a0.05
16.0
$1.11
27.00
34.05
22.01
21.31
19.00
28.74
13.13
Perturb, 3ID
99.63
94.34
59.03
97.61
36.99
66.08
94.21
80.1
88.66
70.55
Petturt, R3F
59.75
90.10
57.99
82.52
$3.99
50.73
61.38
39.48
30.06
20.71
Portur, TEis
59.77
90.58
63.15
50.32
32.33
22.33
19.0
11.0
23.76
5.90
WCG-VE Pet-vss_Pemtrh Pemur
13.46
51.28
12.66
26.35
61.7
BLU--AE NLD-VE
41.91
35.79
PBF
RBF
RII MCCD-ME
$0.44
03.07
09.7
IBF RBF
46.43
37.22
43.5
28.16
18.61
12:46
0.54
6.15
375
1.34
FBT Tins
06.09
41.71
10.01
20.23
30.9
0.63
7.36
1.70
0.0
0.0
15nu Naraurt.
190.0
90.00
190.0
95.20
29 39
92.07
95.57
57.38
95.62
79:13
TEns, MCD
100.0
91.90
100.0
95:05
100.6
91.75
96.57
67.7
98.39
18.96
TEroRil MCD-VAS
96.
88
77.02
66.7
Thndiil SAr
15.01
16
TEnoFit Lins
65.19
46.19
43:52
20.96
28.32
17.15
23.00
11.49
27.40
0.3
0.4
0.5
0.6
0.7
Bassed prwd
81
Bisad pred.
Rused peec.
A11
Brandd pred.
71
Ruaad pred.
Partutb, Partul.
66.06
63.08
57.87
50.0
41.45
34.79
24.68
19.26
13.90
10.19
Perturb, MCD
99.82
95.63
09.81
94.17
99.18
90.61
38.09
83.76
93
78.64
27.51
28
30.59
20:23
MCD-WA
20.71
10.95
30.00
M.D-WA Iirs
31.34
22.29
50.97
38.19
36.11
23.92
22,87
13.89
06
8.9
RBF Perturt.
99.04
98.05
98:48
94.02
97:47
90.70
30.04
01.25
92.05
77.51
IBF McDOVAE
90.45
36.95
96.1
93.59
92.54
85:44
$7.47
70.54
18.47
63.43
RBF REF
20.22
59.39
60 14
45:05
59.25
35.6
68.55
20.54
37.31
19:74
RMV Tims
25:21
90.97
41.09
25.8
17.42
18.04
Têne
88.03
62.46
55.67
22.49
Ibrand /hraab.
59.97
92.05
83.92
$0.24
95
12.4
TEndiil MCD VAE
96.32
88.03
95.73
94.14
92.84
78.8
89.95
72.92
85.42
62.11
Tinsh REP
83.35
20.87
73
56.5
05.3
41.06
59.09
30.23
33.09
28.15
TErsfi Ters
72.05
$2.91
50 65
32.04
37.5
20.39
29.15
14.08
24.23
10.52
Table 7: Percentage of instances where the sensitive attribute (race) was recognized the most important feature with gSHAP for adversarial attacks with one (top table) or two unrelated fea- tures (bottom table). The columns labeled Biased pred. represent the results on instances on which the biased classifier was deployed, while columns labeled All represent the results evaluation sets. The numbers above represent the used threshold. The row labels are in the form <explainer>> -cciiiierss.caddessrrriia where <explainer>> denotes the generator used in the explanation method and <adversarial> denotes the generator used in the training of the adversarial model.
the whole
22

---

Under review as a conference paper at ICLR 2021
0.3
0.4
0.5
0.6
0.7
Bisced pred.
AI
Blaced pred.
3eedod prod
AI
Blacod prod.
Blased pred.
All
Perturb, Parturb
59.22
97.22
92.23
88.37
76.7
68.8
51.78
41.88
24.6
Pertubb MCD VAE
100.
103.0
95.53
1000
19.55
00.18
95.45
00.1
Peturb, Thrardl
97.
90.31
87.54
85.03
08.00
48.22
VAp Perturt.
500
57.34
10.81
39.96
10.13
47.23
31.13
25.24
MID-MEE NCD-MME
37 92
34.47
47.35
30
21.19
TErsfil
TExFFi Patuat..
89.16
TExsFI MCD-VAS
TEnaiil
0,3
0.4
0.5
0.5
0.7
Resed pred.
A1
Blased prod
Biased pred
A1
Blasdd pred.
AI
Blases pred.
All
Aeturt, Nester
1b.A7
V8.49
90.15
98.14
$4.81
99.15
95.82
WL.u
94.17
Perturt, MED--ME
59.8
97.73
99.1
97.57
99.81
97.57
59.8
97.57
99.78
91.73
Peturt: TErFFI
93.39
91.75
94.56
91.26
96.7
91.50
07.91
91.01
98.14
92.07
KCO VAE Pertrb.
36.18
54.60
96.16
$2.75
53.35
48.38
51.25
40.51
48.85
48.38
NCO VAE MCD VAE
29.91
12.07
09.61
02.01
33.03
TENSFE
06.96
02 04
08.66
61.66
64.5
57.28
L
90.29
Table 8: Percentages of instances where the sensitive attribute (race) was recognized as the most important feature with gIME for adversarial attacks with one (top table) or two unrelated features (bottom table). The columns labeled Biased pred. represent the results on instances on which the biased classifier was deployed, while the columns labeled All represent the results the whole evaluation sets. The numbers above represent the used threshold. The row labels are in the form susplainer>=<addvrsaarrt where <explainer'> denotes the generator used in the explanation method and <adversarial> denotes the generator used in the training of the adversarial model.
23


Reviews:
[{"review_id": "vDQRQ1dDEot", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a defense against the adversarial attacks on explanation methods described in Slack et al. (2020). In particular, by using sampling methods that more closely resemble the original data distribution, the authors make it difficult for the Out-of-Distribution detector to successfully discriminate between instances used for predicting and instances used for explaining.\n\nPositives: The paper represents a nice use case for this suite of data generators. The necessary background information is explained well, and the testing is thorough with respect to comparing robustness across the data generators\n\nNegatives: Firstly,  I'm concerned that the evaluation metric you use does not fully  capture the nature of the problem. In particular, it is not clear that in testing you ensure that the biased classifier is deployed on all testing instances. The % of the time that the sensitive variable appears in the top position will also depend on how aggressively the biased classifier is used. When testing/explaining points are separable, this is less of an issue, as the biased explainer will be used on all (or almost all) testing points. However, the purpose of your method is to make testing/explaining points more difficult to distinguish. In this case we might imagine that the biased explainer is being used more conservatively - that is, if the adversarial classifier cannot distinguish between testing points and sampling points and therefore deploys the unbiased classifier on some testing points, the sensitive attribute may correctly go undetected. The true metric of interest should relate to the % time the sensitive variable is identified when it is being used, at different levels of conservatism. \n\nThe authors of LIME, SHAP, and IME make careful design choices for reasons of 1) breaking data correlations, 2) ensuring the satisfaction of certain axioms, and 3) run-time. It would be important to relate this new methodology to its predecessors along these lines as well. In particular, sampling only from the manifold of the original training data can be expected to maintain the correlation structure of the original data. This is useful for fooling the adversarial classifier but you may sacrifice an ability to differentiate between the model's usage of correlated features.\n\nI'd like to see some sensitivity analysis to the number of samples. You point out that IME is \"already quite robust\" but certainly this seems counterintuitive at small sample sizes. At which sample size does this become true?\n\nRelated work: Saito et al. address this problem concurrently https://arxiv.org/pdf/2006.12302.pdf\n\nAll in all, I find the work to be a useful step forward, but believe that it would benefit from more thorough analysis before publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sytKzhZ_Wd_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nFollowing the work of Slack et al (2020), which presents adversarial attacks on explanations, this work proposes a solution, that is to use improved perturbation data generators that produce instances more similar to samples in the training set. \nThis work also shows that the IME method is more resilient to adversarial attacks in comparison to LIME & SHAP, while both LIME and SHAP would benefit from the proposed data generators. \n\n+ves: \n-------\n- Overall, the solution to use improved data generators that closely match the training data distribution is a good one including the comparison between the different data generators. The result on the robustness of IME method is good. \n- Authors have submitted modified version of the code i.e. gLIME & gSHAP which use the proposed improved data generators. Both the code and the result on IME is expected to benefit the AI explainability community and practitioners that more or less rely on either LIME or SHAP today. \n\nPossible improvements\n--------------------------------\n- It would be good to comment on how the data generators work on images. \n- Using training data distribution may perhaps improve the overall quality of explanations as well i.e. beyond making them robust to adversarial attacks, it might be good to discuss any such benefits in the paper by considering explainability metrics such as monotonicity, faithfulness, etc. \n- One thing which is unclear is author's recommendation on which data generators ( among the 3 evaluated ) to eventually use - what are their pros/cons. Does this depend on the type / distribution of data or explainability method or both. \n\nConclusion: \n---------------\nOverall, this is a nice piece of work which leverages existing data generators to show that adversarial robustness of LIME & SHAP based explanations can be improved. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "R85WAl7Ay39", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "ICLR 2021 Review - Better Sampling in Explanation Dieselgate\n\nSummary: The paper suggests to replace the perturbations part for the existing post-hoc explanation methods like LIME and SHAP with on-data manifold sampling methods. \n\nSHAP and LIME use perturbations or randomly generated points to explain the decision of the black-box models. These points are out-of-distribution data, that leads to a new avenue for adversarial behavior discussed in Slack et al. (2020). The authors use existing data generators to produce better perturbations. They further empirically evaluate the robustness of explanations generated after proposed changes on real-life datasets.\nComments:\n1. It is not clear what exactly the contribution of the paper. The problem is identified by existing papers [slack et al (2020), https://arxiv.org/pdf/2007.09969.pdf], etc, and mentioned that such attacks fail trivially if perturbations are from data distributions. The data generators are used from the existing literature. A recent paper [https://arxiv.org/pdf/2006.01272.pdf] proposes more efficient and theoretically sound on-data manifold SHAP computations.  \n2. The definition of robustness is not formally stated in the paper. The usual robustness in explanations [https://arxiv.org/pdf/1806.08049.pdf] bounded/negligible change in the explanation if the point of interest it changed slightly. It is not clear how random perturbations around the point of interest affect robustness.\n3. The evaluations in the paper are weak, it is trivial that if perturbations are from data distributions the attack proposed in Slack et. al (2020) will fail (it is discussed in Slack et. al (2020) as well). Moreover, the paper does not evaluate the effects of used sampling methods in explanation. The data generating model itself a black-box model and involves more uncertainties in explanations. Minor: why one can’t use the training dataset itself to generate model explanations rather than using black-box data generators?  \n \n\n**After Rebuttal** \n\nI would like to thank the authors for their rebuttal. I agree that it is not fair to assess the merits of the current work based on papers that were not available at the time of submission (or that, strictly speaking, have not been published at the time of submission). Indeed, to an extent, pointing out the ArXiv paper encourages authors to simply submit their works there to get a \"publication\" stamp, which on a community level is undesirable (papers on ArXiv aren't reviewed and citing them as scientific sources is problematic to say the least). I suppose the only point is that there exist works that do similar things in a more compelling fashion. \n\nIt's encouraging to see that the authors checked for robustness of their method, and I appreciate the efforts. \n\nWith these two issues resolved to a certain extent, I am willing to increase my score. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JZFhzaFmFMU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on the adversarial scenario presented in Slack 2020 \"Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods\": an adversarial entity can design a model with obvious biases that will look innocuous to regulators when analyzed by post-hoc explanation methods such as LIME, SHAP, etc. This is achieved by leveraging the idea that the perturbations used by LIME, SHAP and other methods follow a different distribution than the original data, and therefore the adversary can learn how to distinguish the perturbed samples from the real ones and then run an unbiased version of the model when it is being probed.\n\nThis paper looks at this scenario from the eyes of the regulator that has to probe the model to decide whether it is biased. The main proposal of the paper is to alter the way in which LIME, SHAP, etc generate the perturbations needed to compute the explanation. In particular, this paper proposes to use perturbations that closely follow the data distribution, making it harder for the adversary to distinguish between genuine samples (that should go through the biased model) and perturbed samples (that should go through the unbiased model, as they imply that the model is being probed.)\n\nAlthough I liked the idea exposed in the paper and enjoyed reading the background and related work, the experimental section and the conclusions interpreted from results seem a bit preliminary. \n\nExperimentation is not very thorough, covering only robustness of the proposed sampling when pairing different generators and discriminators. The only quantitative results are provided through Figure 2 and are color coded, making them hard to compare. A table would have likely been a better way of presenting these results. More details on how the discriminator d in (eq(1)) is designed and trained would also have been of interest, particularly since different discriminators could have been evaluated.\n\nAdditional comments and questions:\n- Figure 2, the use of green and red colors is inconsistent between what is described in text and figure. Text says \"The green colour means that the explanation method was deceived in less than 30% of cases, and the red means that it was deceived in more than 70% of cases\" but figure legend has 0 to 0.5 being red and 0.5 to 1 being green. \n- One underlying assumption is that changing the perturbation used by the explanation method will not hinder the validity of the explanations. Yet, of course, explanation methods are sensitive to how the perturbations are created (trivially, one could use Gaussian noise with a very large variance to create perturbations that are not useful to generate good explanations). The paper focuses on the impact on the robustness to attacks, but more discussion and empirical results about the impact on explainability of the original method would be required.\n- From section 4.2, \"We consider deception successful if the sensitive feature was not recognized as the most important by the explanation method\". Does that mean that deception can't be successful for samples where the sensitive feature is the most important feature on the unbiased model? Or are these features removed in the unbiased model? It is not completely clear from the explanation.\n- How is the discriminator d (eq(1)) defined/trained? I could not find this information in the paper.\n- Learning of the perturbation requires access to copious amounts of data from the real distribution, which may not actually be accessible to the regulator, rendering some/all of the defenses ineffective. \n- \"Dieselgate\" is not common term?\n\nOverall, despite the interesting idea, the paper looks to be in a preliminary state.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The overall impression on the paper is rather positive, however, even after rebuttal, it still seem that the paper requires further work and definitely a second review round before being ready for publication. Thus, I encourage the authors to continue with the work started during the rebuttal to address the reviewers' comment, which although moved in the right direction would still benefit from further work.  Especially, I believe the experiments could be significantly improved (by for example bringing some results to the main paper). Moreover, a more thorough comparison theoretically and empirically with previous work would increase the impact of the paper. 
Decision: Reject