Year: 2020

Abstract:
Machine learning models are used in many sensitive areas where, besides predictive accuracy, their comprehensibility is also important. Interpretability of prediction models is necessary to determine their biases and causes of errors and is a prerequisite for users’ confidence. For complex state-of-the-art black-box models, post-hoc model-independent explanation techniques are an established solution. Popular and effective techniques, such as IME, LIME, and SHAP, use perturbation of instance features to explain individual predictions. Recently, Slack et al. (2020) put their robustness into question by showing that their outcomes can be manipulated due to poor perturbation sampling employed. This weakness would allow dieselgate type cheating of owners of sensitive models who could deceive inspection and hide potentially unethical or illegal biases existing in their predictive models. This could undermine public trust in machine learning models and give rise to legal restrictions on their use. We show that better sampling in these explanation methods prevents malicious manipulations. The proposed sampling uses data generators that learn the training set distribution and generate new perturbation instances much more similar to the training set. We show that the improved sampling increases the LIME and SHAP’s robustness, while the previously untested method IME is already the most robust of all.

heading: 1 INTRODUCTION
text: Machine learning models are used in many areas where besides predictive performance, their comprehensibility is also important, e.g., in healthcare, legal domain, banking, insurance, consultancy, etc. Users in those areas often do not trust a machine learning model if they do not understand why it made a given decision. Some models, such as decision trees, linear regression, and naı̈ve Bayes, are intrinsically easier to understand due to the simple representation used. However, complex models, mostly used in practice due to better accuracy, are incomprehensible and behave like black boxes, e.g., neural networks, support vector machines, random forests, and boosting. For these models, the area of explainable artificial intelligence (XAI) has developed post-hoc explanation methods that are model-independent and determine the importance of each feature for the predicted outcome. Frequently used methods of this type are IME (Štrumbelj & Kononenko, 2013), LIME (Ribeiro et al., 2016), and SHAP (Lundberg & Lee, 2017).
To determine the features’ importance, these methods use perturbation sampling. Slack et al. (2020) recently noticed that the data distribution obtained in this way is significantly different from the original distribution of the training data as we illustrate in Figure 1a. They showed that this can be a serious weakness of these methods. The possibility to manipulate the post-hoc explanation methods is a critical problem for the ML community, as the reliability and robustness of explanation methods are essential for their use and public acceptance. These methods are used to interpret otherwise black-box models, help in debugging models, and reveal models’ biases, thereby establishing trust in their behavior. Non-robust explanation methods that can be manipulated can lead to catastrophic consequences, as explanations do not detect racist, sexist, or otherwise biased models if the model owner wants to hide these biases. This would enable dieselgate-like cheating where owners of sensitive prediction models could hide the socially, morally, or legally unacceptable biases present in their models. As the schema of the attack on explanation methods on Figure 1b shows,
owners of prediction models could detect when their models are examined and return unbiased predictions in this case and biased predictions in normal use. This could have serious consequences in areas where predictive models’ reliability and fairness are essential, e.g., in healthcare or banking. Such weaknesses can undermine users’ trust in machine learning models in general and slow down technological progress.
In this work, we propose to change the main perturbation-based explanation methods and make them more resistant to manipulation attempts. In our solution, the problematic perturbation-based sampling is replaced with more advanced sampling, which uses modern data generators that better capture the distribution of the training dataset. We test three generators, the RBF network based generator (Robnik-Šikonja, 2016), random forest-based generator, available in R library semiArtificial (Robnik-Šikonja, 2019), as well as the generator using variational autoencoders (Miok et al., 2019). We show that the modified gLIME and gSHAP methods are much more robust than their original versions. For the IME method, which previously was not analyzed, we show that it is already quite robust. We release the modified explanation methods under the open-source license1.
In this work, we use the term robustness of the explanation method as a notion of resilience against adversarial attacks, i.e. as the ability of an explanation method to recognize the biased classifier in an adversary environment. This type of robustness could be more formally defined as the number of instances where the adversarial model’s bias was correctly recognized. We focus on the robustness concerning the attacks described in Slack et al. (2020). There are other notions of robustness in explanation methods; e.g., (Alvarez-Melis & Jaakkola, 2018) define the robustness of the explanations in the sense that similar inputs should give rise to similar explanations.
The remainder of the paper is organized as follows. In Section 2, we present the necessary background and related work on explanation methods, attacks on them, and data generators. In Section 3, we propose a defense against the described weaknesses of explanation methods, and in Section 4, we empirically evaluate the proposed solution. In Section 5, we draw conclusions and present ideas for further work.

heading: 2 BACKGROUND AND RELATED WORK
text: In this section, we first briefly describe the background on post-hoc explanation methods and attacks on them, followed by data generators and related works on the robustness of explanation methods.

heading: 2.1 POST-HOC EXPLANATION METHODS
text: The current state-of-the-art perturbation-based explanation methods, IME, LIME, and SHAP, explain predictions for individual instances. To form an explanation of a given instance, they measure the difference in prediction between the original instance and its neighboring instances, obtained
1https://anonymous.4open.science/r/5d550c62-5c5c-4ee3-81ef-ab96fe0838ca/
with perturbation sampling. Using the generated instances, the LIME method builds a local interpretable model, e.g., a linear model. The SHAP and IME methods determine the impact of the features as Shapley values from the coalitional game theory (Shapley, 1988). In this way, they assure that the produced explanations obey the four Shapley fairness axioms (Štrumbelj & Kononenko, 2013). Due to the exponential time complexity of Shapley value calculation, both methods try to approximate them. The three methods are explained in detail in the above references, and a formal overview is presented in Appendix A, while below, we present a brief description. In our exposition of the explanation methods, we denote with f the predictive model and with x the instance we are explaining.
Explanations of instances with the LIME method is obtained with an interpretable model g. The model g has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that it is interpretable). Therefore in LIME, g is a linear regression model trained on the instances sampled in the neighborhood of the explained instance x, which is first normalized.The perturbed explanation instances are generated by adding Gaussian noise to each feature of x independently. Using linear models to determine the contributions of features, LIME is fast but may produce poor explanations for instances close to complex decision boundaries.
SHAP estimates Shapley values using the approach of LIME (i.e. local linear models computed with the weighted least squares algorithm) but using a different neighborhood and loss function for local accuracy, which ensures that Shapley values are obtained. The sampling of explanation instances in SHAP relies on the distribution set D, obtained from the training set, by computing the centroids of k-means clusters.
IME computes Shapley values using a pure sampling approach (no proxy with an interpretable model). Its sampling replaces part of the explained instance with existing values of selected features and then averages differences in the prediction of explained instance and samples. This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency.

heading: 2.2 ATTACKS ON EXPLANATIONS
text: The idea of the attack on explanations presented in Slack et al. (2020) is that the attacker creates an adversarial (biased) model whose behavior it wants to hide from explanation methods, e.g., a racist model that does not grant credits to blacks. As Figure 1b illustrates, a part of the attacker’s adversarial model is an unbiased model, which, e.g., does not take disputed attributes such as race into account. The adversarial model tries to manipulate the explanation method by behaving like an unbiased model on perturbed instances used for the explanation – in this case, it uses the unbiased model’s output. On instances from the original distributions that are a part of normal model use, the adversarial model uses the biased model’s output. To distinguish between the original and perturbation-sampled data, the adversarial model contains a decision model that selects the right type of model (biased or unbiased) based on the predicted type of instances.
The reason that the decision function can distinguish between the original data and sampled instances are the shortcomings of explanation methods that use the perturbation sampling (see Figure 1a). For that graph, we created new instances based on the COMPAS dataset (see Section 4) with the Gaussian noise as in the LIME explanation method. We mapped the original and sampled instances to 2D space using the PCA transformation. We can see that the sampled data is distributed differently than the original data. As the difference is revealed with PCA, other machine learning algorithms can also effectively distinguish between sampled and original data.
The attack can be formally defined in the following way. The attacker has a biased model b and a sample S from the distribution Xdist. The attacker has no additional information about the distribution Xdist. The attacker’s goal is to produce an adversarial model e that behaves like b on the data from Xdist but does not reveal b’s bias to the explanation methods. We assume that the attacker also has an unbiased model ψ that hides the f ’s bias. The attacker creates a decision model d, that should output 1 if the input instance x is from Xdist and 0 otherwise. The model d is trained on S and generated perturbation samples. The attacker creates the adversarial model e with its output defined by the following equation:
e(x) =
{ b(x), d(x) = 1
ψ(x), d(x) = 0 (1)

heading: 2.3 DATA GENERATORS
text: We used three different data generators based on different algorithms, modeling the distribution of the training set: variational autoencoder with Monte Carlo dropout (Miok et al., 2019), RBF network (Robnik-Šikonja, 2016), and random forest ensemble (Robnik-Šikonja, 2019). In the remainder of the paper, we refer to the listed generators consecutively as MCD-VAE, rbfDataGen, and TreeEnsemble.
Autoencoder (AE) consists of two neural networks called encoder and decoder. It aims to compress the input instances by passing them through the encoder and then reconstructing them to the original values with the decoder. Once the AE is trained, it can be used to generate new instances. Variational autoencoder (Doersch, 2016) is a special type of autoencoder, where the vectors z in the latent dimension (output of the encoder and input of the decoder) are normally distributed. Encoder is therefore approximating the posterior distribution p(z|x), where we assume p(z|x) ∼ N (µx,Σx). The generator proposed by Miok et al. (2019) uses the Monte Carlo dropout (Gal & Ghahramani, 2016) on the trained decoder. The idea of this generator is to propagate the instance x through the encoder to obtain its latent encoding z. This can be propagated many times through the decoder, obtaining every time a different result due to the Monte Carlo dropout but preserving similarity to the original instance x.
The RBF network (Moody & Darken, 1989) uses Gaussian kernels as hidden layer units in a neural network. Once the network’s parameters are learned, the rbfDataGen generator (Robnik-Šikonja, 2016) can sample from the normal distributions, defined with obtained Gaussian kernels, to generate new instances.
The TreeEnsemble generator (Robnik-Šikonja, 2019) builds a set of random trees (forest) that describe the data. When generating new instances, the generator traverses from the root to the leaves of a randomly chosen tree, setting values of features in the decision nodes on the way. When reaching a leaf, it assumes that it has captured the dependencies between features. Therefore, the remaining features can be generated independently according to the observed empirical distribution in this leaf. For each generated instance, all attributes can be generated in one leaf, or another tree can be randomly selected where unassigned feature values are filled in. By selecting different trees, different features are set in the interior nodes and leaves.

heading: 2.4 RELATED WORK ON ROBUSTNESS OF EXPLANATIONS
text: The adversarial attacks on perturbation based explanation methods were proposed by Slack et al. (2020), who show that LIME and SHAP are vulnerable due to the perturbation based sampling used. We propose the solution to the exposed weakness in SHAP and IME based on better sampling using data generators adapted to the training set distribution.
In general, the robustness of explanation methods has been so far poorly researched. There are claims that post-hoc explanation methods shall not be blindly trusted, as they can mislead users (deliberately or not) and disguise gender and racial discrimination (Lipton, 2016). Selbst & Barocas (2018) and Kroll et al. (2017) showed that even if a model is completely transparent, it is hard to detect and prevent bias due to the existence of correlated variables.
Specifically, for deep neural networks and images, there exist adversarial attacks on saliency map based interpretation of predictions, which can hide the model’s bias (Dombrowski et al., 2019; Heo et al., 2019; Ghorbani et al., 2019). Dimanov et al. (2020) showed that a bias of a neural network could be hidden from post-hoc explanation methods by training a modified classifier that has similar performance to the original one, but the importance of the chosen feature is significantly lower.
The kNN-based explanation method, proposed by Chakraborty et al. (2020), tries to overcomes the inadequate perturbation based sampling used in explanation methods by finding similar instances to the explained one in the training set instead of generating new samples. This solution is inadequate for realistic problems as the problem space is not dense enough to get reliable explanations. Our defense of current post-hoc methods is based on superior sampling, which has not yet been tried. Saito et al. (2020) use the neural CT-GAN model to generate more realistic samples for LIME and prevent the attacks described in Slack et al. (2020). We are not aware of any other defenses against the adversarial attacks on post-hoc explanations.

heading: 3 ROBUSTNESS THROUGH BETTER SAMPLING
text: We propose the defense against the adversarial attacks on explanation methods that replaces the problematic perturbation sampling with a better one, thereby making the explanation methods more robust. We want to generate the explanation data in such a way that the attacker cannot determine whether an instance is sampled or obtained from the original data. With an improved sampling, the adversarial model shown in Figure 1b shall not determine whether the instance x was generated by the explanation method, or it is the original instance the model has to label. With a better data generator, the adversarial model cannot adjust its output properly and the deception, described in Section 2.2, becomes ineffective.
The reason for the described weakness of LIME and SHAP is inadequate sampling used in these methods. Recall that LIME samples new instances by adding Gaussian noise to the normalized feature values. SHAP samples new instances from clusters obtained in advance with the k-means algorithm from the training data.
Instead of using the Gaussian noise with LIME, we generate explanation samples for each instance with one of the three better data generators, MCD-VAE, rbfDataGen, or TreeEnsemble (see Section 2.3). We call the improved explanation methods gLIME and gSHAP (g stands for generatorbased). Using better generators in the explanation methods, the decision function in the adversarial model will less likely determine which predicted instances are original and which are generated for the explanation purposes.
Concerning gLIME, we generate data in the vicinity of the given instance using MCD-VAE, as the LIME method builds a local model. Using the TreeEnsemble and rbfDataGen generators, we do not generate data in the neighborhood of the given instance but leave it to the proximity measure of the LIME method to give higher weights to instances closer to the explained one.
In SHAP, the perturbation sampling replaces the values of hidden features in the explained instance with the values from the distribution set D. The problem with this approach is that it ignores the dependencies between features. For example, in a simple dataset with two features, house size, and house price, let us assume that we hide the house price, but not the house size. These two features are not independent because the price of a house increases with its size. Suppose we are explaining an instance that represents a large house. Disregarding the dependency, using the sampled set D, SHAP creates several instances with a low price, as such instances appeared in the training set. In this way, the sampled set contains many instances with a small price assigned to a large house, from which the attacker can determine that these instances were created in perturbation sampling and serve only for the explanation.
In the proposed gSHAP, using the MCD-VAE and TreeEnsemble generators, the distribution set D is generated in the vicinity of the explained instance. In the sampled instances, some feature values of the explained instance are replaced with the generated values, but the well-informed generators consider dependencies between features detected in the original distribution. This will make the distribution of the generated instances very similar to the original distribution. In our example, the proposed approach generates new instances around the instance representing a large house, and most of these houses will be large. As the trained generators capture the original dataset’s dependencies, these instances will also have higher prices. This will make it difficult for the attacker to recognize the generated instances used in explanations. The advantage of generating the distribution set close to the observed instance is demonstrated in Appendix B.
The rbfDataGen generator does not support the generation of instances around a given instance. Therefore, we generate the sampled set based on the whole training set and not for each instance separately (we have this option also for TreeEnsemble). This is worse than generating the distribution set around the explained instance but still better than generating it using the k-means sampling in SHAP. There are at least three advantages. First, the generated distribution set D can be larger. The size of the k-means distribution set cannot be arbitrarily large because it is limited by the number of clusters in the training set. Second, the centroids of the clusters obtained by the k-means algorithm do not necessarily provide a good summary of the training set and may not be valid instances from training set distribution. They also do not capture well the density of instances (e.g., most of the data from the training set could be grouped in one cluster). Third, using the proposed generators, SHAP becomes easier to use compared to the k-means generator, where users have to determine
the number of clusters, while the data generators we tested can be used in the default mode without parameters,

heading: 4 EVALUATION
text: To evaluate the proposed improvements in the explanation methods, we first present the used datasets in Section 4.1, followed by the experiments. In Section 4.2, we test the robustness of gLIME, gSHAP, and gIME against the adversarial models. To be more realistic, we equipped the adversarial models with the same improved data generators we used in the explanation methods. It is reasonable to assume that attackers could also use better data generators when preparing their decision function, making their attacks much stronger. As the evaluation metric for the success of deception, we use the proportion of instances where the adversarial model deceived the explanation methods so that they did not detect sensitive attributes as being important in the prediction models. In Section 4.3, we test if enhanced generators produce different explanations than the original ones. As the attacker might decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method, we test different thresholds of the decision function d from Equation (1) (currently set to 0.5). We report on this analysis in Section 4.4.

heading: 4.1 SENSITIVE DATASETS PRONE TO DECEPTION
text: Following (Slack et al., 2020), we conducted our experiments on three data sets from domains where a biased classifier could pose a critical problem, such as granting a credit, predicting crime recidivism, and predicting the violent crime rate. The basic information on the data sets is presented in Table 1. The statistics were collected after removing the missing values from the data sets and before we encoded categorical features as one-hot-encoded vectors.
COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a risk assessment used by the courts in some US states to determine the crime recurrence risk of a defendant. The dataset (Angwin et al., 2016)) includes criminal history, time in prison, demographic data (age, gender, race), and COMPAS risk assessment of the individual. The dataset contains data of 6,172 defendants from Broward Couty, Florida. The sensitive attribute in this dataset (the one on which the adversarial model will be biased) is race. African Americans, whom biased model associates with a high risk of recidivism, represent 51.4% of instances from the data set. This set’s target variable is the COMPAS score, which is divided into two classes: a high and low risk. The majority class is the high risk, which represents 81.4% of the instances.
The German Credit dataset (German for the rest of the paper) from the UCI repository (Dua & Graff, 2019) includes financial (bank account information, loan history, loan application information, etc.) and demographic data (gender, age, marital status, etc.) for 1,000 loan applicants. A sensitive attribute in this data set is gender. Men, whom the biased model associates with a low-risk, represent 69% of instances. The target variable is the loan risk assessment, divided into two classes: a good and a bad customer. The majority class is a good customer, which represents 70% of instances.
Communities and Crime (CC) data set (Redmond & Baveja, 2002) contains data about the rate of violent crime in US communities in 1994. Each instance represents one community. The features are numerical and represent the percentage of the community’s population with a certain property or the average of population in the community. Features include socio-economic (e.g., education, house size, etc.) and demographic (race, age) data. The sensitive attribute is the percentage of the white race. The biased model links instances where whites’ percentage is above average to a low rate of violent crime. The target variable is the rate of violent crime divided into two classes: high and low. Both classes are equally represented in the data set.

heading: 4.2 ROBUSTNESS OF EXPLANATION METHODS
text: To evaluate the robustness of explanation methods with added generators (i.e. gLIME, gSHAP, and gIME), we split the data into training and evaluation set in the ratio 90% : 10%. We used the same training set for the training of adversarial models and explanation methods. We encoded categorical features as one-hot-encoded vectors.
We simulated the adversarial attack for every combination of generators used in explanation methods and adversarial models (except for method IME, where we did not use rbfDataGen, which cannot generate instances in the neighborhood of a given instance). When testing SHAP, we used two variants of the TreeEnsemble generator for explanation: generating new instances around the explained instance and generating new instances according to the whole training set distribution. In the LIME testing, we used only the whole training set variant of TreeEnsemble inside the explanation. In the IME testing, we only used the variant of Tree Ensemble that fills in the hidden values (called TEnsFillIn variant). For training the adversarial models, we used the whole training set based variant of TreeEnsemble, except for IME, where we used the TEnsFillIn variant. These choices reflect the capabilities of different explanation method and attempt to make both defense and attack realistic (as strong as possible). More details on training the decision model d inside the adversarial model can be found in Appendix D.
In all cases, the biased model b (see Section 2.2) was a simple function depending only on the value of the sensitive feature. The unbiased model ψ depended only on the values of unrelated features. The sensitive and unrelated features are shown on the right-hand side of Table 1. Features random1 and random2 were uniformly randomly generated from the {0, 1} set. On COMPAS and CC, we simulated two attacks, one with a single unrelated feature (the result of ψ depends only on the value of the unrelated feature 1), and another one with two unrelated features.
For every instance in the evaluation set, we recorded the most important feature according to the used explanation method (i.e. the feature with the largest absolute value of the contribution as determined by the explanation method). The results are shown as a heatmap in Figure 2. The green color means that the explanation method was deceived in less than 30% of cases (i.e. the sensitive feature was recognized as the most important one in more than 70 % of the cases as the scale on the heatmap suggests), and the red means that it was deceived in more than 70% of cases (the sensitive feature was recognized as the most important one in less than 30 % of the cases as the scale on the heatmap suggests). We consider deception successful if the sensitive feature was not recognized as the most important by the explanation method (the sensitive features are the only relevant features in biased models b).
The gLIME method is more robust with the addition of rbfDataGen and TreeEnsemble than LIME and less robust with the addition of MCD-VAE. This suggests that parameters for MCD-VAE were not well-chosen, and this pattern repeats for other explanation methods. Both TreeEnsemble and rbfDataGen make gLIME considerably more robust on COMPAS and German datasets, but not on the CC dataset. We believe the reason for that is that features in CC are strongly interdependent, and many represent the same attribute as a fraction of value, e.g., we have the share of the white population, the share of the Asian population, and so on. This interdependence dictates that all fractions have to add up to 1. The data generators are unlikely to capture such strong conditional dependencies, but the adversarial model’s decision function is likely to approximate it.
The gSHAP method is most robust when using the TreeEnsemble generator, but it shows less robust behavior than the gLIME method. The reason for that could be in the feature value replacement strategy used by the SHAP and gSHAP methods, which change only some of the feature’s values with the help of the distribution set. This can lead to out-of-distribution instances if the instances in the distribution set are not close enough to the explained instance. With gLIME, we generate complete instances that are more likely to be in the distribution of the training set.
IME is quite robust even with the perturbation sampling, as we expected. This suggests that IME is the most robust of all three tested explanation methods and shall be considered the chosen method in sensitive situations. The gIME results when using the TreeEnsemble generator (TEnsFillIn variant) are comparable to the original IME variant results. This suggests that sampling from a smaller data set, which represents the neighborhood of the explained instance, does not decrease the method’s robustness.

heading: 4.3 COMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHODS
text: We checked if improved data generators affect explanations in a non-adversary environment. We compared the explanations of original and modified methods for four different classification models on three different datasets. As a measure of differences, we use the mean of the absolute difference between the original and enhanced explanations over the whole evaluation set. Details of these experiments are reported in Appendix F, below we shortly summarize the results, contained in Table 4.
The differences between original LIME and gLIME explanations are considerable (see the top part of Table 4). This is not surprising since LIME fits local linear models in its local explanations, which can strongly differ even for small perturbations of the model’s input. The differences in SHAP and IME explanations are small (the average MAD is almost negligible). We can conclude that explanations of gSHAP and gIME are not significantly different from SHAP and IME in the non-adversary environment.

heading: 4.4 BEHAVIOR OF ADVERSARIAL CLASSIFIER WITH DIFFERENT PREDICTION THRESHOLDS
text: The attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method. We express different levels of attacker’s conservatism by changing the decision function threshold d from Equation (1), where it is currently set to 0.5. We tested the robustness of modified explanation methods on the COMPAS dataset with adversarial models that use different certainty thresholds for deploying the biased classifier (simulating different levels of aggressiveness). For results and a more detailed description of the experiment, see Appendix G. Below, we shortly summarize the results.
Even with different thresholds, gIME is still the most robust of the three explanation methods, and treeEnsemble still gives the best results as the data generator. The sensitive feature is recognized as the most important more often when the threshold is lower, which is expected as in this case the adversarial model behaves more aggressively. While the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher values of the thresholds, it remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeEnsemble as data generator). We can conclude that the enhanced explanation methods remain robust enough, even with more conservative adversarial models.

heading: 5 CONCLUSIONS
text: We presented the defense against adversarial attacks on explanation methods. The attacks exploit the shortcomings of perturbation sampling in post-hoc explanation methods. This sampling used in these methods produces instances too different from the original distribution of the training set. This allows unethical owners of biased prediction models to detect which instances are meant for explanation and label them in an unbiased way. We replaced the perturbation sampling with data generators that better capture the distribution of a given data set. This prevents the detection of instances used in explanation and disarms attackers. We have shown that the modified gLIME and gSHAP explanation methods, which use better data generators, are more robust than the original variants, while IME is already quite robust. The difference in explanation values between original and enhanced gSHAP and gIME is negligible, while for gLIME, it is considerable. Our preliminary results in Appendix C show that using the TreeEnsemble generator, the gIME method converges faster and requires from 30-50% fewer samples.
The effectiveness of the proposed defense depends on the choice of the data generator and its parameters. While the TreeEnsemble generator turned out the most effective in our evaluation, in practice, several variants might need to be tested to get a robust explanation method. Inspecting authorities shall be aware of the need for good data generators and make access to training data of sensitive prediction models a legal requirement. Luckily, even a few non-deceived instances would be enough to raise the alarm about unethical models.
This work opens a range of possibilities for further research. The proposed defense and attacks shall be tested on other data sets with a different number and types of features, with missing data, and on different types of problems such as text, images, and graphs. The work on useful generators shall be extended to find time-efficient generators with easy to set parameters and the ability to generate new instances in the vicinity of a given one. Generative adversarial networks (Goodfellow et al., 2014) may be a promising line of research. The TreeEnsemble generator, currently written in pure R, shall be rewritten in a more efficient programming language. The MCD-VAE generator shall be studied to allow automatic selection of reasonable parameters for a given dataset. In SHAP sampling, we could first fix the values of the features we want to keep, and the values of the others would be generated using the TreeEnsemble generator.

heading: A DETAILS ON POST-HOC EXPLANATION METHODS
text: For the sake of completeness, we present further details on the explanation methods LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), and IME (Štrumbelj & Kononenko, 2013). Their complete description can be found in the above-stated references. In our exposition of the explanation methods, we denote with f the predictive model, with x the instance we are explaining, and with n the number of features describing x.
A.1 LIME
Explanations of instances with the LIME method is obtained with an interpretable model g. The model g has to be both locally accurate (so that it can obtain correct feature contributions) and simple (so that it is interpretable).
The explanation of the instance x for the predictive model f , obtained with the LIME method, is defined with the following equation:
ξ(x) = arg min g∈G
(L(f, g, πx) + Ω(g)), (2)
where Ω(g) denotes the measure of complexity for interpretable model, πx(z) denotes the proximity measure between x and generated instances z, L(f, g, πx) denotes the measure of local fidelity of interpretable model g to the prediction model f , and G denotes the set of interpretable models. We use the linear version of the LIME method, where G represents the set of linear models. With x′ we denote the normalized presentation of instance x, i.e. the numerical attributes have their mean set to 0 and variance to 1, and the categorical attributes contain value 1 if they have the same value
as the exoplained instance and 0 otherwise. The proximity function πx is defined on the normalized instances (hence we use the notation πx′ ), and uses the exponential kernel: πx′(z′) = e d(x′,z′) σ2 , where d(x′, z′) denotes a distance measure between x′ and z′. The local fidelity measure L from Equation (2) is defined as:
L(f, g, πx′) = ∑ z′∈Z πx′(z ′)(f(z)− g(z′))2, (3)
whereZ denotes the set of samples. In LIME, each generated sample is obtained by adding Gaussian noise to each feature of x′ independently.
Using linear models as the set of interpretable models, LIME is relatively fast but may produce poor explanations for instances close to complex decision boundaries.
A.2 SHAP
We refer to SHAP as the method called Kernel SHAP by (Lundberg & Lee, 2017). SHAP essentially estimates Shapley values using LIME’s approach, which means that the calculation of explanations is fast due to the use of local linear models computed with the weighted least squares algorithm. The explanation of the instance x with the SHAP method are feature contributions φi, i = 1, 2, ..., n that are coefficients of the linear model g(x′) = φ0 + ∑n i=1 φi · x′i, obtained with LIME, where x′i ∈ {0, 1} for i ∈ {1, 2, ..., n}. As LIME does not compute Shapley values, this property is assured with the proper selection of functions Ω(g), πx′(z′), and L(f, g, πx′). Let us first define the function hx(z′) that maps from the {0, 1}n to the original feature space. The function hx is defined implicitly with the equation f(hx(z
′)) = E[f(z)|zi = xi ∀i ∈ {j; z′j = 1}]. This is the expected value of the model f when we do not know the values of features at indices where z′ equals 0 (these values are hidden). Functions Ω(g), πx′(z ′) and L(f, g, πx′) that enforce the computation of Shapley values are:
Ω(g) = 0,
πx′(z ′) = n( n |z′| ) · |z′| · (n− |z′|) ,
L(f, g, πx′) = ∑ z′∈Z (f(hx(z ′))− g(z′))2 · πx′(z′),
where |z′| denotes the number of nonzero features of z′ and Z ⊆ 2{0,1}n . The main purpose of the sampling in this method is to determine the value f(hx(z′)) because in general predictive models cannot work with hidden values. To determine f(hx(z′)), SHAP uses the distribution set D which we obtain from the training set. For D, SHAP takes the centroids of clusters obtained by the k-means clustering algorithm on the training set. The number of clusters is set by the user. Value of f(hx(z′)) is determined by the following sampling:
f(hx(z ′)) = E[f(z)|zi = xi ∀i ∈ {j; z′j = 1}] =
1 |D| ∑ d∈D f(x[xi=di,z′i=0]), (4)
where x[xi=di,z′i=0] denotes instance x with features that are 0 in z ′ being set to the feature values from d.
A.3 IME
The explanation of x with the method IME are feature contributions φi, i = 1, 2, ..., n. Štrumbelj & Kononenko (2013) shoved that Shapley values are the only solution that takes into account contributions of interactions for every subset of features in a fair way. The i-th feature contribution can be calculated with the following expresiion:
φi(x) = 1
n ∑ π∈Sn ∑ w∈X p(w) · (f(w[wi=xi,i∈Prei(π)∪{i}])− f(w[wi=xi,i∈Prei(π)])), (5)
where Sn denotes a group of permutations of n elements, X denotes the training set instances, Prei(π) represents the set of indices that precedes i in the permutation π, i.e. Prei(π) = {j;π(j) < π(i)}). Let p(w) denote the probability of the instance w in X and let w[formula] denote the instance w with some of its features values changed according to the formula. To calculate φi(x), we have to go through |X | · n! iterations, which can be slow. Therefore, the method IME uses the following sampling. The sampling population of i-th feature is Vπ,w = (f(w[wi=xi,i∈Prei(π)∪{i}]) − f(w[wi=xi,i∈Prei(π)])) for every combination of the permutation π and instance w. IME draws mi samples V1, ..., Vmi at random with repetition. The estimate of φi(x) is defined with the equation:
φ̂i = 1
mi mi∑ i=1 Vi. (6)
Contrary to SHAP, IME does not use approximation with linear models, which compute all features’ contributions at once but has to compute the Shapley values by averaging over a large enough sample for each feature separately. This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency in the space of normalized features.

heading: B DEMONSTRATION OF BETTER SAMPLING IN SHAP
text: The graphs in Figure 3 show the PCA based 2D space of the evaluation part of the COMPAS dataset (see Section 4 for the dataset description). The left-hand side shows the SHAP-generated sampled instances using the k-means algorithm (14 clusters determined by the silhouette score (Rousseeuw (1987)). The sample produced with the MCD-VAE generator in gSHAP is shown on the right-hand side. This sample is much more similar to the original distribution compared to the SHAP sampling.
C IMPROVED IME CONVERGENCE RATE WITH THE TREEENSEMBLE GENERATOR
Preliminary, we tested how better generators affect the convergence rate of the IME explanation method. The preliminary results in Table 2 show a significant reduction in the number of needed samples and a slight increase in the error compared to the original perturbation sampling. Note that the error measure deployed is biased in favor of the perturbation sampling, which was used to determine the gold standard. This was determined with the sampling population’s variance, as described in Štrumbelj & Kononenko (2010).

heading: D TRAINING DISCRIMINATOR FUNCTION OF THE ATTACKER
text: Details of training attacker’s decision models d (see Figure 1b) is described in Algorithms 1, 2, and 3. We used a slightly different algorithm for each of the three explanation methods, LIME, SHAP,

heading: CC dataset
text: and IME, as each method uses a different sampling. Algorithms first create out-of-distribution instances by method-specific sampling. The training sets for decision models are created by labeling the created instances with 0; the instances from sample S (to which the attacker has access) from distribution Xdist are labeled with 1. Finally, the machine learning model dModel is trained on this training set and returned as d. In our experiments, we used random forest classifier as dModel and the training part of each evaluation dataset as S.

heading: E HEATMAPS AS TABLES
text: We present the information contained in Figure 2 in a more detailed tabular form in Table 3.

heading: F COMPARING EXPLANATIONS OF ORIGINAL AND MODIFIED METHODS
text: We check if improved data generators affect explanations in non-adversary environment. We split the dataset into training and evaluation set in the ratio 90% : 10%, and trained four classifiers from Python scikit-learn (Pedregosa et al. (2011)) library: Gaussian naive Bayes, linear SVC (SVM for classification), random forest, and neural network. We explained the predictions of each classifier on the evaluation set with every combination of explanation methods and generators used in the adversarial attack experiments. For instances in the evaluation set, we measured the mean absolute
Algorithm 1: Training of the decision model d, used by the attacker to distinguish between instances from distributionXdist and samples produced by explanation methods LIME or gLIME. Input: S = {(xi)mi=1}: training set, nSamples: number of generated instances for each instance xi ∈ D, gen: data generator, dModel: machine learning algorithm Output: Classifier d that outputs 1 if its input x is from Xdist and 0 otherwise X ← ∅ // Training set for dModel gen.fit(S) // Train the data generator on S for i = 1 to m do
X ← X ∪ (xi, 1) // Add an instance from distribution G← gen.newdata(nSamples, xi) // Generate nSamples new samples around xi for j = 1 to nSamples do // Add nSamples out of distribution instances
X ← X ∪ (G[j], 0) // Add j-th instance from set G to X end
end d← dModel.fit(X) // Fit model dModel to set X and save it in d return d
Algorithm 2: Training of the decision model d, used by attacker to distinguish between instances from distribution Xdist and samples produced by explanation methods SHAP or gSHAP. Input: S = {(xi)mi=1}: training set, nSamples: number of generated instances for each instance
xi ∈ D, k: size of the generated distribution set, gen: data generator, dModel: machine learning algorithm
Output: Classifier d that outputs 1 if its input x is from Xdist and 0 otherwise X ← ∅ // Training set for dModel gen.fit(S) // Train the data generator on S if gen == KMeans or gen == rbfDataGen or gen == treeEnsemble then
D ← gen.newdata(k) // Generate the distribution set with KMeans, rbfDataGen or treeEnsemble end for i = 1 to nSamples do // Add nSamples out of distribution instances
x← random instance from S if gen == MCD − V AE or gen == treeEnsembleF ill then
w ← gen.newdata(1, x) // Generate an instance w in the vicinity of x end else // KMeans, treeEnsemble or rbfDataGen
w ← take a random instance from D end M ← choose a random subset of {1, 2, ..., len(x)} // Choose random features x[M ]← w[M ] // Replace the values of chosen features in x with values from w as in SHAP method X ← X ∪ (x, 0) // Add out of distribution instance
end for i = 1 to m do // Add instances from distribution
X ← X ∪ (xi, 1) end d← dModel.fit(X) // Fit model dModel to set X and save it in d return d
difference (MAD) of modified explanation methods, defined with the following equation:
MADgen(x) = 1
n n∑ i=1 |φgeni (x)− φi(x)|, (7)
where φgeni (x) and φi(x) represent the explanations of i-th feature returned by the modified and original explanation method, respectively (recall that n denotes the number of features in the data set).
We experimented on three datasets. The COMPAS dataset is described in Section 4.1. In addition to that, we used synthetic dataset condInd from Robnik-Šikonja & Kononenko (2008), and Ionosphere
Algorithm 3: Training of the decision model d, used by attacker to distinguish between instances from distribution Xdist and samples produced by explanation methods IME or gIME. Input: S = {(xi)mi=1}: training set, nSamples: number of generated instances for each instance xi ∈ D, gen: data generator, dModel: machine learning algorithm Output: Classifier d that outputs 1 if its input x is from Xdist and 0 otherwise X ← ∅ // Training set for dModel gen.fit(S) // Train the data generator on set S for i = 1 to m do
X ← X ∪ (xi, 1) // Add an instance from distribution for j = 1 to nSamples do // Add nSamples out of distribution instances
w ← gen.newdata(1, xi) // Generate an instance w in the vicinity of xi b1 ← w // First out of distribution instance b2 ← w // Second out of distribution instance π ← choose a random permutation from Slen(xi) // i.e. a random permutation of xi’s
features idx← choose a random number from {1, 2, ..., len(xi)} M1 ← {k ∈ {1, 2, ..., len(xi)}, π(k) < π(idx)} // Features that precede idx in permutation π M2 ←M1 ∪ {idx} b1 ← xi[M1] // Vector b1 as in IME method (Štrumbelj & Kononenko (2013)) b2 ← xi[M2] // Vector b2 as in IME method (Štrumbelj & Kononenko (2013)) X ← X ∪ {(b1, 0), (b2, 0)} // Add out of distribution instances
end end d← dModel.fit(X) // Fit model dModel to set X and save it in d return d
dataset from UCI repository (Dua & Graff (2019)). Both datasets represent a binary classification problem. Apart from the target variable, condInd consists of 8 binary features, while Ionosphere consists of 34 numerical attributes. The condInd datasets contains 2000 instances and Ionosphere contains 351 instances.
The results are shown in Table 4. The differences between original LIME and gLIME explanations are considerable (see the top table). This is not surprising since LIME fits local linear models in its local explanations, which can strongly differ even for small perturbations of the model’s input. SHAP and IME explanations are very similar (the average MAD is almost negligible). We can conclude that explanations of gSHAP and gIME are not significantly different from SHAP and IME in the non-adversary environment.

heading: G ADVERSARIAL CLASSIFIERS WITH DIFFERENT PREDICTION THRESHOLDS
text: In a realistic scenario, the attacker might be wary of being discovered by the regulator and decide to employ deception only when it is really certain that the predicted instance is used inside the explanation method. In these circumstances, the number of times when the sensitive variable is used inside the model will also depend on how aggressively the biased classifier is used. This reflection leads to another metric for the defense’s success, which measures the proportion of instances when the sensitive variable is identified when it is being used, i.e. at different levels of conservatism.
We express different levels of attacker’s conservatism by changing the threshold of the decision function d. The output of the adversarial model et, which uses threshold t is now defined with the equation:
et(x) = { b(x), dprob(x) ≥ t ψ(x), dprob(x) < t , (8)
where dprob(x) denotes the probability that x is from distribution Xdist, returned by decision model d. Threshold t therefore represents the required certainty of the attacker that given instance is from Xdist before it deploys the biased classifier. Larger values of t will result in biased classifier being deployed fewer times but with larger certainty that it has to be used. If we use t = 0.5 in Equation (8), we get the adversarial model that behaves identically to the one, described with Equation (1) in Section 2.2.
In this experiment, we used only the COMPAS dataset, which we split into the training and evaluation set in ratio 90% : 10%. We performed the same experiment as in section 4.2, but with different thresholds for the adversarial models. We used five different thresholds: 0.3, 0.4, 0.5, 0.6 and 0.7.
We first checked how different thresholds affect the deployment of the biased model. For each generator used in the adversarial model’s training and the certainty threshold used by the adversarial model, we measured the proportion of instances from the evaluation set, where the biased classifier was deployed. The results are shown in Table 5. As expected, the percentage of instances where the biased classifier is deployed decreases with increasing t. The biased classifier is deployed less frequently when using treeEnsemble and rbfDataGen in its training, which is also expected, as these two generators give the best results inside the explanation methods. This means that for these generators, it is more difficult for an attacker to distinguish between the actual instances and generated instances. The biased classifier is deployed the most often in the case of the LIME adversarial model and least often in the SHAP adversarial model.
Next, we tested the proportion of instances where the sensitive feature (in case of COMPAS dataset this is race) was recognized as the most important feature by the explanation method on two different subsets of instances: instances from the evaluation set on which the biased classifier was deployed and the whole evaluation set. We measured these numbers for all combinations of thresholds and explanation methods. The results are shown in Tables 6 (gLIME), 7 (gSHAP) and 8 (gIME) for gLIME, gSHAP, and gIME, respectively. We can observe the same pattern as in section 4.2: gIME is still the most robust, and the treeEnsemble generator still gives the best results. The sensitive attribute is recognized as the most important feature more often when the decision threshold values are lower, which is expected as in these cases, the adversarial model behaves more aggressively. Whilst the percentage of the instances on which the racist behavior of the adversarial model is recognized drops with higher thresholds, it remains high enough to warn the regulator about the problematic behavior of the prediction model (especially in the case of gSHAP and gIME using treeEnsemble as data generator). From that, we can conclude that modified explanation methods remain robust enough, even with more conservative adversaries.

Review summaries:
This paper proposes a defense against the adversarial attacks on explanation methods described in Slack et al. (2020). In particular, by using sampling methods that more closely resemble the original data distribution, the authors make it difficult for the Out-of-Distribution detector to successfully discriminate between instances used for predicting and instances used for explaining.