Better sampling in explanation methods can prevent dieselgate-like deception

1 INTRODUCTION . Machine learning models are used in many areas where besides predictive performance , their comprehensibility is also important , e.g. , in healthcare , legal domain , banking , insurance , consultancy , etc . Users in those areas often do not trust a machine learning model if they do not understand why it made a given decision . Some models , such as decision trees , linear regression , and naı̈ve Bayes , are intrinsically easier to understand due to the simple representation used . However , complex models , mostly used in practice due to better accuracy , are incomprehensible and behave like black boxes , e.g. , neural networks , support vector machines , random forests , and boosting . For these models , the area of explainable artificial intelligence ( XAI ) has developed post-hoc explanation methods that are model-independent and determine the importance of each feature for the predicted outcome . Frequently used methods of this type are IME ( Štrumbelj & Kononenko , 2013 ) , LIME ( Ribeiro et al. , 2016 ) , and SHAP ( Lundberg & Lee , 2017 ) . To determine the features ’ importance , these methods use perturbation sampling . Slack et al . ( 2020 ) recently noticed that the data distribution obtained in this way is significantly different from the original distribution of the training data as we illustrate in Figure 1a . They showed that this can be a serious weakness of these methods . The possibility to manipulate the post-hoc explanation methods is a critical problem for the ML community , as the reliability and robustness of explanation methods are essential for their use and public acceptance . These methods are used to interpret otherwise black-box models , help in debugging models , and reveal models ’ biases , thereby establishing trust in their behavior . Non-robust explanation methods that can be manipulated can lead to catastrophic consequences , as explanations do not detect racist , sexist , or otherwise biased models if the model owner wants to hide these biases . This would enable dieselgate-like cheating where owners of sensitive prediction models could hide the socially , morally , or legally unacceptable biases present in their models . As the schema of the attack on explanation methods on Figure 1b shows , owners of prediction models could detect when their models are examined and return unbiased predictions in this case and biased predictions in normal use . This could have serious consequences in areas where predictive models ’ reliability and fairness are essential , e.g. , in healthcare or banking . Such weaknesses can undermine users ’ trust in machine learning models in general and slow down technological progress . In this work , we propose to change the main perturbation-based explanation methods and make them more resistant to manipulation attempts . In our solution , the problematic perturbation-based sampling is replaced with more advanced sampling , which uses modern data generators that better capture the distribution of the training dataset . We test three generators , the RBF network based generator ( Robnik-Šikonja , 2016 ) , random forest-based generator , available in R library semiArtificial ( Robnik-Šikonja , 2019 ) , as well as the generator using variational autoencoders ( Miok et al. , 2019 ) . We show that the modified gLIME and gSHAP methods are much more robust than their original versions . For the IME method , which previously was not analyzed , we show that it is already quite robust . We release the modified explanation methods under the open-source license1 . In this work , we use the term robustness of the explanation method as a notion of resilience against adversarial attacks , i.e . as the ability of an explanation method to recognize the biased classifier in an adversary environment . This type of robustness could be more formally defined as the number of instances where the adversarial model ’ s bias was correctly recognized . We focus on the robustness concerning the attacks described in Slack et al . ( 2020 ) . There are other notions of robustness in explanation methods ; e.g. , ( Alvarez-Melis & Jaakkola , 2018 ) define the robustness of the explanations in the sense that similar inputs should give rise to similar explanations . The remainder of the paper is organized as follows . In Section 2 , we present the necessary background and related work on explanation methods , attacks on them , and data generators . In Section 3 , we propose a defense against the described weaknesses of explanation methods , and in Section 4 , we empirically evaluate the proposed solution . In Section 5 , we draw conclusions and present ideas for further work . 2 BACKGROUND AND RELATED WORK . In this section , we first briefly describe the background on post-hoc explanation methods and attacks on them , followed by data generators and related works on the robustness of explanation methods . 2.1 POST-HOC EXPLANATION METHODS . The current state-of-the-art perturbation-based explanation methods , IME , LIME , and SHAP , explain predictions for individual instances . To form an explanation of a given instance , they measure the difference in prediction between the original instance and its neighboring instances , obtained 1https : //anonymous.4open.science/r/5d550c62-5c5c-4ee3-81ef-ab96fe0838ca/ with perturbation sampling . Using the generated instances , the LIME method builds a local interpretable model , e.g. , a linear model . The SHAP and IME methods determine the impact of the features as Shapley values from the coalitional game theory ( Shapley , 1988 ) . In this way , they assure that the produced explanations obey the four Shapley fairness axioms ( Štrumbelj & Kononenko , 2013 ) . Due to the exponential time complexity of Shapley value calculation , both methods try to approximate them . The three methods are explained in detail in the above references , and a formal overview is presented in Appendix A , while below , we present a brief description . In our exposition of the explanation methods , we denote with f the predictive model and with x the instance we are explaining . Explanations of instances with the LIME method is obtained with an interpretable model g. The model g has to be both locally accurate ( so that it can obtain correct feature contributions ) and simple ( so that it is interpretable ) . Therefore in LIME , g is a linear regression model trained on the instances sampled in the neighborhood of the explained instance x , which is first normalized.The perturbed explanation instances are generated by adding Gaussian noise to each feature of x independently . Using linear models to determine the contributions of features , LIME is fast but may produce poor explanations for instances close to complex decision boundaries . SHAP estimates Shapley values using the approach of LIME ( i.e . local linear models computed with the weighted least squares algorithm ) but using a different neighborhood and loss function for local accuracy , which ensures that Shapley values are obtained . The sampling of explanation instances in SHAP relies on the distribution set D , obtained from the training set , by computing the centroids of k-means clusters . IME computes Shapley values using a pure sampling approach ( no proxy with an interpretable model ) . Its sampling replaces part of the explained instance with existing values of selected features and then averages differences in the prediction of explained instance and samples . This makes the method slower but also potentially more robust as the method does not assume the shape of the dependency . 2.2 ATTACKS ON EXPLANATIONS . The idea of the attack on explanations presented in Slack et al . ( 2020 ) is that the attacker creates an adversarial ( biased ) model whose behavior it wants to hide from explanation methods , e.g. , a racist model that does not grant credits to blacks . As Figure 1b illustrates , a part of the attacker ’ s adversarial model is an unbiased model , which , e.g. , does not take disputed attributes such as race into account . The adversarial model tries to manipulate the explanation method by behaving like an unbiased model on perturbed instances used for the explanation – in this case , it uses the unbiased model ’ s output . On instances from the original distributions that are a part of normal model use , the adversarial model uses the biased model ’ s output . To distinguish between the original and perturbation-sampled data , the adversarial model contains a decision model that selects the right type of model ( biased or unbiased ) based on the predicted type of instances . The reason that the decision function can distinguish between the original data and sampled instances are the shortcomings of explanation methods that use the perturbation sampling ( see Figure 1a ) . For that graph , we created new instances based on the COMPAS dataset ( see Section 4 ) with the Gaussian noise as in the LIME explanation method . We mapped the original and sampled instances to 2D space using the PCA transformation . We can see that the sampled data is distributed differently than the original data . As the difference is revealed with PCA , other machine learning algorithms can also effectively distinguish between sampled and original data . The attack can be formally defined in the following way . The attacker has a biased model b and a sample S from the distribution Xdist . The attacker has no additional information about the distribution Xdist . The attacker ’ s goal is to produce an adversarial model e that behaves like b on the data from Xdist but does not reveal b ’ s bias to the explanation methods . We assume that the attacker also has an unbiased model ψ that hides the f ’ s bias . The attacker creates a decision model d , that should output 1 if the input instance x is from Xdist and 0 otherwise . The model d is trained on S and generated perturbation samples . The attacker creates the adversarial model e with its output defined by the following equation : e ( x ) = { b ( x ) , d ( x ) = 1 ψ ( x ) , d ( x ) = 0 ( 1 ) 2.3 DATA GENERATORS . We used three different data generators based on different algorithms , modeling the distribution of the training set : variational autoencoder with Monte Carlo dropout ( Miok et al. , 2019 ) , RBF network ( Robnik-Šikonja , 2016 ) , and random forest ensemble ( Robnik-Šikonja , 2019 ) . In the remainder of the paper , we refer to the listed generators consecutively as MCD-VAE , rbfDataGen , and TreeEnsemble . Autoencoder ( AE ) consists of two neural networks called encoder and decoder . It aims to compress the input instances by passing them through the encoder and then reconstructing them to the original values with the decoder . Once the AE is trained , it can be used to generate new instances . Variational autoencoder ( Doersch , 2016 ) is a special type of autoencoder , where the vectors z in the latent dimension ( output of the encoder and input of the decoder ) are normally distributed . Encoder is therefore approximating the posterior distribution p ( z|x ) , where we assume p ( z|x ) ∼ N ( µx , Σx ) . The generator proposed by Miok et al . ( 2019 ) uses the Monte Carlo dropout ( Gal & Ghahramani , 2016 ) on the trained decoder . The idea of this generator is to propagate the instance x through the encoder to obtain its latent encoding z . This can be propagated many times through the decoder , obtaining every time a different result due to the Monte Carlo dropout but preserving similarity to the original instance x . The RBF network ( Moody & Darken , 1989 ) uses Gaussian kernels as hidden layer units in a neural network . Once the network ’ s parameters are learned , the rbfDataGen generator ( Robnik-Šikonja , 2016 ) can sample from the normal distributions , defined with obtained Gaussian kernels , to generate new instances . The TreeEnsemble generator ( Robnik-Šikonja , 2019 ) builds a set of random trees ( forest ) that describe the data . When generating new instances , the generator traverses from the root to the leaves of a randomly chosen tree , setting values of features in the decision nodes on the way . When reaching a leaf , it assumes that it has captured the dependencies between features . Therefore , the remaining features can be generated independently according to the observed empirical distribution in this leaf . For each generated instance , all attributes can be generated in one leaf , or another tree can be randomly selected where unassigned feature values are filled in . By selecting different trees , different features are set in the interior nodes and leaves .

Summary: This paper proposes a defense against the adversarial attacks on explanation methods described in Slack et al. (2020). In particular, by using sampling methods that more closely resemble the original data distribution, the authors make it difficult for the Out-of-Distribution detector to successfully discriminate between instances used for predicting and instances used for explaining.