Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods

As machine learning black boxes are increasingly being deployed in domains such as healthcare and criminal justice, there is growing emphasis on building tools and techniques for explaining these black boxes in an interpretable manner. Such explanations are being leveraged by domain experts to diagnose systematic errors and underlying biases of black boxes. In this paper, we demonstrate that post hoc explanations techniques that rely on input perturbations, such as LIME and SHAP, are not reliable. Specifically, we propose a novel scaffolding technique that effectively hides the biases of any given classifier by allowing an adversarial entity to craft an arbitrary desired explanation. Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous. Building Adversarial Classifiers to Fool Explanation Techniques Setting: Assume that there is an adversary with an incentive to deploy a biased classifier f for making a critical decision (e.g., parole, bail, credit) in the real world. The adversary must provide black box access to customers and regulators Regulation (2016), who may use post hoc explanation techniques to better understand f and determine if f is ready to be used in the real world. If customers and regulators detect that f is biased, they are not likely to approve it for deployment. The goal of the adversary is to fool post hoc explanation techniques and hide the underlying biases of f . Output: The output of our framework will be a scaffolded classifier e (referred to as adversarial classifier henceforth) that behaves exactly like f when making predictions on instances sampled from the data distribution, but will not reveal the underlying biases of f when probed with leading post hoc explanation techniques such as LIME and SHAP. Intuition LIME and SHAP (and several other post hoc explanation techniques) explain individual predictions of a given black box model by constructing local interpretable approximations (e.g., linear models). Each such local approximation is designed to capture the behavior of the black box within the neighborhood of a given data point. These neighborhoods constitute synthetic data points generated by perturbing features of individual instances in the input data. However, instances generated using such perturbations could potentially be off-manifold or out-of-distribution (OOD) Mittelstadt et al. (2019). Building Adversarial Classifiers Let f be the biased classifier described earlier. The adversary would like to design a framework such that if and when end users generate explanations of this black box, post hoc techniques can be fooled into thinking that the model is innocuous. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. Figure 1: COMPAS: % of data points for which each feature (color coded) shows up in top 3 (according to LIME and SHAP’s ranking of feature importances) for the biased classifier f (left), our adversarial classifier where ψ uses only one uncorrelated feature to make predictions (middle), and our adversarial classifier where ψ uses two uncorrelated features to make predictions (right). Assuming ψ is a unbiased classifier (e.g., makes predictions based on innocuous features that are uncorrelated with sensitive attributes), the adversarial classifier e takes the following form: e(x) = { f(x), if x ∈ Xdist ψ(x), otherwise In order to create such a classifier, we need to be able to decide whether a given data point x comes from Xdist or not. To build a classifier is_OOD that detects if a given data point is an outof-distribution (OOD) sample (is_OOD(x) = True) or not, we construct a new dataset from the instances in X based on the perturbation strategy of the explanation method the adversarial classifier is attempting to fool. We then train a classifier on this data set. Experimental Results We construct f to be a perfectly discriminatory classifier that uses a sensitive feature i.e., f makes predictions purely based on race using the COMPAS data set Angwin et al. (2016). For LIME, we generate perturbations for every data point in the data set X by adding random noise sampled from N (0, 1) to each feature value. For SHAP, we randomly choose a subset of features for each data point and mark their values as missing by replacing them with their corresponding values from background distribution1. We build the unbiased classifier ψ by constructing synthetic uncorrelated features that have zero correlation with sensitive attributes (e.g., race or gender). We experiment with one or two uncorrelated features. When we only have one uncorrelated feature in a particular experiment, ψ solely uses that to make predictions (if uncorrelated feature = 1, then predict 1, else predict 0). On the other hand, when we have two uncorrelated features in an experiment, we base the predictions on the xor of those two features. In case of LIME, the adversarial classifier e successfully shifts the feature importance in LIME from the true informative feature (the sensitive feature used by f ) to the uncorrelated feature in 100% of held-out points for COMPAS (Figure 1). These results demonstrate that the LIME explanation technique has been effectively fooled by the adversarial classifier e for this data set. For SHAP, when a single uncorrelated feature is used for the attack, the adversarial classifier e successfully shifts the feature importance from the sensitive feature in 84% of data points in COMPAS (Figure 1 bottom and middle). When two uncorrelated features are used in the attack, the adversarial classifier is less successful in removing the bias feature from 1st place in the ranking. See Lundberg and Lee (2017) for more details.