Title: Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine Learning Explainability

Machine Learning Interpretability
Abstract: With the increased deployment of machine learning models in various real-world applications, researchers and practitioners alike have emphasized the need for explanations of model behaviour. To this end, two broad strategies have been outlined in prior literature to explain models. Post hoc explanation methods explain the behaviour of complex black-box models by identifying features critical to model predictions; however, prior work has shown that these explanations may not be faithful, in that they incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture, meaning their explanations are naturally faithful, but they often exhibit poor predictive performance due to their limited expressive power. In this work, we identify a key reason for the lack of faithfulness of feature attributions: the lack of robustness of the underlying black-box models, especially the erasure of unimportant distractor features in the input. To address this issue, we propose Distractor Erasure Tuning (DiET), a method that adapts black-box models to be robust to distractor erasure, thus providing discriminative and faithful attributions. This strategy naturally combines the ease-of-use of post hoc explanations with the faithfulness of inherently interpretable models. We perform extensive experiments on semi-synthetic and real-world datasets, and show that DiET produces models that (1) closely approximate the original black-box models they are intended to explain, and (2) yield explanations that match approximate ground truths available by construction.

Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability

Abstract

With the increased deployment of machine learning models in various real-world applications, researchers and practitioners alike have emphasized the need for explanations of model behaviour. To this end, two broad strategies have been outlined in prior literature to explain models. Post hoc explanation methods explain the behaviour of complex black-box models by identifying features critical to model predictions; however, prior work has shown that these explanations may not be faithful, in that they incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture, meaning their explanations are naturally faithful, but they often exhibit poor predictive performance due to their limited expressive power. In this work, we identify a key reason for the lack of faithfulness of feature attributions: the lack of robustness of the underlying blackbox models, especially to the erasure of unimportant distractor features in the input. To address this issue, we propose Distractor Erasure Tuning ( DiET ), a method that adapts black-box models to be robust to distractor erasure, thus providing discriminative and faithful attributions. This strategy naturally combines the ease of use of post hoc explanations with the faithfulness of inherently interpretable models. We perform extensive experiments on semi-synthetic and real-world datasets, and show that DiET produces models that (1) closely approximate the original black-box models they are intended to explain, and (2) yield explanations that match approximate ground truths available by construction. Our code is made public here.

1 Introduction

An important desideratum of machine learning models is for their predictions to be explainable. This allows both human domain experts as well as laypeople to better understand and trust the decisions made by models, and furthermore, is also a regulatory requirement for high-stakes settings. For example, both the European General Data Protection Regulation (GDPR) [1] and the US AI Bill of Rights [2], require organizations to provide explanations for decisions made in high-stakes settings. A common approach to producing such explanations from black-box models in a post hoc manner is via feature attribution, which aims to identify important input features influencing a model prediction. These methods typically work by locally approximating non-linear models with linear ones [3] under some input perturbations such as feature erasure. Intuitively, if the underlying model is more sensitive

to the erasure of feature A than feature B, these methods aim to attribute a higher 'importance' to feature A than feature B. A fundamental prerequisite for a feature to be considered important by a model is that it must first be useful in predicting the label, that is, it must be discriminative for the task. If a feature does not contain information relating to the output label, then it cannot be used to predict the label, and thus feature attribution methods must not consider them important. However, recent works [4, 5] have found this not to be case - feature attribution methods often highlight non-discriminative features. This motivates a natural question: what causes feature attributions to highlight such non-discriminative features, making them unfaithful?

Black-Box

Model

Natural

Inputs

Blonde

Sensitive to

Distractor

Erasure

Non-

discriminative

attributions

Model Predictions

❌

❌

DiET

Model

Robust to

Distractor

Erasure!

Discriminative

attributions

Task: Hair Color Classification

Task: Hair Color Classification

Distractor

Erased

Inputs

Model Predictions

Answering this question has been hard because of a lack of theoretical understanding of the faithfulness of feature attributions. While these notions have been used empirically [4, 5] to assess the quality of attributions, the theoretical characterization of optimally faithful feature attributions is missing in the literature. In this work, we tackle this problem by proposing a framework for feature attribution methods emphasizing faithfulness and particularly discriminability, formalized via the signal-distractor decomposition for datasets. Essentially, the signal denotes the discriminative parts of the input (relative to a given task), while the distractor denotes the unimportant parts. Feature attribution methods are then evaluated on how well they are able to recover the signal, thus also providing a well-defined notion of a 'ground-truth'. We theoretically identify an important criterion to recover this ground-truth, that being the robustness of the model to the erasure of the input distractors. To enable black-box models to recover such ground truth attributions, we propose Distractor Erasure Tuning ( DiET ), a method that adapts models to be robust to the erasure of input distractors. Given that these distractor regions are not known in advance, our method works by alternating feature attribution and model adaptation. At a high level, our work still uses feature attribution methods while adapting black-box models to have faithful attributions. Thus, this strategy naturally combines the ease of use of post hoc explanation methods with the faithfulness benefits of inherently interpretable models by providing the best of both alternatives. Our contributions are the following:

We present a formalism for feature attribution that emphasizes discriminability and allows for a notion of well-defined ground truth, via the signal-distractor decomposition of a dataset. We show that it is necessary for models to be robust to distractor erasure for them to be able to recover this ground truth.

We propose distractor erasure tuning ( DiET ), a novel method that adapts black-box models to make them robust to distractor erasure.

We perform experiments on semi-synthetic and real-world computer vision datasets and show that DiET outputs models that are faithful to its original input models, and that their feature attributions are interpretable and match ground-truth feature attributions.

2 Related Work

Post-Hoc Explainability. Post-hoc explainability methods aim to explain the outputs of fully trained black-box models either on an instance-level or global level. The most common post-hoc methods are feature attribution methods that rank the relative importance of features, either by explicitly producing perturbations [6, 7], or by computing variations of input gradients [8-10]. Perturbation-based methods are especially popular in computer vision literature [11-14], which use feature removal strategies adapted specifically for image data. However, these methods all assume a specific form for feature removal, and we show theoretically in Section 3 that this can lead to unverifiable attributions.

Inherently Interpretable Models. Inherently interpretable models are constructed such that we know exactly what they do, either through their weights or explicit modular reasoning. As such, the explanations provided by these models are more accurate than those given by post-hoc methods; however, the performance of interpretable models often suffers when compared to unconstrained black-box architectures. The most common inherently interpretable model classes include linear models, decision trees and rules with limited depth, GLMs, GAMs [15], JAMs [16-18], prototypeand concept-based models [19, 20], and weight-input aligned models [21]. While [19, 20] leverage the expressivity of deep networks, they constrain hypothesis classes significantly and still often suffer from a decrease in performance. Among these, our work most closely relates to JAMs, which amortise feature attribution generation using a learnt masking function to generate attributions in a single forward pass, and trains black-box models using input dropout. On other hand, JAMs (1) trains models from scratch, whereas DiET can interpret black-box models, (2) amortises feature attributions using a masking function resulting in less accurate attributions, (3) trains models to be robust to a large set of candidate masks via input dropout, leading to low predictive accuracy, whereas DiET trains models only to be robust to the optimal mask, leading to more flexibility and higher predictive accuracy.

Evaluating Correctness of Explanations. As explainability methods grow in number, so does the need for rigorous evaluation of each method. Research has shown that humans naively trust explanations regardless of their 'correctness' [22], especially when explanations confirm biases or look visually appealing. Common approaches to evaluate explanation correctness rely on feature / pixel perturbation [23, 9, 24], i.e., an explanation is correct if perturbing unimportant feature results in no change of model outputs, whereas perturbing important features results in large model output change. Hooker et al. [5] proposed remove-and-retrain (ROAR) for evaluating feature attribution methods by training surrogate models on subsets of features denoted un/important by an attribution method and found that most gradient-based methods are no better than random. While prior works focused on developing metrics to evaluate correctness of explanations, our method DiET produces models that have explanations that are accurate by design, according to pixel-perturbation methods.

3 Theory of Discriminative Feature Attributions

In this section, we provide a theoretical framework for feature attribution, including a well-defined ground truth. We start by identifying a common feature of feature attributions, their reliance on perturbations or erasure. Intuitively, feature attribution methods work by simulating removing certain features and estimating how the model behaves when those features are removed: removing unimportant features should not change model behaviour. Typically, this erasure is implemented by replacing features with scalar values, such as the mean of the dataset [11, 12]. However, this can result in out-of-distribution inputs that can confuse a classifier, thus making it difficult to create meaningful attributions.

To ground this argument in an example, consider a model that classifies cows and camels. For an image of a camel, a feature attribution might note that only the hump of the camel and the sand it stands on are important for classification. As such, we would expect that the sky was irrelevant to the classifier's prediction, and we can concretely test this by altering it and creating a counterfactual sample. For example, we could mask the sky with an arbitrary uniform color; however, this may result in the sample being out-of-distribution for the model, and its prediction may change drastically even if the sky was not important for prediction. There are two strategies to overcome this problem. The first solution involves masking the sky in a manner that preserves the naturalness of the image,

but this solution involves using large-scale generative models, which themselves can contain biases and be uninterpretable. The second solution requires the classifier to be invariant to the erasure of the pixels corresponding to the sky, which is our solution in this paper. We formalize this argument below by defining an erasure-based feature attribution method called ( ϵ, Q ) -feature attribution.

Notation. Throughout this paper, we shall assume the task of classification with inputs x ∼ X with x ∈ R d and y ∈ [1 , 2 , ...C ] with C -classes. We consider the class of deep neural networks f : R d →△ C which map inputs x onto a C -class probability simplex. This paper considers binary feature attributions, which are represented as binary masks m ∈ { 0 , 1 } d , where m i = 1 indicates an important feature and m i = 0 indicates an unimportant feature.

3.1 Feature Attributions with Input Erasure

We first define an erasure-based feature attribution such that the feature replacement method is explicit, and features are replaced with samples from a counterfactual distribution Q . Particularly, we are interested in binary attributions (i.e., a feature is considered important or not) instead of real-valued ones.

Definition 1. ( ϵ, Q )-feature attribution (QFA) is a binary mask m ( f, x , Q ) that relies on a model f ( · ) , an instance x , and a d -dimensional counterfactual distribution Q , and is given by

Thus, an ( ϵ, Q )-feature attribution (henceforth, QFA ) refers to the sparsest mask that can be applied to an image such that the model's output remains approximately unchanged. QFA depends on the feature replacement distribution Q , where Q is independent of both x and y . This generalizes the commonly used heuristics of replacing unimportant features with the dataset mean, in which case Q is a Dirac delta distribution at the mean value. The choice of Q is indeed critical, as an incorrect choice can hurt our ability to recover the correct attributions due to the resulting inputs being out-of-distribution and the classifier being sensitive to such changes. Specifically, an incorrect Q can result in QFA being less sparse, as masking even a few features with the wrong Q would likely cause large deviations in the model's outputs. As a result, given a model, we must aim to find the Q that leads to the sparsest QFA masks. However, the problem of searching over Q is complex, as it requires searching over the space of all d -dimensional distributions, and furthermore, if the underlying model is non-robust, there may not exist any Q that leads to sparse attributions. To avoid this, we consider the inverse problem: given Q , we find the class of models that have the sparsest QFAs w.r.t. that particular Q . We call this the Q -robust model class, which we define below:

Definition 2. Q -robust model class F v ( Q ) : For some given distribution Q , the class of models F v for which Q has the sparsest QFA mask as opposed to any other Q ′ , such that for all f ∈ F v ( Q ) ,

Intuitively, Q -robust models result in the sparsest QFA masks and can be thought of as being robust to the erasure of "irrelevant" input features. Recalling our example of the cows and camels, we would like models to be robust to the replacement of the pixels corresponding to the sky but not necessarily robust to pixels corresponding to the camel or the cow itself. This distinguishes it from classical robustness definitions, which require models to be robust to small perturbations (rather than erasure) at every feature uniformly. Thus Q -robustness is equivalent to enforcing robustness to the erasure of distractor features, a notion that is central to this work. For the rest of this paper, we shall refer to QFA applied to a model from a Q -robust model class as a "matched" feature attribution - the same Q is used to both define the model class and the feature attribution.

3.2 Recovering the Signal-Distractor Decomposition

In the study of feature attribution, the 'ground truth' attributions are often unspecified. Here, we show that for datasets that are signal-distractor decomposable, formally defined below, there exists a ground truth attribution, and feature attributions for optimal verifiable models are able to recover it.

Intuitively, given an object classification dataset between cows and camels, the "signal" refers to the regions in the image that are discriminative, or correlated with the label, such as the cows or camels. The distractor refers to everything else, such as the background or sky. Note that if objects in the background are spuriously correlated with the label, i.e. sand or grass, those would be part of the signal, not the distractor. We first begin by formally defining the signal-distractor decomposition.

Definition 3. A labelled dataset D = { ( x , y ) N i =1 } has a signal-distractor decomposition defined by masks m ( x ) ∈ { 0 , 1 } d for every input x , where

x ⊙ m ( x ) is the discriminative signal, where p ( y | x ) = p ( y | x ⊙ m ( x ))

x ⊙ (1 -m ( x )) is the non-discriminative distractor, where p ( y | x ⊙ (1 -m ( x ))) = p ( y )

m ( x ) is the sparsest mask, i.e., m ( x ) = arg min m ′ ( x ) ∥ m ′ ( x ) ∥ 0 , such that (1) and (2) are satisfied.

We propose to use the masks m ( x ) implied by the signal-distractor decomposition as ground truth feature attributions. These are meaningful as they precisely highlight the discriminative components of the image and ignore the non-discriminative regions. Discriminability has previously been considered an important criterion to evaluate feature attributions [4, 5] however, we here take a step further and propose its usage as ground truth.

We observe first that the masks m ( x ) of the signal-distractor decomposition always exist: setting m ( x ) as an all-ones vector trivially satisfies conditions (1) and (2). When multiple masks exist, condition (3) requires us to choose the sparsest such mask m ( x ) . Using the definitions provided, we show below an asymptotic argument stating that Q -robustness is a necessary condition to recover the optimal masks defined by the signal-distractor decomposition.

Remark : A dataset D is said to have a 'non-redundant signal' when the sparsest mask in condition (3) of the signal-distractor decomposition is equal to the sparsest mask when (1) alone is satisfied.

Theorem 1. QFA applied to Q -robust models recover the ground-truth masks when applied to the Bayes optimal predictor f ∗ v ∈ F v ( Q ) , for datasets D with a non-redundant signal.

Proof Idea. Wefirst note that the optimal Q for QFA is equal to the ground truth distractor distribution, as this leads to the sparsest QFA. If a Q -robust model aims to recover the sparsest masks, then its QFA mask must equal that obtained by setting Q equal to the distractor. From the uniqueness argument in the definition of the signal-distractor decomposition, this is possible only when the optimal mask is recovered by QFA.

Corollary. QFA fails to recover the ground-truth masks when applied to predictors f ̸∈ F v ( Q ) .

This follows from the fact that for any f ̸∈ F v ( Q ) , there exists some other Q ′ that results in a sparser mask, indicating that the ground truth masks are not recovered. Thus, this shows that feature attributions applied to the incorrect model class can be less effective - in this case, they fail to recover the ground truth masks. Further, the Bayes optimality is an important condition because it ensures that the resulting model is sensitive to all discriminative features in the input - sub-optimal models are sub-optimal precisely because they fail to capture the signal from all the discriminative components of the input, and this can interfere with such models being able to recover ground truth masks. In practice, if we expect our models to be highly performant, we can expect them to be sensitive to all discriminative parts of the input and thus approximately recover ground truth masks. Finally, in practice, we do not have access to the ground truth masks for natural datasets, as the discriminative and the non-discriminative regions are not known in advance. In order to use these notions of ground truth, it is thus vital to construct semi-synthetic datasets where the discriminative parts are known. Thus, one can use semi-synthetic datasets to validate a feature attribution approach and then apply it to gain insight into real datasets with unknown signal and distractor components.

To summarize, we have defined a feature attribution method with the feature removal process made explicit via the counterfactual distribution Q . To minimize the sparsity of the attribution masks with a given Q , we use models from the Q -robust model class F v ( Q ) . Finally, we find that feature attributions derived from Bayes optimal models in the model class F v ( Q ) are able to recover the ground-truth masks and fail to do so otherwise.

4 DiET : Distractor Erasure Tuning

In the previous section we showed that given a Q -robust model f v ∈ F v ( Q ) , we are able to apply QFA to recover the ground truth masks. In this section, we shall discuss how to practically build such robust models, given a pre-defined counterfactual distribution Q that defines the erasure method.

Relaxing QFA. We note that QFA as defined in definition 1 is difficult to optimize in its current form due to its use of ℓ 0 regularization and its constrained form. To alleviate this problem, we perform two relaxations: first, we relax the ℓ 0 objective into an ℓ 1 objective, and second, we convert the constrained objective to an unconstrained one by using a Lagrangian. The resulting objective function is given in equation 1. Assuming the model f v is known to us, we can minimize this objective function to obtain ( ϵ, Q )-feature attributions for each point x ∈ X .

Enforcing Model Robustness via Distillation. Assuming that the optimal masks denoting the signal-distractor decomposition are known w.r.t. every training data point (i.e., { m ( x ) } x ∈X ), one can project any black-box model into a Q -robust model via distillation. Specifically, we can use equation 2 for this purpose, which contains (1) a data distillation term to enforce the ϵ constraint in QFA, and (2) a model distillation term to enforce that the resulting model and original model are approximately equal. Accordingly, the black-box model f b and our resulting model f v both have the same model architecture, and we initialize f v = f b .

Alternating Minimization between θ and m . We are interested in both of the above objectives: we would like to recover the optimal masks from the dataset, as well as use those masks to enforce ( ϵ, Q ) constraints via distillation to yield our Q -robust models. We can thus formulate the overall optimization problem as the sum of these terms, as shown in equation 3. Notice that both these objectives assume that either the optimal masks, or the robust model is known, and in practice, we know neither. A common strategy in cases that involve optimizing over multiple sets of variables is to employ alternating minimization [25], which involves repeatedly fixing one of the variables and optimizing the other. We handle the constrained objective on the mask variables via projection, i.e., using hard-thresholding / rounding to yield binary masks.

Iterative Mask Rounding with Increasing Sparsity. In practice, mask rounding makes gradientbased optimization unstable due to sudden jumps in the variables induced by rounding. This problem commonly arises when dealing with sparsity constraints. To alleviate this problem, use a heuristic that is common in the model pruning literature [26] called iterative pruning, which involves introducing a rounding schedule, where the sparsity of the mask is gradually increased during optimization steps. Inspired by this choice, we employ a similar strategy over our mask variables.

Practical Details. We implement these objectives as follows. First, we initialize the robust model to be the same as the original model, f v = f b , and the mask to be all ones, D s = m ⊙ D d , m = 1 . We then iteratively (1) simplify D s by optimizing L QFA until m converges, (2) round m such that it is binary (i.e. D s is a subset of features in D d rather than a weighting of them), and (3) update f v by minimizing L train such that D s is equally as informative as D d to f v and f v is functionally equivalent

to f b . As per Definition 2, we replace masked pixels in D s with a pre-determined counterfactual distribution Q . This ensures that the given Q is the optimal counterfactual distribution for f v , meaning f v comes from the Q -robust model class F v ( Q ) . The pseudocode is given in Algorithm 1.

Algorithm 1 Distractor Erasure Tuning

Mask Scale. In order to encourage greater 'human interpretability,' we increase the pixel size of the masks by lowering their resolution. We do this by downscaling the masks before optimization. Concretely, we initialize the masks to be of size m d = x d /u , where x d is the dimension of the image x and u is the pixel size we wish to consider. We then upsample the mask to be of dimension x d before applying it to x . The more we downscale the mask by (i.e. the greater u is), the more interpretable and visually cohesive the distilled dataset x s is.

5 Experimental Evaluation

In this section, we present our empirical evaluation in detail. We consider various quantitative and qualitative metrics to evaluate the correctness of feature attributions given by DiET models as well as the faithfulness of DiET models to the models they are meant to explain. We also evaluate DiET models' ability to explain models manipulated to have arbitrary uninformative input gradients. Finally, we analyze the effect of the mask downscaling hyperparameter on attributions. Comparisons to additional baselines beyond those shown in Figure 2 are given in 8.

Datasets. Hard MNIST: The first is a harder variant of MNIST where the digit is randomly placed on a small subpatch of a colored background. Each sample also contains small distractor digits and random noise patches. For this dataset, we consider the signal to be all pixels contained within the large actual digit, and the distractor to be all pixels in the background, noise, and smaller digits. Chest X-ray: Second, we consider a semi-synthetic chest x-ray dataset for pneumonia classification [27]. To control exactly what information the model leverages such that we can create ground truth signal-distractor decompositions, we inject a spurious correlation into this dataset. We randomly place a small, barely perceptible noise patch on each image in the 'normal" class. We confirm that the model relies only on the spurious signal during classification by testing the model's drop in performance when flipping the correlation (adding the noise patches to the 'pneumonia' class) and seeing that the accuracy goes from 100% to 0%. As such, for this dataset, the signal is simply the noise patch and the distractor is the rest of the xray. CelebA: The last dataset is a subset of CelebA [28] for hair color classification with classes {dark hair, blonde hair, gray hair}. We correlate the dark hair class with glasses to allow for qualitative evaluation of each methods' ability to recover spurious correlations. This dataset does not have a ground truth signal distractor decomposition, as there are many unknown discriminative spurious correlations the model may rely upon.

Models. For all experiments, we use ImageNet pre-trained ResNet18s for our baseline models, f b . All models achieve over 96% test accuracy. We train DiET models with Q ∼ ✶ d ∗ d ∗ N ( µ ( D d ) , σ 2 ( D d )) , meaning that each image is masked with a uniform color drawn from a normal distribution around the mean color of the dataset. For all evaluation, we use Q ∼ ✶ d ∗ d ∗N ( µ ( D d ) , 0)

Raw Image

Feature

Attributions

DiET (Ours)

FullGrad

GradCAM

SmoothGrad

Gradients

glasses

hair

hair, old

age

h

D

G

(the dirac delta of the dataset mean) to ensure that masked samples are minimally out-of-distribution for the baseline models f b .

5.1 Evaluating the Correctness of Feature Attributions

Pixel Perturbation Tests. We test the faithfulness of our explanations via the pixel perturbation variant proposed in [9, 5], where we mask the k least salient pixels as determined by any given attribution method and check for degradation in model performance with the mean of the dataset. This metric evaluates whether the k masked pixels were necessary for the model to make its prediction. As mentioned in previous works, masked samples come from a different distribution than the original samples, meaning poor performance after pixel perturbation can either be a product of the model's performance on the masks or the feature attribution scores being incorrect. To disentangle the correctness of the attributions from the robustness of the model, we perform pixel perturbation tests on ground truth feature attributions, with results reported in the appendix. Note that our method returns binary masks, but this metric requires continuous valued attributions to create rankings. As such, for this experiment we use the attributions created by our method before rounding .

Results are shown in Figure 3. We find that the attributions produced by DiET , used in conjunction with DiET models outperform all baselines. Furthermore, DiET attributions tested on the baseline model also generally perform better than gradient-based attributions.

Intersection Over Union. We further evaluate the correctness of our attributions by measuring their Intersection Over Union (IOU) with the 'ground truth' attributions. We use the signal from the ground truth signal-distractor decomposition as described in 5 for the ground truth attributions. For each image, if the ground truth attribution is comprised of n pixels, we take the intersection over union of the top n pixels returned by the explanation method and the n ground truth pixels, meaning an IOU of 1 corresponds to a perfectly aligned/correct attribution. Results are shown in 1, where our method performs the best for both datasets. We report mean and standard deviation over the dataset for each method.

5.2 Evaluating the Faithfulness of DiET Models

To ensure that the DiET model ( f v ) returned by our method is faithful the the original model ( f b ) it approximates, we test the accuracy of DiET models with respect to the predictions produced by the original model. Specifically, we take the predictions of the original model f b to be the labels of the dataset. We evaluate DiET models on both the original dataset ( f v ( D d ) ≈ f b ( D d ) ) and the simplified dataset ( f v ( D s ) ≈ f b ( D d ) ), with results shown in 2. We see that DiET models are highly faithful to the baseline models they approximate.

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.90

0.92

0.94

0.96

0.98

0.2

0.4

0.6

0.8

1.0

MNIST

0.990

0.992

0.994

0.996

0.998

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Pneumonia Chest X-ray

0.5

0.6

0.7

0.8

0.9

1.0

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

CelebA

CelebA (Manipulated Gradients)

DiE

T

DiE

T Attributions,

Original Model

SmoothGrad

GradCAM

Grad

FullGrad

Learnt Mask Sparsity

Original Model

5.3 Qualitative Analysis of Simplified Datasets

We first explore how well DiET recovers signal-distractor decompositions. For CelebA, we see that all methods except for input gradients correctly recover the spurious correlation for the 'dark hair/glasses' class, however only our method provides useful insights into the other two classes. We see that our method correctly identifies hair as the signal for the 'blonde hair' class, whereas other methods simply look at the eyes, which are not discriminative. Furthermore, we see that for the 'gray hair' class, our method picks up on hair, as well as initially unknown spurious correlations such as wrinkles and bowties. For Hard MNIST, we see that our method clearly isolates the signal and ignores the distractor. FullGrad and GradCAM suffer from a locality bias and tend to highlight the center of each digit. SmoothGrad and vanilla gradients are much noisier and highlight edges and many random background pixels. For the Chest X-ray dataset, we see that our method and FullGrad perfectly highlight the spurious signal. GradCAM again suffers from a centrality bias, and cannot highlight pixels on the edge. SmoothGrad and gradients appear mostly random to the human eye.

We also consider the visual quality of our attributions compared with the baselines (examples are shown in 2). We find that our method, FullGrad, and GradCAM appear the most interpretable, as opposed to SmoothGrad and vanilla gradients, because they consider features at the superpixel level rather than individual pixels. We also see that GradCam and FullGrad seem relatively class invariant, and tend to focus on the center of most images, rather than the discriminative features for each class, providing for less useful insights into the models and datasets.

5.4 Robustness to Adversarial Manipulation of Explanations

In this section, we highlight our method's robustness to adversarial explanation manipulation. To this end, we follow the manipulation proposed in [29], which adversarially manipulates gradient-based explanations. This is achieved by adding an extra term to the training objective that encourages input gradients to equal an arbitrary uninformative mask of pixels in the top left corner of each image. Note that model accuracy on the classification task is the same as training with only cross-entropy loss.

We repeat experiments for all evaluation metrics on these manipulated models, with pixel-perturbation shown below 6, and IOU, model faithfulness, and model robustness in the appendix. We see that gradient-based methods perform significantly worse on manipulated models; however, our method

remains relatively invariant. We also note while the models are only manipulated to have arbitrary input gradients, SmoothGrad and GradCAM are also heavily affected such that their attributions are entirely uninterpretable as well, as shown below.

0.90

0.92

0.94

0.96

0.98

0.2

0.4

0.6

0.8

1.0

MNIST (Manipulated Gradients)

Ours - Our Model

Ours - Original Model

SmoothGrad

GradCAM

Grad

FullGrad

Learnt Mask Sparsity

Raw Image

Explanation

Ours

FullGrad

GradCAM

SmoothGrad

Gradients

4

4

5.5 Attribution Sensitivity to Hyperparamaters

We conduct an ablation study on the choice of how much to downscale the mask by. The less we downscale by, the more fine-grained the mask is, allowing for optimization over a larger set of masks. However, the more we downscale by, the visually cohesive or 'interpretable' to humans the masks are. We evaluate the trade-off between these two via pixel perturbation tests over multiple downscaling factors and with qualitative evaluations of the final masks in 5. We see that a downscaling factor of 8 performs the best on pixel perturbation tests. Increased factors of downscaling impose a greater locality constraint that results in informative pixels being masked, as shown in the visualization.

0.90

0.92

0.94

0.96

0.98

0.2

0.4

0.6

0.8

1.0

MNIST

Upsample Factor

8

16

32

56

0.90

0.92

0.94

0.96

0.98

0.5

0.6

0.7

0.8

0.9

CelebA

Upsample Factor

8

16

32

56

Ups = 8

Ups = 16

Ups = 32

Ups = 56

4

6 Discussion

In this paper, we seek to build machine learning models such that their feature attributions remain discriminative. In particular, we propose DiET , a method that adapts black-box models into those that are robust to distractor erasure. We empirically evaluate DiET and find that the resulting models are highly faithful to the original and produce interpretable attributions that closely match the ground truth ones. Limitations of DiET include requiring full access to the training dataset and the baseline model. Furthermore, while it produces verifiable feature attributions that tell us how important each feature is to the model's prediction, it does not tell us what the relationship between important features and the output/label is, as is true with all feature attributions.

Acknowledgments and Disclosure of Funding

This work is supported in part by the NSF awards IIS-2008461, IIS-2040989, IIS-2238714, Kempner Institute Graduate Fellowship, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design (D 3 ) Institute at Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.

References

The European Commission. General data protection regulation (gdpr). 2018. URL https://commission.europa.eu/law/law-topic/data-protection/reform/ rules-business-and-organisations/principles-gdpr_en .

The White House. Blueprint for an ai bill of rights. 2022. URL https://www.whitehouse. gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf .

Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. arXiv preprint arXiv:2206.01254 , 2022.

Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems , 34:2046-2059, 2021.

Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems , 32, 2019.

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining , pages 1135-1144, 2016.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 4765-4774. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/ 7062-a-unified-approach-to-interpreting-model-predictions.pdf .

Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 , 2017.

Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. Advances in neural information processing systems , 32, 2019.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618-626, 2017.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13 , pages 818-833. Springer, 2014.

Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision , pages 3429-3437, 2017.

Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2950-2958, 2019.

Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems , 30, 2017.

Trevor J Hastie. Generalized additive models. In Statistical models in S , pages 249-307. Routledge, 2017.

Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning , pages 883-892. PMLR, 2018.

Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In International Conference on Learning Representations , 2019.

Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics , pages 1459-1467. PMLR, 2021.

Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems , 32, 2019.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment is all we need for interpretability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10329-10338, 2022.

Himabindu Lakkaraju and Osbert Bastani. " how do i fool you?" manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pages 79-85, 2020.

Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and KlausRobert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems , 28(11):2660-2673, 2016.

Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. Advances in Neural Information Processing Systems , 35:15784-15799, 2022.

Prateek Jain and Purushottam Kar. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning , 10, 2017. URL all_papers/JainK17_FTML.pdf .

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems , 28, 2015.

Daniel Kermany, Kang Zhang, Michael Goldbaum, et al. Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley data , 2(2):651, 2018.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Information Processing Systems , 32, 2019.

Supplementary Material

7 Proofs

Theorem 2. Q -feature attributions applied to Q -robust models ( F v ( Q ) ) recover the signal-distractor decomposition for Bayes-optimal predictor f ∗ v ∈ F v ( Q ) .

Proof. Consider QFA with ϵ = 0 . Let f ∗ be the Bayes-optimal model, which implies that f ∗ y ( x ) = p bayes ( y | x ) = p ( y | x ) = p ( x | y ) ∑ i p ( x | y = i ) , i.e., the model estimates the correct conditional probabilities from the data given the class-conditional generative probabilities. Note that this is only defined for inputs x ∈ X in the support of the data distribution and not outside, i.e., p bayes ( y | x ) = p ( y | x ) only when x ∈ X .

Let us define x ⊙ (1 -m ) = x distractor ∼ X distractor . This is the distribution of distractor images, which (recall) are independent of the label y . Using this, we consider an idealized version of QFA , with Q ideal = X distractor . From Definition 1, this results in generation of simplified inputs x s = x ⊙ m + q ⊙ (1 -m ) . For q ∼ Q ideal = X distractor, we observe that x s ∈ X , the data distribution. Recall the Bayes optimal classifier is defined across the data distribution X , and applying QFA results in the sparsest m such that p ( y | x ) = p bayes ( y | x ) = p bayes ( y | x s ) = p ( y | x ⊙ m + q ⊙ (1 -m )) = p ( y | x ⊙ m ) . The last equality holds because q ⊙ (1 -m ) is independent of y . This corresponds to the definition of the signal-distractor, and thus it implies that QFA with Q ideal recovers the mask defined by the signal-distractor decomposition .

For any other value of Q ̸ = Q ideal , we first consider the Q -robust Bayes optimal predictor p Q bayes ( y | x ) . This has the property that p Q bayes ( y | x ) = p bayes ( y | x ) for x ∈ X . We now compare mask m Q derived from applying QFA on p Q bayes and mask m Q ideal from applying QFA with Q ideal on p Q bayes . From the previous paragraph, we know that m Q ideal = m ideal is the ideal sparsest mask. However from Definition 2, m Q is the sparsest mask. Thus it is the case that m Q = m ideal , proving our overall result.

We now present proof for an additional statement not described in the main text, where we connect QFA to other commonly used feature attributions via the local function approximation framework [3] as follows.

Theorem 3. QFA is an instance of the local function approximation framework (LFA), with (1) random binary perturbations, and (2) an interpretable model class consisting of linear models with binary weights

Proof. Assume a black box model given by f b ( x ; m ) = ✶ ( E q ∥ f ( x s ( m , q )) -f ( x ) ∥ 2 ≤ ϵ ) ,loss function ℓ ( f, g, x, ξ ) = ( f ( x ; ξ ) -g ( ξ )) 2 , neighborhood perturbation Z = Uniform (0 , 1) d , and an interpretable model family G being the class of binary linear models.

For these choices, it is easy to see that

This above objective is minimized when g = m ∗ , i.e., the ideal ϵ Q -FA mask, because this sets the first term to be zero by definition, and the second sparsity term ensures the minimality of the mask.

8 Additional Results

Model Verifiability. We further test the verifiability of our models by evaluating how the model's performance changes when performing the pixel perturbation test on groundtruth attributions. This enables us to disentangle the verifiability of the model from the correctness of the attributions, as we know that our attributions are correct. We use the same groundtruth attributions as in 5.1. We report the ℓ 1 norm between predictions made on the original samples and predictions made on the masked samples. We compare our verifiable models to the baseline models they approximate, as well as models trained with input dropout, which [18] proposes as their verifiable model class. Training with input dropout is equivalent to training f v with random masks and cross-entropy loss rather than optimized masks and f b prediction matching. Results are shown in 3, where we see that our model performs similarly for masked and normal samples, whereas the other models do not.

Table 3: Model Verifiability

Robustness to Explanation Attacks. We report additional results on Chest X-ray and CelebA for the pixel perturbation tests, IOU tests, and model faithfulness tests for baseline models trained with manipulated gradients, as outlined in 5.4. We see that DiET models are still highly faithful and produce correct explanations even when derived from models adversarially trained to have manipulated explanations.

Signal vs Distractor Masking. For our CelebA experiments, there may have been unintended spurious correlations that we did not foresee and that our method did not recover, leading to a vacuous decomposition. To further support the results in Figure 3, we also perform experiments on masking the signal instead of the distractor, by masking out 'important' pixels as opposed to the unimportant ones. The results for this experiment are shown in 6.

0.990

0.992

0.994

0.996

0.998

Percent Masked

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

Pneumonia Chest X-ray (Manipulated Gradients)

Ours - Our Model

Ours - Original Model

SmoothGrad

GradCAM

Grad

FullGrad

Learnt Mask Sparsity

0.800

0.825

0.850

0.875

0.900

0.925

0.950

0.975

Percent Masked

0.4

0.5

0.6

0.7

0.8

0.9

1.0

CelebA (Manipulated Gradients)

Ours - Our Model

Ours - Original Model

SmoothGrad

GradCAM

Grad

FullGrad

Learnt Mask Sparsity

These results indicate that for the DiET models, there do not exist pixels outside the shown signal regions in Figure 2 that contain information about the label. These results show that the signaldistractor decomposition for CelebA is very likely non-vacuous.

Finetuning on less data. We recognize that a limitation of this work is that it requires further training of the given models on their training datasets. To address this issue, we briefly explore whether DiET can be finetuned on a small subset of the data distribution (without labels) as follows for MNIST. The original model is trained on 8835 samples from the train split. DiET is finetuned on 1500 samples from a separate unlabeled validation split. We perform pixel perturbation on the remaining 8500 test samples. We compare this to DiET models finetuned on the full train set and note a minimal change in performance in 7.

Overall, this shows that there exist ways to decrease the computational complexity of our procedure when applied to larger datasets, which we haven't fully investigated yet.

Effect on standard robustness. We conduct additional robustness experiments for Gaussian and Bernoulli noise with varying standard deviations and probabilities. Results are shown in 8 and 9. We find that DiET models are generally more robust than regular models. Note that neither DiET models nor the original models are explicitly trained or tuned for robustness on these distributions.

Additional Baselines: SHAP and BCosNets. We consider two additional explanation baselines: SHAP [7] and BCosNets [21]. Results for pixel perturbation tests and visualizations are shown in 7. Results for IOU tests are shown in 10.

We note that in general SHAP does not often perform as well as most gradient-based methods for image data. We find that it performs similarly to random attributions.

We add B-CosNets as an inherently-interpretable model baseline. We find that B-CosNets performs comparably to our method for the IOU test on Hard MNIST. We were unable to train it to convergence on the Chest X-ray dataset. We also find that visualizations created by B-CosNets align with our expectations and appear visually interpretable. However, our method still significantly outperforms B-CosNets on the pixel perturbation test, showing that B-CosNets are not robust to perturbations of the distractor (i.e. are not verifiable). We also note that our method can be applied to a trained black-box model of any architecture and training procedure, whereas B-CosNets, like all inherently interpretable models, cannot.

Effect of Choice of Q. We perform an ablation to test the effect that the choice of Q has empirically. We consider various parameterizations of the normal distribution used for Q, and find that results are relatively consistent across the different choices of Q, as shown in 8.

9 Additional Implementation and Computation Details

Models were trained on the original train/test split given by https://github.com/jayaneetha/ colorized-MNIST for Hard MNIST and [27] for the Chest X-ray dataset and with a random 80/20 split for CelebA. Baseline models were trained with Adam for 10 epochs with learning rate 1 e -4 and batch size 256. All hyperparameters are included in the code for this paper. The model distillation and data distillation terms are weighted with λ 1 = λ 2 = 1 . We learn our masks with SGD (lr= 300 , batch size = 128 ) and our robust models with Adam (lr= 1 e -4 , batch size = 128 ). We ran all experiments on a single A100 80 GB GPU with 32 GB memory.

90

92

94

96

98

0.2

0.4

0.6

0.8

1.0

MNIST

VerT, VerT Model

VerT

SmoothGrad

GradCAM

Grad

FullGrad

SHAP

BCosNets

50

60

70

80

90

100

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

CelebA

VerT, VerT Model

VerT

SmoothGrad

GradCAM

Grad

FullGrad

SHAP

BCosNets

SHAP

(1x1)

SHAP

(8x8)

SHAP

(16x16)

SHAP

(32x32)

B-CosNets

4

4

0.90

0.92

0.94

0.96

0.98

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Effect of Q Ablation Study

STD = 0.01

STD = 0.1

STD = 1

STD = 2

10 Broader Impact

Our method, DiET , aims to transform black-box models into distractor robust, interpretable models, which produce easily verifiable feature attributions. As such, if applied correctly, it can help users and stakeholders of machine learning models better understand a model's predictions and behavior by isolating the features necessary for each prediction, which can help highlight biases, overfitting, mistakes, and more. It can also help to identify spurious correlations that naturally exist in datasets and are leveraged by models through identification of the signal-distractor decomposition. However, even if DiET does not identify a spurious correlation, that does not mean that further dataset cleaning, processing, or curation is not needed, as a different model may still learn a spurious correlation that was not leveraged by the original model. Furthermore, feature attributions often do not constitute a

complete explanation of a model. For instance, while an attribution tells us what was important, it does not tell us how it was important or how the model uses that feature. In all high stakes applications, it is still imperative that stakeholders think critically about each prediction and explanation, rather than blindly trusting either.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VerT, a method to distill *verifiable* models from black-box models. Concretely, These verifiable models are distilled by fitting a model $f_v$ that reproduces the predictions of the black-box model $f_b$ on a training set. Unlike the black-box model, the model $f_v$ is made verifiable by making its predictions self-consistent if a mask isolating the signal is applied to the input data. This mask is fitted by adding an extra objective to the model distillation objective, which guarantees that the mask is sparse and that the predictions of $f_v$ is similar for both the masked and the unmasked input. The methods is evaluated on 3 image datasets, including 2 where ground-truth feature importance is known. This analysis demonstrates that VerT outperforms standard gradient-based feature importance methods.\n\nSTRENGTHS:\n**Good writing.** The paper was really easy to read and follow. The notations are on-point and intuitive. \n\n**Solid empirical validation.** The experiments provided in Section 5 are convincing and thoroughly verify important claims: the features identified as salient by VerT have the strongest impact when masked (which is not surprising given the optimization objective underlying VerT), these features have a reasonable overlap with the ground-truth salient features in a setting where the later are known and VerT shows improved robustness. The authors also show that VerT improves the robustness to adversarial attacks on the explanations  with respect to gradient-based feature importance methods (which is unsurprising as the attacks are designed to fool gradient-based methods specifically). Insightful illustrative examples to understand the gains of VerT are provided in Figure 2.\n\nWEAKNESSES:\n**Restrictive assumption on replacement distribution.** In Lines 125-137, the authors discuss the importance of the choice of the replacement distribution $\\mathcal{Q}$ in order to avoid creating OOD examples by masking. It should be mentioned that some important masking stategies, such as [Gaussian blurs](https://arxiv.org/abs/1704.03296), are omitted from this discussion as these are conditioned on the input image $\\mathbf{x}$. I am wondering if it is even possible to define masking strategies with replacement distributions *independent* of the input $\\mathbf{x}$ that do not create OOD examples. Intuitivelly, it is legitimate to expect the replacement input $\\mathbf{q}$ to have (at least) some information about $\\mathbf{x}$ to avoid replacing the features of $\\mathbf{x}$ by OOD values. I would recommend the author to discuss this point thoroughly and (possibly) acknowledge this as a limitation of the work.\n\n**Unrealistic signal-distractor decomposition.** The signal-distractor decomposition defined in Definition 3 is key for Theorem 1. It appears to me that this decomposition is unrealistic for a simple reason: in the underlying DGP, the mask $\\mathbf{m}$ and the signal $\\mathbf{s}$ are independent. To make this point more clear, I would like to consider the example discussed in the paper. If we consider a cow detection task, the signal $\\mathbf{s}$  would typically correspond to the cow-part of the image. If that is the case, the mask $\\mathbf{m}$ should depend on the position of the cow on the image (e.g. performing a translation on the signal $\\mathbf{s}$ should result in a similar translation on the mask $\\mathbf{m}$ in principle). Again, I would recommend the authors to comment on the realism of their assumptions.  \n\n\n\n### Minor Weaknesses \n\n- Algorithm1: what is $M(\\mathbf{x})$? Is this the same as $\\mathbf{m}(\\mathbf{x})$? Also, should there be a minus sign in front of the gradients if the objectives are minimized? \n- In the appendices, Theorem 2 corresponds to Theorem 1 the main paper.\n\nQUESTIONS:\nAll my questions and recommendations are contained in the weakness section. I will not repeat them here to avoid redundancies. I will consider improving my recommendation if the above weaknesses are addressed by the authors.\n\nLIMITATIONS:\nSome limitations of the work are discussed in Section 6 of the main paper. I do not believe that negative societal impacts are a real concern for this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe present work introduces a theoretical framework of verifiability of feature attributions based on the sparest (binary) feature attribution mask that only barely changes the models' output. The authors theoretically (and empirically) show that for signal-distractor decomposable datasets off-the-shelf black-box models cannot be verified according to their definition of verifiability. The main reason being that off-the-shelf models cannot handle the OOD samples created by the masking intervention (\"feature replacement\"). To overcome this, they propose a finetuning scheme, in which they make off-the-shelf black-box models robust to such feature interventions. Specifically, they alternatingly optimize for the sparsest mask (that only barely changes the models' output) and apply a distillation loss. While the former makes the model robust to the feature interventions, the latter ensures the similarity to the original model. Experimentally, they show that VerT improves interpretability over gradient-based methods. VerT robustly identifies the signals (most salient features) of the inputs, while retaining the performance of the original model.\n\nSTRENGTHS:\n- The theoretical verifiability framework is simple yet sound.\n- The theoretical analysis (Sec 3.2) is interesting and supported both theoretically as well as empirically.\n- The finetuning scheme is simple yet effective. It does not change the models' output while simultaneously enhancing its interpretability by making it robust to the input feature removal.\n- The paper is clearly written (except for some small issues; see questions & suggestions below) and easy to follow.\n- Code is provided.\n\nWEAKNESSES:\n- The major weakness of the present work is that it does not compare to any removal-based feature attribution methods (e.g., [SHAP](https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html), [FastShap](https://openreview.net/forum?id=Zq2G_VTV53T), etc.), models also trained on feature attributions (e.g., [right for the right reasons](https://www.ijcai.org/proceedings/2017/371)), nor inherently interpretable models (e.g., [JAM](https://arxiv.org/abs/2103.01890) or [B-Cos](https://openaccess.thecvf.com/content/CVPR2022/html/Bohle_B-Cos_Networks_Alignment_Is_All_We_Need_for_Interpretability_CVPR_2022_paper.html)); only comparisons to gradient-based methods are provided.\n- The experiments always include manually-defined spurious correlations (to contain clear signals and distractors). While this provides empirical evidence for their theoretical framework, it would be meaningful to compare the method on “untouched”, more challenging datasets besides Celeb, e.g., ImageNet.\n- The finetuning scheme may change the model’s behavior (but not its output due to Eq. 2). While Tab. 2 shows that the prediction stays similar, it may change what type of signals in the inputs the classifier considers for its prediction (and its inner workings). Consequently, the finetuned model may not be faithful to which signal it uses (or how it processes it) for its prediction.\n\nQUESTIONS:\n- In Fig. 2, do all methods use the same model (finetuned or not finetuned version)?\n- Is there any explanation for the large difference for the non-finetuned and finetuned models using VerT in Fig. 4? Is the adversarial training objective also used during finetuning? More generally, would finetuning without model distillation (Eq. 2) and data distillation in Eq. 1 lead to more robust (and interpretable) models that only focus on the important parts of the image?\n- How did the authors set $\\lambda_1, \\lambda_2$?\n\n**Suggestions**\n- [B-Cos networks](https://openaccess.thecvf.com/content/CVPR2022/html/Bohle_B-Cos_Networks_Alignment_Is_All_We_Need_for_Interpretability_CVPR_2022_paper.html) or [BagNets](https://openreview.net/forum?id=SkfMWhAqYQ) could also be mentioned as inherently interpretable models.\n- The abbreviations in L68 could be introduced.\n- There are several recent advancements of concept-based models that close the performance gap, e.g., [post-hoc CBMs](https://openreview.net/forum?id=HAMeOIRD_g9) that could be mentioned in the related work.\n- Theorem 2 in the supplemental should have the same number as Theorem 1 in the main text as well as formulation to make it easier for readers to find.\n- In the experiments section there are several missing “Table/Figure {num}”.\n- The hyperparameters are neither described in the main text nor supplement.\n- To which Figure/Table are L288-299 referring to?\n- There are several typos throughout the present submission.\n- Results for Celeb in Table 3 in the supplement are missing.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method called Verifiability Tuning (VerT), which transforms black-box models into models that naturally yield faithful and verifiable feature attributions. Authors further conduct experiments on semi-synthetic and real-world datasets to verify the effectiveness of the proposed VerT method.\n\nSTRENGTHS:\n1. The motivation of this paper is clear.\n\n2. This paper focuses on the faithfulness of post hoc explanation methods, which is a very important topic in XAI.\n\nWEAKNESSES:\n1. The equation in Definition 1 is problematic. For example, let three positive input variables $a=b=c>0$ have a MAX operation, $output = \\\\max \\\\{a,b,c,0\\\\}$. Then, we mask any two input variables will result in different results. Specifically, we can mask $a$ and $b$, and keep $c$ unchanged. We can also mask $c$ and $b$, and keep $b$ unchanged. These two masking methods will result in different explanation results, but the actual importance of $a$, $b$, $c$ to the inference is the same. Hence, the equation in Definition 1 is problematic.\n\n2. I disagree with your claim on the optimal Q. Specifically, the optimal Q can be found. Theoretically, the optimal Q should be the same as the distributions of the input image, i.e., setting q=x (in this case $\\epsilon=0$ in Definition 1). Although this setting q=x conflicts with the motivation of masking the input, it is indeed the optimal solution to q in mathematics.\n\n3. The ground truth of attribution methods constructed in experiments (in Figure 2) is too simple.\n\n(1) If a classifier is powerful enough to just use five pixels for inference, instead of using the entire foreground of “4,” then is this classifier better or worse? The proposed evaluation does not consider this as a good explanation if annotating the entire foreground as the ground truth.\n\n(2) Defining “4” as the foreground also seems problematic. The edge feature of “4”contains both pixels in foregrounds and pixels in background. Hence, information encoded in background can be used for inference theoretically. In this way, pixels outside of the “4” should also be considered as the ground truth. However, there exists another problem that how many pixels outside of“4”should be included as the ground truth.\n\n(3) The correlation between dark hair and glass in CelebA dataset is just assumed. Specifically, this correlation is not a necessary condition for hair color classification, because the DNN can either exclusively use glasses for classification, or exclusively use hair, or use both hair and glasses for classification. The correlation is assumed and used for evaluation, but the correlation is not a certificated truth of a DNN.\n\nThis is a typical case when the input information for inference is redundant. When a small part of foreground objects are already enough for classification, it is difficult to annotate the ground-truth attention of a DNN.\n\n4. Pixel perturbation tests are circular arguments, because the proposed method is learned by minimizing the loss. The loss is designed to mask unimportant pixels for inference. \n\n5. Only the accuracy is not enough to evaluate the faithfulness. Please compare the advantages of the proposed method with advantages of the Shapley value. We cannot assume that the direct change of the output caused by the masking of an input variable is the exact importance of the input variable. Please apply more sophisticated evaluation metrics for attributions proposed in recent years.\n\n6. Authors just compare the proposed method with gradient-based explanation method, which is a quite weak competing method. Authors are suggested to compare the proposed method with more sophisticated baselines, such as Shapley values, DeepLIFT, IG, etc.\n\n7. Why “QFA applied to a model from a Q-verifiable model class” can be considered as “a verifiable feature attribution?” Authors do not provide proofs to support this claim. Moreover, what is the definition of “a verifiable feature attribution?” Please clarify how a feature attribution can be considered as “verifiable.”\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nNo, the authors do not discuss limitations of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a way to verifiably get feature attributions of the ground truth signal when the input can be decomposed into independent signals and distractor features, assuming there is a counterfactual generator Q that can provide sparse attributions. The process consists of first deriving (\\epsilon, Q) feature attributions for each point to get optimal masks, and second use distillation to train a new model that matches the outputs of the QFA with the original prediction while staying close to the original model. The first and second steps are alternated to minimize the overall loss. A rounding scheme is used to stabilize training with hard masks and superpixels are used. The approach is tested on MNIST, Chest X-ray, and CelebA. \n\nUpdate: After the discussion, I have updated my score accordingly. Since the authors believe scenarios that satisfy the signal-distractor decomposition are likely to exist, it would be great to incorporate such a concrete example somewhere in the text.\n\nSTRENGTHS:\nThe paper proposed an approach that, in principle, could potentially extract the ground truth signal (if all assumptions are met). \n\nThe approach, even though it trains a new model, replicates the original fairly well. \n\nThe experiments are couple with 2 ablations on adversarial robustness and sensitivity to hyperparameters.\n\nWEAKNESSES:\nThe signal decomposition assumes that the signal and distractors are generated independently, and that therefore the correct feature attribution is the sparsest one. This is not going to always be the case in real settings. \n\nThe method requires a strong counterfactual generator Q to ensure correct recovery of sparse attributions. However, in practice the authors just use a dirac delta of the dataset mean which is not really a counterfactual generator. \n\nThe evaluation is specific to vision, and the two non-MNIST datasets have spurious correlations injected into the dataset. It'd be nice to have some kind of breadth here beyond vision, or to have some results on more natural data that did not need to be artificially correlated. \n\nThe claim that one can change any black box model into a verifiably interpretable model might be a bit strong---it seems to also depend on a number of assumptions (ground truth decomposability) and having a powerful generator in order for the \"verified\" part to hold true.\n\nQUESTIONS:\nTheorem 1 states that QFA applied to optimal predictors from the function class recovers the signal distribution. The corollary then states that QFA does not recover the signal when applied outside of the function class. Neither of these address non-optimal predictors from the function class---what happens there? The last sentence of the section seems to state that feature attributions from the function class are in fact able to recover the signal, but this does not seem to strictly follow from the theorem and corollary.  \n\nWhat superpixels are being used? \n\nHow do we know that we've reached the optimal f^*? It seems important to know this due to Theorem 1 requiring an optimal predictor. \n\nThe practical details states at the end that replacing masked pixels with a counterfactual distribution Q somehow ensures that Q is an optimal counterfactual distribution. It is then asserted that this means the f comes from the Q-verifiable model class. I am not following either of these logical jumps, can you explain? There is nothing about Q in the pseudocode either. This seems to be critical, since we need to find an optimal f from Q for the signal to be recovered. Is there an ablation you can use to show how much of an effect Q has experimentally? \n\nIt is not entirely clear what the simplified dataset is, which appears to be spontaneously used in Section 5. \n\nWhy is the evaluation focused on gradient-based feature attributions? Why not other classic local surrogates like LIME/SHAP which are also widely used?\n\nLIMITATIONS:\nThe limitations do in fact mention that a decomposition must exist.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}