Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability
Authors: Usha Bhalla
Suraj Srinivas
Himabindu Lakkaraju
Venue: NeurIPS 2023 poster (2023)
Keywords: Machine Learning Explainability
Machine Learning Interpretability

Abstract: With the increased deployment of machine learning models in various real-world applications, researchers and practitioners alike have emphasized the need for explanations of model behaviour. To this end, two broad strategies have been outlined in prior literature to explain models. Post hoc explanation methods explain the behaviour of complex black-box models by identifying features critical to model predictions; however, prior work has shown that these explanations may not be faithful, in that they incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture, meaning their explanations are naturally faithful, but they often exhibit poor predictive performance due to their limited expressive power. In this work, we identify a key reason for the lack of faithfulness of feature attributions: the lack of robustness of the underlying black-box models, especially the erasure of unimportant distractor features in the input. To address this issue, we propose Distractor Erasure Tuning (DiET), a method that adapts black-box models to be robust to distractor erasure, thus providing discriminative and faithful attributions. This strategy naturally combines the ease-of-use of post hoc explanations with the faithfulness of inherently interpretable models. We perform extensive experiments on semi-synthetic and real-world datasets, and show that DiET produces models that (1) closely approximate the original black-box models they are intended to explain, and (2) yield explanations that match approximate ground truths available by construction.

Diseriminative Feature Attributions: Bridging Post
Hoc Explainability and Inherent Interpretability
Usha Bhalla*
Suraj Srinivas Harvard University
Harvard University
usha_bhalla@g harvard.edu
ssrinivas@seas. hawvard. edu
Himabindu Lakkaraju Harvard University tlakkaraju@hbe edu
Abstract
With the increased deployment of machine learning models in various real-world applications, researchers and practitioners alike have emphasized the need for explanations of model behaviour. To this end, two broad strategies have been outlined in prior literature to explain models. Post hoc explanation methods explain the behaviour of complex black-box models by identifying features critical model predictions; however, prior work has shown that these explanations may not be faithful, in that they incorrectly attribute high importance to features that are unimportant or non-diseriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture, meaning their explanations are naturally faithful, but they often exhibit poor predictive performance due to their limited expressive power. In this work, we identify key reason for the lack of faithfulness of feature attributions: the lack of robustness of the underlying black- box models, especially to the erasure of unimportant distractor features in the input. To address this issue, we propose Distractor Erasure Tuning (DiET), method that adapts black-box models to be robust to distractor erasure, thus providing discriminative and faithful attributions. This strategy naturally combines the ease use of post hoc explanations with the faithfulness of inherently interpretable models. We perform extensive experiments datasets, and show that DiET produces models that (1) closely approximate the original black-box models they intended explain, and (2) yield explanations that match approximate ground truths available by construction. Our code is made public here
semi-synthetic and real-world
Introduction
An important desideratum of machine learning models is for their predictions to be explainable. This allows both human domain experts as well as laypeople to better understand and trust the decisions made by models, and furthermore, is also regulatory requirement for high-stakes settings. For example, both the European General Data Protection Regulation (GDPR) [I] and the US Bill of Rights (2). require organizations to provide explanations for decisions made in high-stakes settings. common approach to producing such explanations from black-bor models in a post hoc manner is via feature attribution, which aims to identify important input features influencing model prediction. These methods typically work by locally approximating non-lineat models with linear ones Bl under some input perturbations such feature erasure. Intuitively, if the underlying model more sensitive
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to the erasure of feature A than feature B, these methods aim to attribute higher "importance" to feature than feature B. A fundamental prerequisite for feature be considered important by model is that it must first be useful in predicting the label, that is, it must be discriminative for the task. If feature does not contain information relating to the output label, then it cannot be used to predict the label, and thus feature attribution methods must not consider them important. However, recent works [4 [5] have found this not to be case feature attribution methods often highlight non-discriminative features. This motivates natural question: what causes feature attributions to highlight such non-discriminative features, making them unfaithful?
Tank: Har Color Claxefiaation
vvnsilive
Brunette
Black-Box Model
Brunelte Blonde
Impuss
Non. deccimnnaive
Blakk Hai
X
Cator Chasdiaation
Brunette
Robust
Disractor firsed
DIET Model
Brunette
Imputa
Daciminatii
atributons
Figure 1: Illustration of our method, Distractor Erasure Tuning. DiET models exhibit robustness to distractor erasure non-discriminative features such as backgrounds), allowing for the recovery of discriminative attributions.
Answering this question has been hard because of lack of theoretical understanding of the faith- fulness of feature attributions. While these notions have been used empirically [4] [5] to assess the quality of attributions, the theoretical characterization of optimally faithful feature attributions is missing in the literature. In this work, we tackle this problem by proposing framework for feature attribution methods emphasizing faithfulness and particularly discriminability. formalized signal-distracio decomposition for datasets. Essentially, the signal denotes the discriminative parts of the input (relative to a given task), while the distractor denotes the unimportant parts. Feature attri- bution methods then evaluated on how well they able recover the signal, thus also providing a well-defined notion of "ground-truth, truth"" We theoretically identify an important criterion to recover this ground-truth truth, that being the robustness of the model to the crasure of the input distractors. To enable black-box models to recover such ground truth attributions, we propose Distractor Erasure Tuning (DiET), method that adapts models to be robust the erasure of input distractors. Given that these distractor regions are not known in advance, our method works by alternating feature attribution and model adaptation. At high level, our work still uses feature attribution methods while adapting black-box models to have faithful attributions. Thus, this strategy naturally combines the ease of use of post hoc explanation methods with the faithfulness benefits of inherently interpretable models by providing the best of both alternatives. Our contributions are the following:
1. We present formalism for feature attribution that emphasizes discriminability and allows for a notion of well-defined ground truth, via the signal-dis signal-distractor decomposition of a dataset. We show that necessary for models to robust to distractor erasure for them to be able to recover this ground truth.
2. We propose distractor erasure tuning (DiET), a novel method that adapts black-box models make them robust distractor erasure.
3. We perform experiments on semi-synthetic and real-world computer vision datasets and show that DiET outputs models that are faithful to its original input models, and that their feature attributions are interpretable and match ground-trutt feature attributions.

---

2
Related Work
Post-Hoc Explainability. Post-hoc explainability methods aim to explain the outputs of fully trained black-box models either on an instance-level or global level. The most common post-hoc methods are feature attribution methods that rank the relative importance of features, either by explicitly producing perturbations [6] 71 or by computing variations of input gradients [8] Perturbation based methods are especially popular computer vision literature [i][4], which use feature removal strategies adapted specifically for image data. However, these methods all assume a specific form for feature removal, and we show theoretically in Section 3 that this can lead to unverifiable attributions.
Inherently Interpretable Models. Inherently interpretable models are constructed such that we know exactly what they do, either through their weights or explicit modular reasoning. As such, the explanations provided by these models are more accurate than those given by post-hoc methods; however, the performance of interpretable models often suffers when compared to unconstrained black-box architectures. The most common inherently interpretable model classes include linear models, decision trees and rules with limited depth, GLMs, GAMs [5]. JAMs [16118] prototype- and concept-based models [19 (0),, and weight-input aligned models 21 While [9, 20) leverage
the expressivity of deep networks, they constrain hypothesis classes significantly and still often suffer from a decrease in performance. Among these, our work most closely relates to JAMs, which amortise feature attribution generation using learnt masking function to generate attributions in a single forward pass, and trains black-box models using input dropout. On other hand, JAMs (1) trains models from scratch, whereas DiET can interpret black-box models, (2) amortises feature attributions using a masking function resulting in less accurate attributions. (3) trains models to be robust a large set of candidate masks via input dropout, leading to low predictive accuracy, whereas DiET trains models only to be robust to the optimal mask, leading to more flexibility and higher predictive accuracy.
Evaluating Correctness of Explanations. As explainability methods grow in number, so does the need for rigorous evaluation of each method. Research has shown that humans naively trust explanations regardless of their "correctness" [22,, especially when explanations confirm biases or look visually appealing. Common approaches evaluate explanation correctness rely on feature / pixel perturbation [23 9 [24], i.e., an explanation is correct if perturbing unimportant feature results in change of model outputs, whereas perturbing important features results in large model output change. Hooker et al. (5] proposed remove-and-retrain (ROAR) for evaluating feature attribution methods by training surrogate models on subsets of features denoted un/important by an attribution method and found that most maudient-based methods are no better than random. While prior works focused on developing metrics evaluate correctness of explanations, our method DiE produces models that have explanations that are accurate by design, according to perturbatt methods.
3 Theory of Discriminative Feature Attributions
In this section, we provide theoretical framework for feature attribution, including a well-defined ground truth. We start by identifying a common feature of feature attributions, their reliance on perturbations or erasure. Intuitively, feature attribution methods work by simulating removing certain features and estimating how the model behaves when those features are removed: removing unimportant features should not change model behaviour. Typically, this erasure is implemented by replacing features with scalar values, such as the mean of the dataset [1] [2]. However, this can result in autoo-diistrbuttin inputs that can confuse a classifier, thus making it difficult to create meaningful attributions.
To ground this argument in an example, consider model that classifies cows and camels. For an image of a camel, feature attribution might note that only the hump of the camel and the sand it stands on are important for classification. As such, we would expect that the sky was irrelevant to the classifier's 's prediction, and we can concretely test this by altering it and creating counterfactual sample. For example, we could mask the sky with an arbitrary uniform color; however, this may result in the sample being out-of-distribution for the model, and its prediction may change drastically even if the sky was not important for prediction. There are two strategies to overcome this problem. The first solution involves masking the sky in manner that preserves the naturalness of the image,
3

---

but this solution involves using harge-scale generative models, which themselves can contain biases and be uninterpretable. The second solution requires the classifier to be invariant the erasure of the pixels corresponding to the sky. which is our solution in this paper. We formalize this argument below by defining an erasure-based feature attribution method called (c. Q)- attribution.
Notation. Throughout this paper, we shall assume the task of classification with inputs x X with x E Rd and y E [I. with C-classes. We consider the class of deep neural networks f [Rd + AC which map inputs onto C-class probability simplex. This paper considers binary feature attributions, which are represented as binary masks E [0, where indicates an important feature and m; 0 indicates an unimportant feature.
3.1 Feature Attributions with Input Erasure
We first define trasure-based feature attribution such that the feature replacement method is explicit, and features are replaced with samples from a counterfactual distribution Q. Particularly, we are interested in binary attributions (i.c., a feature is considered important not) instead of real-valued ones.
Definition 1. (e, Q)feetuure attribution (QFA) is a binary mask m(f.x, Q) that relies on a model f(), an instance x, and a d-dimensional counterfactual distribution Q and is given by
m(,,x, Q)
arg min m'
||mmm| I such that
E q~2
[1((xx 9)) -- [(x]]] Se
where X2 (m g)
m) Qq
Thus, an (c, Q)- attribution (henceforth, QFA) refers to the sparsest mask that can be applied to an image such that the model's output remains approximately unchanged. QFA depends on the feature replacement distribution Q, where Q is independent of both x and This generalizes the commonly used heuristics of replacing unimportant features with the dataset mean, which case Q is a Dirac delta distribution at the mean value. The choice of Q is indeed critical, as an incorrect choice can hurt our ability to recover the correct attributions due to the resulting inputs being -istr--distiitution and the classifier being sensitive to such changes. Specifically, an incorrect Q can result in QFA being less sparse, as masking even a few features with the wrong Q would likely cause large deviations in the model's s outputs. As result, given model, we must aim to find the Q that leads to the sparsest QFA masks. However, the problem of searching over is complex, as requires searching over the space of all d-dimensional distributions, and furthermore, if the underlying model is non-robust, there may not exist any Q that leads to sparse attributions. To avoid this, we consider the inverse problem: given Q, we find the class of models that have the sparsest QFAs w.r.t. that particular Q. We call this the Q-robust model class, which we define below:
Definition 2. Q-robust model class (2): For some given distribution Q. the class of models F, for which Q has the sparsest QFA mask as opposed to any other Q', such that for all f E (.))).
Q arg min E. |m 2o
Intuitively, Q-robust models result in the sparsest QFA masks and can be thought of being robust to the erasure of "irrelevant" input features. Recalling our example of the cows and camels, we would like models to be robust to the replacement of the pixels corresponding to the sky but not necessarily robust to pixels corresponding to the camel or the cow itself. This distinguishes it from classical robustness definitions, which require models be robust to small perturbations (rather than erasure) at every feature uniformly. Thus Q-robustness is equivalent to enforcing robustness to the erasure of distractor features, notion that is central to this work. For the rest of this paper. we shall refer to QFA applied to model from a Q-robust model class as "matched" feature attribution - the same Q used to both define the model class and the feature attribution.
3.2 Recovering the Signal-Distractor Decomposition
In the study of feature attribution, the 'ground truth' attributions are often unspecified. Here, we show that for datasets that are signal-distractor decomposable, formally defined below, there exists a ground truth attribution, and feature attributions for optimal verifiable models are able to recover it.

---

Intuitively, given an object classification dataset between cows and camels, the "signal" refers to the regions in the image that are discriminative, or correlated with the label, such as the cows or camels. The distractor refers to everything else, such as the background or sky. Note that if objects in the background are spuriously correlated with the label, i.e. sand or grass, those would be part of the signal, not the distractor. We first begin by formally defining the signal-distractor decomposition.
Definition 3. A labelled dataset D = {(x, y) ) has regnal-ditt decomposition defined by masks m (x) ) E {0, 1) for every input x, where
1. X ④ mxx is the discriminative signal, where ply xx - ply x @ m(x))
2. x @ (1 m(x)) is the non oon-discimminative distractor, where pyy x @ (1 m(x))) = ply)
3. m(x) is the sparsest mask, i.e., m(x) = arg milm'(x) [lmoox)]loo such that (1) and (2) are
satisfied.
We propose to use the masks m(x) implied by the signal- -istrrcctr decomposition as ground truth feature attributions. These are meaningful as they precisely highlight the discriminative components of the image and ignore the non- dindiiirrmiiaeiee regions. Discriminability has previously been considered an important criterion to evaluate feature attributions propose its usage as ground truth.
4 S however, we here take a step further and
We observe first that the masks m(x) of the signal-distractor decomposition always exist: setting m(x) as an all-ones vector trivially satisfies conditions (I) and (2). When multiple masks exist, condition (3) requires us to choose the sparsest such mask mxx). Using the definitions provided, we show below an asymptotic argument stating that Q-robustness is a necessary condition to recover the optimal masks defined by the hignal-distracee decomposition.
Remark: A dataset D is said to have a "non-redundant signal" when the sparsest mask in condition (3) of the signal- distracoo decomposition is equal to the sparsest mask when (1) alone is satisfied.
Theorem 1. QFA applied Q-robust models recover the ground-trath masks when applied the Bayes optimal predictor 3 E FF (Q), for datasets 'D with a non-redundant signal.
Proof Idea. We first note that the optimal Q for QFA is equal to the ground truth distractor distribution, as this leads the sparsest QFA. If Q-robust model aims to recover the sparsest masks, then its QFA mask must equal that obtained by setting Q equal to the distractor. From the uniqueness argument in the definition of the signal-distractor decomposition, this is possible only when the optimal mask is recovered by QFA.
Corollary. QFA fails to recover the ground-truth masks when applied to predictors I f& (.()).
This follows from the fact that for any f & (Q).,, there exists some other Q' that results in a sparser mask, indicating that the ground truth masks are not recovered. Thus, this shows that feature attributions applied to the incorrect model class can be less effective - in this case, they fail to recover the ground truth masks. Further, the Bayes optimality is an important condition because it ensures that the resulting model is sensitive to all discriminative features in the input sub-optimal models are sub-optimal precisely because they fail to capture the signal from all the discriminative components of the input, and this can interfere with such models being able to recover ground truth masks. In practice, if we expect our models to be highly performant, we can expect them to be sensitive to all discriminative parts of the input and thus approximately recover ground truth masks. Finally, in practice, we do not have access to the ground truth masks for natural datasets, as the discriminative and the non-discriminative regions are not known in advance. In order to use these notions of ground truth, it is thus vital to construct semi-synthetic datasets where the discriminative parts are known. Thus, one can use semi-synthetic datasets to validate feature attribution approach and then apply it to gain insight into real datasets with unknown signal and distractor components.
To summarize, we have defined feature attribution method with the feature removal process made explicit via the counterfactual distribution Q. To minimize the sparsity of the attribution masks with given Q, we use models from the Q-robust model class F. (Q). Finally, we find that feature attributions derived from Bayes optimal models in the model class F.(O) are able to recover the troun masks and fail to do so otherwise.
5

---

DiET: Distractor Erasure Tuning
In the previous section we showed that given a Q-robust model fo Fi) we are able to apply QFA to recover the ground truth masks, In this section, we shall discuss how to practically build such robust models, given a pre-defined counterfactual distribution Q that defines the erasure method.
Relaxing QFA. We note that QFA as defined definition iis difficult to optimize in its current form due to its of &a regularization and its constrained form. To alleviate this problem, we perform two relaxations: first, we relax the fn objective into an & objective, and second, we convert the constrained objective to unconstrained one by using Lagrangian. The resulting objective function given in equation [ Assuming the model fo is known to us, we can minimize this objective function obtain (c, -e-auture attributions for each point X.
(0, (m(x))xxxx xEX
|m(x)]l +\1 (x)((x;)) fx (m, 0)1
(I)
mask sparsity
data distillation
Enforcing Model Robustness via Distillation. Assuming that the optimal masks denoting the signal-distractor decomposition are known w.r.t. every training data point (i.e., (m(x))xxx)) one can project any black-box model into a Q-robust model via distillation. Specifically, we can use equation 2 for this purpose, which contains (1) data distillation term enforce the constraint in QFA, and (2) model distillation term to enforce that the resulting model and original model are approximately equal. Accordingly, the black-box model fo and our resulting model fo both have the same model architecture, and we initialize fu fo.
Cuaan (x(x) = E XEX (sx(x:0) (xx) +A2 11b(() (x( 0)|
data distillation
model disrillation
Alternating Minimization between and m. We are interested in both of the above objectives: those masks enforce
would like recover the optimal masks from the dataset, well (e, Q) constraints via distillation to yield our Q-robust models. We can thus formulate the overall optimization problem as the sum of these terms, as shown in equation 3 Notice that both these objectives assume that either the optimal masks, or the robust model is known, and in practice, we know neither. A common strategy cases that involve optimizing over multiple sets of variables is to employ alternating minimization (25), which involves repeatedly fixing one of the variables and optimizing the other. We handle the constrained objective on the mask variables via projection, i.c.. using hurd-thresholding rounding to yield binary masks.
8r (x)) = arg min (Cuain(, (m)())) + LoFa (m(x)))) 6.m
(3)
such that
m(x) E 10, VX
Iterative Mask Rounding with Increasing Sparsity. In practice, mask rounding makes gradient- based optimization unstable due to sudden jumps in the variables induced by rounding. This problem commonly arises when dealing with sparsity constraints. To alleviate this problem, common in the model pruning literature (26] called iterative pruning, which involves introducing a rounding schedule, where the sparsity of the mask is gradually increased during optimization steps. Inspired by this choice, we employ a similar strategy over our mask variables.
heuristic that
Practical Details. We implement these objectives as follows. First, we initialize the robust model to be the same as the original model, f. = So. and the mask to be all ones, D, = m @ Da, m 1. We then iteratively (1) simplify D. by optimizing LQFA until m converges, (2) round m such that it is binary (i.e. D. a subset of features in Da rather than weighting of them), and (3) update fú by minimizing Cerain such that D. is equally informative as Da to fo and fo is functionally equivalent

---

to fo. As per Definition 2 we replace masked pixels in D, with a pre-determined counterfactual distribution Q. This ensures that the given is the optimal counterfactual distribution for f... meaning Jo comes from the Q-robust model class (Q). The pseudocode is given in Algorithm
Algorithm 1 Distractor Erasure Tuning
Input: Dataset Da (Ir, v), model Sa. hyperparameter l rounding steps Hyperparameters: k rounding steps, u mask scaling factor, s(t) sparsity at step t {m(x)), s.t. mxx) +- ones with shape Rd/u
> Init mask mxx) to ones Init robust model fu to fb
fo 4 fb
for rounding steps do while LoFA not converged do {m(x)} + (m(x)) + vm end while
m <- round(m, s(t)) A m E (m(x))
while Luain not converged do
Jo t -o + VoLunin
end while
end for return fm
Mask Scale. In order to encourage greater "human interpretability." we increase the pixel size of the masks by lowering their resolution. We do this by downscaling the masks before optimization. Concretely, we initialize the masks be of size my = xa/u uu where Id is the dimension of the image x and u is the pixel size we wish to consider. We then upsample the mask to be of dimension Fa before applying x. The more we downscale the mask by (i.e. the greater u is), the more interpretable and visually cohesive the distilled dataset x, is.
5 Experimental Evaluation
In this section, we present our empirical evaluation in detail. We consider various quantitative and qualitative metrics evaluate the correctness of feature attributions given by DiET models well as the faithfulness of DiET models to the models they are meant to explain. We also evaluate DiET models' ability to explain models manipulated to have arbitrary uninformative input gradients. Finally, we analyze the effect of the mask downscaling hyperparameter on attributions. Comparisons to additional baselines beyond those shown in Figure are given in
The first is a harder variant of MNIST where the digit is randomly placed on a small subpatch of a colored background. Each sample also contains small distractor digits and random noise patches. For this dataset, we consider the signal to be all pixels contained within the large actual digit, and the distractor to be all pixels in the background, noise, and smaller digits. Chest X-ray: Second, we consider semi-synthetic chest x-ray dataset for pneumonia classification [27]. To control exactly what information the model leverages such that we can create ground truth signal-distractor decompositions, we inject spurious correlation into this dataset. We randomly place a small, barely perceptible noise patch on each image in the "normal" class. We confirm that the model relies only the spurious signal during classification by testing the model's drop in performance when flipping the correlation (adding the noise patches to the "pneumonia" class) and seeing that the accuracy goes from 100% 0%. As such, for this dataset, the signal is simply the noise patch and the distractor is the rest of the xray. CelebA: The last dataset is a subset of CelebA 28 for hair color classification with classes (dark hair, blonde hair, gray hair). We correlate the dark hair class with glasses to allow for qualitative evaluation of each methods' ability to recover spurious correlations. This dataset does not have a ground truth signal distractor decomposition, as there are many unknown discriminative spurious correlations the model may rely upon.
Datasets. Hard MNIST:
Models. For all experiments, we use ImageNet pre-trained ResNet18s for our baseline mod- els, fs. All models achieve over 96% test accuracy. We train DiET models with Q ~ Laed (()( (Da)), meaning that each image masked with a uniform color drawn from a normal distribution around the mean color of the dataset. For all evaluation, use Q Lasa N N(n (p(Da).
7

---

4
O
4
3
4
Faruu
4
DETIOunt
Fatiaal
Smechiond
Figure 2: Visualization of datasets and attribution methods considered in this work. Row Raw data samples, Row 2: Ground truth attributions, Row 3: DiET attributions, Rows 4-7: Baseline methods.
(the dirac delta of the dataset mean) to ensure that masked samples minimally out-of-distribution for the baseline models fo-
5.1 Evaluating the Correctness of Feature Attributions
Pixel Perturbation Tests. We test the faithfulness of our explanations the pixel perturbation variant proposed in [9 (5), where we mask the k least salient pixels as determined by any given attribution method and check for degradation in model performance with the mean of the dataset. This metric evaluates whether the k masked pixels were necessary for the model to make its prediction. As mentioned in previous works, masked samples come from a different distribution than the original samples, meaning poor performance after pixel perturbation performance on the masks or the feature attribution scores being incorrect. To disentangle the correctness of the attributions from the robustness of the model, perform pixel perturbation tests on ground truth feature attributions, with results reported the appendix. Note that our method returns binary masks, but this metric requires continuous valued attributions to create rankings. As such, for this experiment we use the attributions created by our method before rounding.
either be product of the model's
Results are shown in Figure 3 We find that the attributions produced by DiET, used in conjunction with DiET models outperform all baselines. Furthermore, DiET attributions tested on the baseline model also generally perform better than gradient-basce bbase attributions.
Intersection Over Union. We further evaluate the correctness of our attributions by measuring their Intersection Over Union (IOU) with the "ground truth" attributions. We use the signal from the ground truth signal-distractor decomposition as described in for the ground truth attributions. For each image, if the ground truth attribution is comprised of n pixels, we take the intersection over union of the top pixels returned by the explanation method and the ground truth pixels, meaning an IOU of corresponds to a perfectly aligned/correct attribution. Results are shown in []] where our method performs the best for both datasets. We report mean and standard deviation over the dataset for each method.
5.2 Evaluating the Faithfulness of DiET Models
To ensure that the DiET model (f+) returned by our method is faithful the the original model (fi) it approximates, we test the accuracy of DiET models with respect to the predictions produced by the original model. Specifically, we take the predictions of the original model fo be the labels of the dataset. We evaluate DiET models on both the original dataset (fa( ze fo (D))) and the simplified dataset (D.) Rs (D)), with results shown 2 faithful to the baseline models they approximate.
We see that ETTmeesss are highly
8

---

MNIST
Preamenis Chust Xeaa
Celebd
020
02
GA2
6-M
2.0
1.81
C99
CET aettbbutonn
Smoothiind
GONOCAN
Gnd
Leamt Wark Starsity
Orig no Nede!
Figure 3: Pixel perturbation tests (higher is better) for MNIST (left), Chest X-ray (middle), and CelebA (right) datasets. DiET's recommended mask sparsity shown as a vertical dashed line. We observe that DiET performs the best overall, Refer to Section 5. for details.
Table 1: Intersection Over Union Results
Table 2: Faithfulness of DiET to Original Model Hard MNIST
Hard MNIST
Chest X-ray
Chest X-ray
CelebA
DiET (Ours)
0.461 +0008
0.821 0.05
SmoothGrad
0.252:10.05
0.045 +0.05
Faithfulness Original Data
0.996
1.00
0.995
GradCAM
0.295 +0.09
0.000 10.00
Input Grad
0.117 +:005
0.017=00033
Faithfulness on 0.987
1.00
0.975
Simplified Data
FullGrad
0.389 +0010
0.528 +0.12
5.3 Qualitative Analysis of Simplified Datasets
We first explore how well DiET recovers signal-distractor decompositions. For CclebA, we see that all methods except for input gradients correctly recover the spurious correlation for the "dark hair/glasses" class, however only our method provides useful insights into the other two classes. We that method correctly identifies hair as the signal for the "blonde hair" class, whereas other methods simply look at the eyes, which are not discriminative. Furthermore, we see that for the "gray hair" class, method picks up on hair, as well as initially unknown spurious correlations such as wrinkles and bowties. For Hard MNIST, we see that our method clearly isolates the signal and ignores the distractor. FullGrad and GradCAM suffer from locality bias and tend to highlight the center of each digit. SmoothGrad and vanilla gradients are much noisier and highlight edges and many random background pixels. For the Chest X-ray dataset, we see that our method and FullGrad perfectly highlight the spurious signal. GradCAM again suffers from a centrality bias, and cannot highlight pixels on the edge. SmoothGrad and gradients appear mostly random to the human eye.
We also consider the visual quality of our attributions compared with the baselines (examples are shown in 2p. We find that our method, FullGrad, and GradCAM appear the most interpretable, as opposed to SmoothGrad and vanilla gradients, because they consider features at the superpixel level rather than individual pixels. We also see that GradCam and FullGrad seem relatively class invariant, and tend to focus on the center of most images, rather than the discriminative features for each class, providing for less useful insights into the models and datasets.
5.4 Robustness to Adversarial Manipulation of Explanations
In this section, we highlight our method's robustness to adversarial explanation manipulation. To this end, we follow the manipulation proposed in [29]. which adversarially manipulates gradient-based explanations. This is achieved by adding an extra term to the training objective that encourages input gradients to equal an arbitrary uninformative mask of pixels in the top left corner of each image. Note that model accuracy on the classification task is the same as training with only cross-entrops loss.
We repeat experiments for all evaluation metrics on these manipulated models, with pirtt-perturbatton shown below [6, and IOU, model faithfulness, and model robustness in the appendix. We see that gradient-based methods perform significantly worse manipulated models; however, method

---

remains relatively invariant. We also note while the models are only manipulated to have arbitrary input gradients, SmoothGrad and GradCAM are also heavily affected such that their attributions are entirely uninterpretable as well, as shown below.
MNIST (Manipulated Gradients)
1.
4
4
4
2.
Mocel
Smaororrd
GradCAM
Giad
4
Loarn Hall Spority
Saw Savinage
Groadhrch
Replactioon
Der
FalGos
Seaut
Figure 4: Pixel perturbation test and example images for models trained with gradient manipulation on Hard MNIST. Results for other datasets are in the appendix. Refer to Section 5.4 for details.
Attribution Sensitivity Hyperparamaters
We conduct ablation study on the choice of how much to downscale the mask by. The less downscale by, the more fine-grained the mask is, allowing for optimization over a larger set of masks. However, the more we downscale by, the visually cohesive or "interpretable" to humans the masks are. We evaluate the trade-off between these two via pixel perturbation tests over multiple downscaling factors and with qualitative evaluations of the final masks in 5 We that downscaling factor of performs the best on pixel perturbation tests. Increased factors of downscaling impose a greater locality constraint that results in informative pixels being masked, shown in the visualization.
Colebi
Pacii
4
1294 1% 378
Fraction NaciaS
Trcttonn Nadiad
Figure 5: Ablation study of pixel perturbation test with varying levels of mask downscaling for MNIST and CelebA. Example images for each factor are shown on the right.
Discussion
In this paper, we seek to build machine learning models such that their feature attributions remain discriminative. In particular, propose DiET, method that adapts black-box models into those that are robust to distractor erasure. We empirically evaluate DiET and find that the resulting models highly faithful the original and produce interpretable attributions that closely match the ground truth ones. Limitations of DiET include requiring full access to the training dataset and the baseline model. Furthermore, while it produces verifiable feature attributions that tell us how important each feature is to the model's prediction, it does not tell us what the relationship between important features and the output/label is, as is true with all feature attributions.
Acknowledgments and Disclosure of Funding
This work is supported part by the NSF awards IIS-2008461, IIS-204098,, IIS-2238714, Kempner Institute Graduate Fellowship, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design (D3) Institute Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
10

---

References
[1] The European Commission. General data protection regulation (gdpr). 2018. URL
https://commissii/ httpa///wwa......././/oooooo/aaa
bulles-business-eee htt.a//onnnnnnemmooorrrraaaaanna htt.aaa----iiiiiieaaaaaaaaaaaaar htaaa//aaaaaaaaaaaaaaaaaaaaaaaaa
[2] The White House. Blueprint for ai bill of rights. 2022. https:/::///w
pdf
[3] Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju. Which explanation should choose? a function approximation perspective to characterizing post hoc explanations. arXiv preprint 1xXiv:2206.01254 2022.
[4] Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? Advances in Neural Information Processing Systems, 34:2046 2059, 2021.
[5] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for in- terpretability methods deep neural networks. Advances in neural information processing systems, 32, 2019.
[6] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1 1135-1144, 2016.
[7] Scott M Lundberg and Su-In Lee. A unified approach interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4765-4774. Curran Associates, Inc., 2017. URL htttt::/ppappr...t nips. ct.ps.ccp/pper// 7062-a-umified-approo -o-terach------tteeeeeett-n----- model modee--peddictionsss
[8] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint 1706.0/7706.3882... 2017.
[9] Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. Advances neural information processing systems, 32, 2019.
[10] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[11] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818 833. Springer, 2014.
[12] Ruth C Fong and Andrea Vedaldi. Interpretable explanations black boxes by meaningful perturbation. In Proceedings of the IEEE international conference on computer vision, pages 3429-3437, 2017.
[13] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEF/CVF international conference on computer vision, pages 2950-2958, 2019.
[14] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017.
[15] Trevor J Hastie. Generalized additive models. In Statistical models in S, pages 249-307.
Routledge, 2017.
[16] Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theerrtiee perspective on model interpretation. In International Conference on Machine Learning, pages 883-892. PMLR, 2018.

---

[17] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wiid variable selection using neural networks. In International Conference on Learning Representations. 2019.
[18] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions their interpretations. In International Conference on Artificial Intelligence and Statistics, pages 1459-1467. PMLR, 2021.
[19] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[20] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference Machine Learning, pages 5338-533 5348. PMLR, 2020.
[21] Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment is all we need for interpretability. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10329-10338, 2022.
[22] Himabindu Lakkaraju and Osbert Bastani. how do fool you?" manipulating trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 79-85, 2020.
[23] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28( 28(1 (826666666- 2016.
[24] Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards transparent evaluation of model explanations. Advances in Neural Information Processing Systems, 35:15784-15799, 2022.
[25] Prateek Jain and Purushottam Kar. Non-convex optimization for machine learning. Foundations and Trends@ in Machine Learning, 10, 2017. URL Dtappprrss/iinn17FFPML.... ULLLaLLLpapprr//aat1TT/T.LL.,,,
[26] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[27] Daniel Kermany, Kang Zhang, Michael Goldbaum, al. Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley data, 2(2):651, 2018.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018
[29] Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Supplementary Material
7 Proofs
Theorem 2. Q-feature attributions applied Q-robust models (Fo(2)
recover the signal-distracee distractor decomposition for Bayes Bayes-optimal optimal predictor fo E (F()..
Proof. Consider QFA with c 0. Let f be the Bayes-optimal model, which implies that fz(
p(xlv) Pbayes(y) (y x) ply x) Σ, p i... i.c., the model estimates the correct conditional probabilities from the data given the class-condie class-conditional conditional generative probabilities. Note that this is only defined for inputs E X' in the support of the data distribution and not outside, i.e., Phages (y x) = p(u x) only when x E X.
Let us define x @ (1 m) = Xdistractor Adistractor- This is the distribution of distractor images, which (recall) are independent of the label y. Using this, we consider an idealized version of QFA, with Qideat = Xdistraccor- From Definition [ this results in generation of simplified inputs X, = x & m + q O (1 m). For q ~ Qideal Kasttaetoor we observe that x, X, the data distribution. Recall the Bayes optimal classifier is defined across the data distribution X' and applying QFA results in the sparsest m such that p(y x) = Pbages( x) Pbayes x,) ) = p(y x O m + @ (1 of y. This corresponds to the definition of the signal- -esttrrritt,,ttor, and thus it implies that QFA with Qideal recovers the mask defined by the signal- iinaa-distractor decomposition.
m)) = p(y O m). The last equality holds because q @ (1 m is independent
For any other value of + Qideat, we first consider the Q-robust Bayes optimal predictor Playes (y x). This has the property that Pbayes (y x) = Pbayea (y x) for x E X. We now compare mask m2 derived from applying QFA on Phayes and mask mQ from applying QFA with Qident on o P Pbayes' From the previous paragraph, we know that m Mideest Mideal is the ideal sparsest mask. However from Definition 2 mg is the sparsest mask. Thus it is the case that mg Mident, proving our overall result.
We now present proof for an additional statement not described in the main text, where we connect QFA to other commonly used feature attributions via the local function approximation framework 3 as follows.
Theorem 3. QFA an instance of the local function approximation framework (LFA), with (1) random binary perturbations, and (2) interpretable model class consisting of linear models with binary weights
Proof. Assume a black box model given by fo(x; (Eg [x( 9)) f(x)]le e) .loss function {(f, 9, I. E) (f(;; {) (()) neighborhood perturbation Z Uniform (0, 1)d, and an interpretable model family G being the class of binary linear models.
For these choices, it is easy to see that
arg min geg
arg min E (fo(x: [f TT) Allgllo geg &
arg min E. 1 E [11(((x -(x)) se) 1 -95) + Aااااl
geg
This above objective is minimized when y m², i.e., the ideal Q-FA mask, because this sets the first term be zero by definition, and the second sparsity term ensures the minimality of the mask.
D
13

---

8 Additional Results
Model Verifiabilityy We further test the verifiability of our models by evaluating how the model's performance changes when performing the pixel perturbation test on groundtruth attributions. This enables to disentangle the verifiability of the model from the correctness of the attributions, as we know that our attributions are correct. We use the same groundtruth attributions as in 3.1] We report the & norm between predictions made on the original samples and predictions made on the masked samples. We compare our verifiable models to the baseline models they approximate, as well as models trained with input dropout, which [8] proposes as their verifiable model class. Training with input dropout is equivalent to training fo with random masks and cross-entropy loss rather than optimized masks and fa prediction matching. Results are shown in where we see that our model performs similarly for masked and normal samples, whereas the other models do not.
Hard MNIST
Chest X-ray
DiET Model (S+)
0.027
0.0009
Original Model (fo)
0.107
0.0032
fo + Input Dropout
0.167
0.0536
Table 3: Model Verifiability
Robustness to Explanation Attacks. We report additional results on Chest X-ray and CelebA for the pixel perturbation tests, IOU tests, and model faithfulness tests for baseline models trained with manipulated gradients, as outlined in 5.4 We see that Di DiETmodels are still highly faithful and produce correct explanations even when derived from models adversarially trained to have manipulated explanations.
Table 4: Faithfulness of DiETModel for Manipulated Models
Hard MNIST
Chest
CelebA
X-ray
Accuracy on Original Data fo(Da) = fv(Da)
0.990
1.00
0.970
Accuracy on Simplified Data (D.) = fo(Da)
0.980
1.00
0.946
Table 5: Intersection Over Union Results for Manipulated Models
MNIST (manipulated)
Chest X-ray (manipulated)
DiET(Ours)
0.454 +-0.08
0.631+0.11
Method
SmoothGrad
0.158 +0.07
0.000 #0.00
GradCAM
0.040+0.06
0.000 #0.00
Input Grad
0.002:40.01
0.000 +0.00
FullGrad
0.333 10.12
0.004 #0.04
Signal vs Distractor Masking, For our CelebA experiments, there may have been unintended spurious correlations that we did not foresee and that our method did not recover, leading to a vacuous decomposition. To further support the results in Figure 3, we also perform experiments on masking the signal instead of the distractor, by masking out "important" pixels as opposed to the unimportant ones. The results for this experiment are shown in
14

---

ChestX-rar X-ray (Manipulated Gradients)
CelebA (Manipulated Gradients)
1.20
Recuracy
Accuracr
0.45
Dus Model
0.-
Dus Mood
0.80
Dus Origina Modu
Sreem
Dus Orginn Modul
0.75
0.89
3.40
naas noscs asos 0.52 6.935 Percent Macked
Percent Macked
Figure 6: Pixel perturbation tests for models trained with gradient manipulation for Chest X-ray (middle) and CelebA (right).
Table 6: Signal Masking experiments
Fraction Masked
Masked Distractor Ace
Masked Signal Acc
Random Mask Acc
0.2
0.971
0.772
0.965
0.4
0.967
0.569
0.935
0.6
0.967
0.392
0.826
0.8
0.960
0.339
0.587
These results indicate that for the DiETmodels, there do not exist pixels outside the shown signal regions in Figure 2 that contain information about the label. These results show that the signal- distractor decomposition for CelebA is very likely non-vacuous.
Finetuning less data. We recognize that limitation of this work is that it requires further training of the given models on their training datasets. To address this issue, we briefly explore whether DiETcan be finetuned
small subset of the data distribution (without labels) follows for MNIST. The original model is trained on 8835 samples from the train split. DiETis finetuned on 1500 samples from separate unlabeled validation split. We perform pixel perturbation on the remaining 8500 test samples. We compare this to DiETmodels finetuned on the full train set and note a minimal change in performance in
Table 7: Small Scale Finetuning
Percent Masked
DiET(Finetuned on Train Split)
DiET(Finetuned on Val Split)
0.977
0.962
0.966
0.937
98
0.738
0.691
99
0.234
0.288
Overall, this shows that there exist ways to decrease the computational complexity of our procedure when applied to larger datasets, which we haven't 't fully investigated yet.
Effect standard robustness. We conduct additional robustness experiments for Gaussian and Bernuulii noise with varying standard deviations and probabilities. Results are shown in & and We find that DiETmodels generally more robust than regular models. Note that neither DiETmodels nor the original models are explicitly trained or tuned for robustness on these distributions.
Additional Baselines: SHAP and BCosNets. We consider two additional explanation baselines: SHAP 7 and BCosNets 21.. Results for pixel perturbation tests and visualizations are shown Results for IOU tests are shown in 10
15

---

Table 8: Robustness experiments (Gaussian noise)
Hard MNIST
CelebA
STD
Ours
Original
Ours
Original
0.2
0.974
0.976
0.959
0.939
0.4
0.961
0.892
0.675
0.537
1.0
0.61
0.176
0.335
0.438
Table 9: Robustness experiments (Bernoulli noise)
Hard MNIST
CelebA
P
Ours
Original
Ours
Original
0.9
0.182
0.155
0.348
0.356
0.75
0.718
0.3
0.393
0.364
0.5
0.922
0.748
0.528
0,453
0.1
0.962
0.969
0.867
0.807
We note that in general SHAP does not often perform as well as most gradient-based methods for image data. We find that it performs similarly to random attributions.
We add B- B-CosNets as an inherently-ineee innrrntly-ineeepeetabble model baseline. We find that B B-CosNets performs comparably to our method for the IOU test on Hard MNIST. We were unable to train it to convergence on the Chest X-ray dataset. We also find that visualizations created by B-CosNets align with our expectations and appear visually interpretable. However, our method still significantly outperforms B-CosNets on the pixel perturbation test, showing that B- B-CosNets are not robust to perturbations of the distractor (i.e. are not verifiable). We also note that our method can be applied to trained black-box model of any architecture and training procedure, whereas B-CosNets, like all inherently interpretable models, cannot.
Table 10: SHAP and B-Cos IOU. Note that B-CosNets did not converge for the Chest X-ray dataset.
Hard MNIST
Chest X-ray
DiET
0.461 0.08
0.821 = 0.05
B-CosNets
0.465 + 0.07
SHAP
0.036 + 0.02
0.016 0.05
Effect of Choice of Q. We perform an ablation to test the effect that the choice of Q has empirically. We consider various parameterizations of the normal distribution used for Q, and find that results relatively consistent across the different choices of Q, as shown in g
9
Additional Implementation and Computation Details
Models were trained on the original train/test split given by https://githutt com/jayaneethal colorized -iiiiieddMIISTTTT for Hard MNIST and (27) for the Chest X-ray dataset and with a random 80/20 split for CelebA. Baseline models were trained with Adam for 10 epochs with learning rate le le-4 4 and batch size 256. All hyperparameters are included in the code for this paper. The model distillation and data distillation terms are weighted with X X2 = 1. We learn our masks with SGD (Ir=300, batch size 128) and our robust models with Adam (Ir=le - 4, batch size 128). We ran all experiments on single 1.00 80 GB GPU with 32 GB memory.
16

---

SHAP
ACUPS
0---
SHAP (16x16)
Fraction Mackod
ColebA
(2x33)
4
O
B-unNie
60
20
100
Fraction
Figure 7: Left: Pixel perturbation tests with added baselines of SHAP (orange) and B-CosNets (blue). Right: Visualization of SHAP and B-CosNets explanations on Hard MNIST and CelebA. Results for SHAP shown varying feature sizes. We note that SHAP explanations appear relatively random at all levels of granularity.
Effect of Ablation Study
Accuracy
0.7
0.6
0.5
510 2.01
370 2.1
3T0
Baiciin Vesel Accency
Fraction Masked
Figure 8: Pixel perturbation tests Hard MNIST for DiET models trained with various Qs.
Q ~ N(j, ) for in 00.
10 Broader Impact
Our method, DiET, aims to transform black-box models into distractor robust, interpretable models, which produce easily verifiable feature attributions. As such, if applied correctly, it can help users and stakeholders of machine learning models better understand a model's 's predictions and behavior by isolating the features necessary for each prediction, which can help highlight biases, overfitting, mistakes, and more. It can also help to identify spurious correlations that naturally exist in datasets and are leveraged by models through identification the signal-distractor decomposition. However, even if DiETdoes not identify spurious correlation, that does not mean that further dataset cleaning, processing, or curation not needed, different model may still learn spurious correlation that was not leveraged by the original model. Furthermore, feature attributions often do not constitute
17

---

complete explanation of model. For instance, while an attribution tells us what was important, it does not tell how was important or how the model uses that feature. In all high stakes applications, it is still imperative that stakeholders think critically about each prediction and explanation, rather than blindly trusting either.
18


Reviews:
[{"review_id": "ByHtnVZI3T", "reviewer": "Reviewer_QRQ5", "summary": "The paper proposes a way to verifiably get feature attributions of the ground truth signal when the input can be decomposed into independent signals and distractor features, assuming there is a counterfactual generator Q that can provide sparse attributions. The process consists of first deriving (\\epsilon, Q) feature attributions for each point to get optimal masks, and second use distillation to train a new model that matches the outputs of the QFA with the original prediction while staying close to the original model. The first and second steps are alternated to minimize the overall loss. A rounding scheme is used to stabilize training with hard masks and superpixels are used. The approach is tested on MNIST, Chest X-ray, and CelebA. \n\nUpdate: After the discussion, I have updated my score accordingly. Since the authors believe scenarios that satisfy the signal-distractor decomposition are likely to exist, it would be great to incorporate such a concrete example somewhere in the text.", "questions": "Theorem 1 states that QFA applied to optimal predictors from the function class recovers the signal distribution. The corollary then states that QFA does not recover the signal when applied outside of the function class. Neither of these address non-optimal predictors from the function class---what happens there? The last sentence of the section seems to state that feature attributions from the function class are in fact able to recover the signal, but this does not seem to strictly follow from the theorem and corollary.  \n\nWhat superpixels are being used? \n\nHow do we know that we've reached the optimal f^*? It seems important to know this due to Theorem 1 requiring an optimal predictor. \n\nThe practical details states at the end that replacing masked pixels with a counterfactual distribution Q somehow ensures that Q is an optimal counterfactual distribution. It is then asserted that this means the f comes from the Q-verifiable model class. I am not following either of these logical jumps, can you explain? There is nothing about Q in the pseudocode either. This seems to be critical, since we need to find an optimal f from Q for the signal to be recovered. Is there an ablation you can use to show how much of an effect Q has experimentally? \n\nIt is not entirely clear what the simplified dataset is, which appears to be spontaneously used in Section 5. \n\nWhy is the evaluation focused on gradient-based feature attributions? Why not other classic local surrogates like LIME/SHAP which are also widely used?", "limitations": "The limitations do in fact mention that a decomposition must exist.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposed an approach that, in principle, could potentially extract the ground truth signal (if all assumptions are met). \n\nThe approach, even though it trains a new model, replicates the original fairly well. \n\nThe experiments are couple with 2 ablations on adversarial robustness and sensitivity to hyperparameters.", "weaknesses": "The signal decomposition assumes that the signal and distractors are generated independently, and that therefore the correct feature attribution is the sparsest one. This is not going to always be the case in real settings. \n\nThe method requires a strong counterfactual generator Q to ensure correct recovery of sparse attributions. However, in practice the authors just use a dirac delta of the dataset mean which is not really a counterfactual generator. \n\nThe evaluation is specific to vision, and the two non-MNIST datasets have spurious correlations injected into the dataset. It'd be nice to have some kind of breadth here beyond vision, or to have some results on more natural data that did not need to be artificially correlated. \n\nThe claim that one can change any black box model into a verifiably interpretable model might be a bit strong---it seems to also depend on a number of assumptions (ground truth decomposability) and having a powerful generator in order for the \"verified\" part to hold true.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive review! Overall, we feel you may have misunderstood some parts of the paper, and we’d like to clarify these below.\n\n*“The signal decomposition assumes that the signal and distractors are generated independently, and that therefore the correct feature attribution is the sparsest one. This is not going to always be the case in real settings.”*\n\nPlease refer to global comments (G2, G4). Can you please provide examples of settings where you believe these may not hold?\n\n---\n\n*“The method requires a strong counterfactual generator Q to ensure correct recovery of sparse attributions. However, in practice the authors just use a dirac delta of the dataset mean which is not really a counterfactual generator. ” “The claim that one can change any black box model into a verifiably interpretable model might be a bit strong”*\n\nPlease refer to (G1, G3) for clarification. Theorem 1 states that **any** Q-verifiable model used with a corresponding QFA can recover the optimal signal-distractor distribution. Note that the Q here is unrelated to the distractor distribution ($\\mathcal{X}_{distractor}$). The important thing here is that QFA is only used on Q-verifiable models with the same “Q” used for both – this is exactly what leads to verifiable explanations. \nNote also that for our method we use a Normal distribution for Q when training VerT models. We use the dirac delta of the dataset mean **only** for pixel perturbation evaluations to remain consistent with prior works and evaluation of baselines. We have also provided an ablation on the choice of Q in (G5). \n\n---\n\n*“Theorem 1 states that QFA applied to optimal predictors from the function class recovers the signal distribution. The corollary then states that QFA does not recover the signal when applied outside of the function class. Neither of these address non-optimal predictors from the function class---what happens there? The last sentence of the section seems to state that feature attributions from the function class are in fact able to recover the signal, but this does not seem to strictly follow from the theorem and corollary. ” “How do we know that we've reached the optimal f^\\*? It seems important to know this due to Theorem 1 requiring an optimal predictor.”*\n\nPlease refer to (G3, G4). The main point is that QFA must be used with a Q-verifiable model class (with the same Q used for both) for recovery. When the predictor is non-optimal, intuitively the model may not learn to “look” at the right portions of the input, and thus may not recover the signal, which is a dataset-dependent quantity. \n\nThanks for bringing to our notice the final line in the section, we will change it to “Finally, we find that feature attributions derived from **optimal models** in model class Fv (Q) are able to recover the signal-distractor decomposition of datasets.”\n\n---\n\n*“What superpixels are being used?”*\n\nWe do not use superpixels in this work. We instead use a low-resolution binary mask (say 8x8) which we upscale (to, say 224x224) using bilinear interpolation. Please refer to the paragraph on “Mask Scale” in line 222 of the paper. We use the word “superpixel” in that section loosely to refer to a group of pixels obtained from this procedure. We shall remove the usage of that word for clarity.\n\n---\n\n*“The practical details states at the end that replacing masked pixels with a counterfactual distribution Q somehow ensures that Q is an optimal counterfactual distribution. It is then asserted that this means the f comes from the Q-verifiable model class. I am not following either of these logical jumps, can you explain? There is nothing about Q in the pseudocode either. This seems to be critical, since we need to find an optimal f from Q for the signal to be recovered. Is there an ablation you can use to show how much of an effect Q has experimentally? ”*\n\nWe’re sorry for the confusion, please refer to (G1-G4). Our algorithm “verifiability tuning” converts a black-box model (which has an unknown Q) into a Q-verifiable model (for some Q), which results in the sparsest possible masks when used with QFA (with the same choice of Q).\n\nThe results of our ablation are given in (G5), where we see that for different parameterizations of Q, the results are still relatively consistent.\n"}, {"review_id": "zGkBjMSx7j", "reviewer": "Reviewer_2ozo", "summary": "This paper proposes a method called Verifiability Tuning (VerT), which transforms black-box models into models that naturally yield faithful and verifiable feature attributions. Authors further conduct experiments on semi-synthetic and real-world datasets to verify the effectiveness of the proposed VerT method.", "questions": "Please see weaknesses.", "limitations": "No, the authors do not discuss limitations of the proposed method.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of this paper is clear.\n\n2. This paper focuses on the faithfulness of post hoc explanation methods, which is a very important topic in XAI.", "weaknesses": "1. The equation in Definition 1 is problematic. For example, let three positive input variables $a=b=c>0$ have a MAX operation, $output = \\\\max \\\\{a,b,c,0\\\\}$. Then, we mask any two input variables will result in different results. Specifically, we can mask $a$ and $b$, and keep $c$ unchanged. We can also mask $c$ and $b$, and keep $b$ unchanged. These two masking methods will result in different explanation results, but the actual importance of $a$, $b$, $c$ to the inference is the same. Hence, the equation in Definition 1 is problematic.\n\n2. I disagree with your claim on the optimal Q. Specifically, the optimal Q can be found. Theoretically, the optimal Q should be the same as the distributions of the input image, i.e., setting q=x (in this case $\\epsilon=0$ in Definition 1). Although this setting q=x conflicts with the motivation of masking the input, it is indeed the optimal solution to q in mathematics.\n\n3. The ground truth of attribution methods constructed in experiments (in Figure 2) is too simple.\n\n(1) If a classifier is powerful enough to just use five pixels for inference, instead of using the entire foreground of “4,” then is this classifier better or worse? The proposed evaluation does not consider this as a good explanation if annotating the entire foreground as the ground truth.\n\n(2) Defining “4” as the foreground also seems problematic. The edge feature of “4”contains both pixels in foregrounds and pixels in background. Hence, information encoded in background can be used for inference theoretically. In this way, pixels outside of the “4” should also be considered as the ground truth. However, there exists another problem that how many pixels outside of“4”should be included as the ground truth.\n\n(3) The correlation between dark hair and glass in CelebA dataset is just assumed. Specifically, this correlation is not a necessary condition for hair color classification, because the DNN can either exclusively use glasses for classification, or exclusively use hair, or use both hair and glasses for classification. The correlation is assumed and used for evaluation, but the correlation is not a certificated truth of a DNN.\n\nThis is a typical case when the input information for inference is redundant. When a small part of foreground objects are already enough for classification, it is difficult to annotate the ground-truth attention of a DNN.\n\n4. Pixel perturbation tests are circular arguments, because the proposed method is learned by minimizing the loss. The loss is designed to mask unimportant pixels for inference. \n\n5. Only the accuracy is not enough to evaluate the faithfulness. Please compare the advantages of the proposed method with advantages of the Shapley value. We cannot assume that the direct change of the output caused by the masking of an input variable is the exact importance of the input variable. Please apply more sophisticated evaluation metrics for attributions proposed in recent years.\n\n6. Authors just compare the proposed method with gradient-based explanation method, which is a quite weak competing method. Authors are suggested to compare the proposed method with more sophisticated baselines, such as Shapley values, DeepLIFT, IG, etc.\n\n7. Why “QFA applied to a model from a Q-verifiable model class” can be considered as “a verifiable feature attribution?” Authors do not provide proofs to support this claim. Moreover, what is the definition of “a verifiable feature attribution?” Please clarify how a feature attribution can be considered as “verifiable.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review!\n\n*“The equation in Definition 1 is problematic. For example, let three positive input variables a=b=c ? 0 have a MAX operation, output= max{a,b,c,0} ...actual importance of a, b, c to the inference is the same...”*\n\nGreat example! This shows that with duplicate features, the QFA optimization problem does not have a unique solution. This is expected, as there is no single “minimal” component of the input that drives the output, and this multiplicity highlights a critical property of the model. \n\nWe’d also like to point out that similar notions of feature importance have been used in literature, see references [8,9,10] in the main paper. We’d like to further point out that defining “actual importance” of variables is an open problem, and there is no agreement in the field regarding what it constitutes (see reference [1] in our paper). Overall, we don’t see why multiplicity makes definition 1 problematic. Can you please elaborate in case we missed something?\n\n---\n\n*“I disagree with your claim on the optimal Q. Theoretically, the optimal Q should be the same as the distributions of the input image, i.e., setting q=x (in this case e=0 in Definition 1)...”*\n\nThis is incorrect. Please note that the distribution Q is independent of the instance x: samples from the same “Q” must be used for all inputs “x”. Thus it cannot be the case that q = x, where q $\\sim$ Q. \n\n---\n\n*“The ground truth of attribution methods constructed in experiments (in Figure 2) is too simple.”\n“(1) If a classifier is powerful enough...”\n“(2) Defining “4” as the foreground also seems problematic...”\n“This is a typical case when the input information for inference is redundant. When a small part of foreground objects are already enough for classification, it is difficult to annotate the ground-truth attention of a DNN.”*\n\nGood observation! For real-world datasets, a precise ground truth signal is fundamentally unknown, which is why we rely on semi-synthetic datasets where at least approximate information about the ground truth signal can be known. For hard-MNIST, while we also expected the classifier not to use all pixels of the digit “4” as the signal, VerT in Figure 2 seems to extract something very close – indicating that most of the pixels of “4” are indeed useful for classifying the digit. The purpose of our IOU experiments is to ensure that unimportant pixels (i.e. background pixels that are definitely not informative to the task) are not given more importance than pixels that may or may not be important (i.e. pixels that lie within the signal). All other baselines methods heavily attribute background pixels, whereas ours does not. We hope this addresses your question, and we are happy to clarify otherwise.\n\n---\n\n*“(3) The correlation between dark hair and glass in CelebA dataset is just assumed...\"*\n\nThis is incorrect, we do not assume the correlation, rather we explicitly designed our dataset such that dark hair and glasses were correlated. All images of people with dark hair were chosen to have glasses, and all images with blonde or white hair do not have glasses. We further test that the model relies on this correlation by testing performance on dark haired people without glasses and blonde and white haired people with glasses, noting that accuracy drops from 97% to 38% (near guessing). We will clarify this in the paper.\n\n---\n\n*“Pixel perturbation tests are circular arguments...”*\n\nWe don’t completely agree – the Q we use to train the model is different from that used to perform the pixel perturbation method. But your observation that our method performs the best because it is robust to masking is correct, and it is precisely the point of our approach! Please see global comments (G1-G4). We think that this is an advantage of our method, as pixel perturbation is a commonly used test and aligns well with our intuitions for what feature attributions must satisfy. We present this evaluation as a sanity check with respect to known metrics for feature attribution.\n\n---\n\n*“Only the accuracy is not enough to evaluate the faithfulness...”*\n\nOur core argument (G1) is that the feature perturbation strategy is critical. Typical implementations of Shapley values provide attributions of black-box models using an arbitrary feature perturbation strategy, whereas we propose to align the feature perturbation method with the model’s robustness to that perturbation method. We present a comparison with SHAP in (G5). We also present an extensive set of evaluations: pixel perturbation (Figure 3), IOU with an approximate ground truth (Table 1), robustness to explanation manipulation (Figure 4) and sensitivity to hyper-parameters (Figure 5). Is there a specific evaluation metric you had in mind that is conceptually different from these? If so, we would be happy to consider it. \n\n---\n\n*“Authors just compare the proposed method with gradient-based explanation method...”*\n\nPlease refer to (G2) for an explanation of our method and how it differs from classical feature attribution methods such as Shapley values, DeepLIFT, etc. Essentially, our method is orthogonal to these works. While most feature attribution methods attempt to perform feature attribution of black-box models, VerT aligns the model to the explanation method (via Q) used. We nonetheless present a comparison to SHAP in (G5) as requested. \n\n---\n\n*“Why “QFA applied to a model from a Q-verifiable model class” can be considered as “a verifiable feature attribution?” Authors do not provide proofs to support this claim. Moreover, what is the definition of “a verifiable feature attribution?” Please clarify how a feature attribution can be considered as “verifiable.””*\n\nPlease refer to (G1-G4) for clarification. We apologize for any confusion! That statement defines “verifiable feature attribution” in our paper, i.e., QFA applied to a model from a Q-verifiable model class. If you think this phrase is confusing, we are happy to consider renaming it.\n"}, {"review_id": "wcalEjpNrM", "reviewer": "Reviewer_qGoL", "summary": "The present work introduces a theoretical framework of verifiability of feature attributions based on the sparest (binary) feature attribution mask that only barely changes the models' output. The authors theoretically (and empirically) show that for signal-distractor decomposable datasets off-the-shelf black-box models cannot be verified according to their definition of verifiability. The main reason being that off-the-shelf models cannot handle the OOD samples created by the masking intervention (\"feature replacement\"). To overcome this, they propose a finetuning scheme, in which they make off-the-shelf black-box models robust to such feature interventions. Specifically, they alternatingly optimize for the sparsest mask (that only barely changes the models' output) and apply a distillation loss. While the former makes the model robust to the feature interventions, the latter ensures the similarity to the original model. Experimentally, they show that VerT improves interpretability over gradient-based methods. VerT robustly identifies the signals (most salient features) of the inputs, while retaining the performance of the original model.", "questions": "- In Fig. 2, do all methods use the same model (finetuned or not finetuned version)?\n- Is there any explanation for the large difference for the non-finetuned and finetuned models using VerT in Fig. 4? Is the adversarial training objective also used during finetuning? More generally, would finetuning without model distillation (Eq. 2) and data distillation in Eq. 1 lead to more robust (and interpretable) models that only focus on the important parts of the image?\n- How did the authors set $\\lambda_1, \\lambda_2$?\n\n**Suggestions**\n- [B-Cos networks](https://openaccess.thecvf.com/content/CVPR2022/html/Bohle_B-Cos_Networks_Alignment_Is_All_We_Need_for_Interpretability_CVPR_2022_paper.html) or [BagNets](https://openreview.net/forum?id=SkfMWhAqYQ) could also be mentioned as inherently interpretable models.\n- The abbreviations in L68 could be introduced.\n- There are several recent advancements of concept-based models that close the performance gap, e.g., [post-hoc CBMs](https://openreview.net/forum?id=HAMeOIRD_g9) that could be mentioned in the related work.\n- Theorem 2 in the supplemental should have the same number as Theorem 1 in the main text as well as formulation to make it easier for readers to find.\n- In the experiments section there are several missing “Table/Figure {num}”.\n- The hyperparameters are neither described in the main text nor supplement.\n- To which Figure/Table are L288-299 referring to?\n- There are several typos throughout the present submission.\n- Results for Celeb in Table 3 in the supplement are missing.", "limitations": "The limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The theoretical verifiability framework is simple yet sound.\n- The theoretical analysis (Sec 3.2) is interesting and supported both theoretically as well as empirically.\n- The finetuning scheme is simple yet effective. It does not change the models' output while simultaneously enhancing its interpretability by making it robust to the input feature removal.\n- The paper is clearly written (except for some small issues; see questions & suggestions below) and easy to follow.\n- Code is provided.", "weaknesses": "- The major weakness of the present work is that it does not compare to any removal-based feature attribution methods (e.g., [SHAP](https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html), [FastShap](https://openreview.net/forum?id=Zq2G_VTV53T), etc.), models also trained on feature attributions (e.g., [right for the right reasons](https://www.ijcai.org/proceedings/2017/371)), nor inherently interpretable models (e.g., [JAM](https://arxiv.org/abs/2103.01890) or [B-Cos](https://openaccess.thecvf.com/content/CVPR2022/html/Bohle_B-Cos_Networks_Alignment_Is_All_We_Need_for_Interpretability_CVPR_2022_paper.html)); only comparisons to gradient-based methods are provided.\n- The experiments always include manually-defined spurious correlations (to contain clear signals and distractors). While this provides empirical evidence for their theoretical framework, it would be meaningful to compare the method on “untouched”, more challenging datasets besides Celeb, e.g., ImageNet.\n- The finetuning scheme may change the model’s behavior (but not its output due to Eq. 2). While Tab. 2 shows that the prediction stays similar, it may change what type of signals in the inputs the classifier considers for its prediction (and its inner workings). Consequently, the finetuned model may not be faithful to which signal it uses (or how it processes it) for its prediction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review!\n\n*“The major weakness of the present work is that it does not compare to any removal-based feature attribution methods (e.g., SHAP, FastShap, etc.), models also trained on feature attributions (e.g., right for the right reasons), nor inherently interpretable models (e.g., JAM or B-Cos); only comparisons to gradient-based methods are provided.”*\n\nPlease refer to our comment (G2) for discussion on differences with classic feature attributions (like SHAP). Nonetheless, we present comparisons with SHAP in comment (G5).\n\nOur paper differs from the paper “Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations” (Ross et al., AAAI 2017) in that while their paper assumes (partial) knowledge of the ground truth feature attributions, our setup does not assume any such knowledge, thus making this incompatible with our setup. \n\nWe have already presented a comparison with JAM in Table 3 in the Supplementary material, where JAMs are “f_b + input (pixel) dropout”, which is precisely the method proposed by JAMs to train interpretable models. We shall change that column to read “JAMs” instead. \n\nWe have added B-CosNetsv2 (Bohle et al., 2023, \"B-cos Alignment for Inherently Interpretable CNNs and Vision Transformers\") as a baseline, with results shown in (G5). We find that they perform comparably for the IOU tests and qualitatively, however VerT still significantly outperforms for pixel perturbation and has the advantage of being applicable post-hoc. Thus, VerT models are verifiable, meaning that any attribution map has a precise meaning in terms of concrete perturbations made to the inputs via definition 1. On the other hand, B-CosNet models are not verifiable, meaning that perturbing non-important pixels leads to output change, thus making the attribution misleading wrt model behaviour, which is precisely what pixel perturbation shows.\n\n---\n\n*“The experiments always include manually-defined spurious correlations (to contain clear signals and distractors). While this provides empirical evidence for their theoretical framework, it would be meaningful to compare the method on “untouched”, more challenging datasets besides Celeb, e.g., ImageNet.”*\n\nFor natural datasets such as ImageNet, a popular evaluation for explanations includes the pixel perturbation test, which we already perform for our existing datasets. Note that this evaluation does not require the use of a ground truth feature attribution. \n\nImageNet experiments also require considerably more computational resources and engineering to train models and store masks for every single training data point, and hence we are unable to do these during the rebuttal. However, we shall prioritize this experiment for a future version of this draft.\n\n---\n\n*“The finetuning scheme may change the model’s behavior (but not its output due to Eq. 2). While Tab. 2 shows that the prediction stays similar, it may change what type of signals in the inputs the classifier considers for its prediction (and its inner workings). Consequently, the finetuned model may not be faithful to which signal it uses (or how it processes it) for its prediction.”*\n\nWhile this is an intriguing hypothesis, note that we are unable to either confirm or deny this, as it is fundamentally unclear what features are used by black-box models! This motivates our framework which emphasizes verifiability. \n\nFurther, we also think any such change is irrelevant – since the Q-verifiable models are equivalent to the original, why not use the verifiable models instead, since they come with the additional benefit of interpretability?\n\n---\n\n*“In Fig. 2, do all methods use the same model (finetuned or not finetuned version)?”*\n\nIn Fig. 2, row 3 (VerT ours) uses the VerT model $f_v$ finetuned via our method to explain $f_b$. All other rows use the model, $f_b$, which is not finetuned via our objective. Note that $f_v$ and $f_b$ are functionally equivalent. \n\n---\n\n*“Is there any explanation for the large difference for the non-finetuned and finetuned models using VerT in Fig. 4? Is the adversarial training objective also used during finetuning? More generally, would finetuning without model distillation (Eq. 2) and data distillation in Eq. 1 lead to more robust (and interpretable) models that only focus on the important parts of the image?”*\n\nPlease refer to global comment (G1) for a clarification about VerT and connection to robustness. We do not use adversarial training for VerT. The objective of VerT training (model + data distillation) is precisely to make the models more mask-robust and hence interpretable. We are unsure if this answers your question, please let us know in case of any clarification.\n\n---\n\n*\"How did the authors set lambda_1, lambda_2?\"*\n\nFor all experiments, we set both $\\lambda_1$ and $\\lambda_2$ to 1. We will be sure to add this to our implementation details. \n\nThank you for your suggestions regarding the writing! We will be sure to address these in the final draft. \n"}, {"review_id": "roBAZUhQ5z", "reviewer": "Reviewer_dxrg", "summary": "This paper introduces VerT, a method to distill *verifiable* models from black-box models. Concretely, These verifiable models are distilled by fitting a model $f_v$ that reproduces the predictions of the black-box model $f_b$ on a training set. Unlike the black-box model, the model $f_v$ is made verifiable by making its predictions self-consistent if a mask isolating the signal is applied to the input data. This mask is fitted by adding an extra objective to the model distillation objective, which guarantees that the mask is sparse and that the predictions of $f_v$ is similar for both the masked and the unmasked input. The methods is evaluated on 3 image datasets, including 2 where ground-truth feature importance is known. This analysis demonstrates that VerT outperforms standard gradient-based feature importance methods. ", "questions": "All my questions and recommendations are contained in the weakness section. I will not repeat them here to avoid redundancies. I will consider improving my recommendation if the above weaknesses are addressed by the authors.  \n", "limitations": "Some limitations of the work are discussed in Section 6 of the main paper. I do not believe that negative societal impacts are a real concern for this work. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Good writing.** The paper was really easy to read and follow. The notations are on-point and intuitive. \n\n**Solid empirical validation.** The experiments provided in Section 5 are convincing and thoroughly verify important claims: the features identified as salient by VerT have the strongest impact when masked (which is not surprising given the optimization objective underlying VerT), these features have a reasonable overlap with the ground-truth salient features in a setting where the later are known and VerT shows improved robustness. The authors also show that VerT improves the robustness to adversarial attacks on the explanations  with respect to gradient-based feature importance methods (which is unsurprising as the attacks are designed to fool gradient-based methods specifically). Insightful illustrative examples to understand the gains of VerT are provided in Figure 2.", "weaknesses": "**Restrictive assumption on replacement distribution.** In Lines 125-137, the authors discuss the importance of the choice of the replacement distribution $\\mathcal{Q}$ in order to avoid creating OOD examples by masking. It should be mentioned that some important masking stategies, such as [Gaussian blurs](https://arxiv.org/abs/1704.03296), are omitted from this discussion as these are conditioned on the input image $\\mathbf{x}$. I am wondering if it is even possible to define masking strategies with replacement distributions *independent* of the input $\\mathbf{x}$ that do not create OOD examples. Intuitivelly, it is legitimate to expect the replacement input $\\mathbf{q}$ to have (at least) some information about $\\mathbf{x}$ to avoid replacing the features of $\\mathbf{x}$ by OOD values. I would recommend the author to discuss this point thoroughly and (possibly) acknowledge this as a limitation of the work.\n\n**Unrealistic signal-distractor decomposition.** The signal-distractor decomposition defined in Definition 3 is key for Theorem 1. It appears to me that this decomposition is unrealistic for a simple reason: in the underlying DGP, the mask $\\mathbf{m}$ and the signal $\\mathbf{s}$ are independent. To make this point more clear, I would like to consider the example discussed in the paper. If we consider a cow detection task, the signal $\\mathbf{s}$  would typically correspond to the cow-part of the image. If that is the case, the mask $\\mathbf{m}$ should depend on the position of the cow on the image (e.g. performing a translation on the signal $\\mathbf{s}$ should result in a similar translation on the mask $\\mathbf{m}$ in principle). Again, I would recommend the authors to comment on the realism of their assumptions.  \n\n\n\n### Minor Weaknesses \n\n- Algorithm1: what is $M(\\mathbf{x})$? Is this the same as $\\mathbf{m}(\\mathbf{x})$? Also, should there be a minus sign in front of the gradients if the objectives are minimized? \n- In the appendices, Theorem 2 corresponds to Theorem 1 the main paper. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review!\n\n\n*“Restrictive assumption on replacement distribution. In Lines 125-137, the authors discuss the importance of the choice of the replacement distribution Q in order to avoid creating OOD examples by masking. It should be mentioned that some important masking stategies, such as Gaussian blurs, are omitted from this discussion as these are conditioned on the input image x. I am wondering if it is even possible to define masking strategies with replacement distributions independent of the input x that do not create OOD examples. Intuitivelly, it is legitimate to expect the replacement input q to have (at least) some information about x to avoid replacing features of x by OOD values. I would recommend the author to discuss this point thoroughly and (possibly) acknowledge this as a limitation of the work.“*\n\nWe apologize for the confusion. Please take a look at our comment (G1) regarding this issue. While it is true that any such masking creates OOD examples, the critical issue here is that we can create models that are robust to such replacement of the unimportant features. In other words, we would like to modify models (via VerT) to be not sensitive to such OOD examples, where pixels are replaced with $Q$ in the non-important features. By predefining our feature replacement distribution $Q$, we can modify our models to be robust to these perturbed OOD examples while still maintaining that the replacement distribution is independent of the input $x$. We will clarify this in the paper.\n\nWe do not discuss gaussian blur because in this case the “replacement distribution” $Q$ is dependent on $x$, however our framework requires them to be independent. For replacement distributions that are not independent of $x$, we are unable to disentangle the effect that the replacement distribution has on the model from the true signal in $x$, i.e., blurring does not completely remove the information in the image, but masking out the pixels does.\n\n---\n\n*“Unrealistic signal-distractor decomposition. The signal-distractor decomposition defined in Definition 3 is key for Theorem 1. It appears to me that this decomposition is unrealistic for a simple reason: in the underlying DGP, the mask m and the signal s are independent. To make this point more clear, I would like to consider the example discussed in the paper. If we consider a cow detection task, the signal s would typically correspond to the cow-part of the image. If that is the case, the mask  m should depend on the position of the cow on the image (e.g. performing a translation on the signal  s should result in a similar translation on the mask m  in principle). Again, I would recommend the authors to comment on the realism of their assumptions.”*\n\nGreat observation! This is a very subtle point about the signal-distractor decomposition. Note that definition 3 has redundant information: the mask being independently sampled from a distribution $m \\sim \\mathcal{M}$ implies $p(y | x) = p (y | s) = p(y | s \\odot m)$ (because the distractor component $d \\odot (1- m)$ is necessarily independent of the label), which is listed as an additional condition. Hence we can eliminate the $m \\sim M$ statement and instead *define* mask $m$ to be a binary mask such that condition (1) $p(y | x) = p(y | s \\odot m)$  and condition (2), that the mask $m$ is minimal, holds. \n\nFrom your example, this ensures that mask $m$ now depends on signal $s$, and a translation of the cow in the image results in a corresponding translation of the mask to satisfy conditions (1,2). This amendment to our definition does not impact Theorem 1 or the rest of our theory, and we hope this makes the resulting graphical model more realistic. We again thank the reviewer for noticing such a subtle issue.\n"}]

Author rebuttal:
We would like to thank all the reviewers for their constructive feedback. We are glad that reviewers found our theory “simple yet sound” (reviewer qGoL) and that our paper had “solid empirical evaluation” (reviewer dxrg). However, we feel that there were also some misunderstandings with respect to our method, and we aim to clarify these here.

**(G1) The core idea of the paper:** We’d like to perform feature attribution of black-box model f using a **masking-based procedure** (i.e., QFA in definition 1, with some input-independent choice of Q). However, this is problematic, as the **underlying black-box model may not be robust to perturbations** introduced by our procedure, leading to large change in model outputs upon masking. Ideally, one needs to find a distribution Q that the model is maximally robust to (if that even exists) and use that choice of Q for QFA to perform attribution. Another solution, which we propose, is to assume a choice of Q, and **fine-tune the black-box model to be robust to masking made with this Q** (also called the Q-verifiable model in definition 2) in the non-important features. 

In particular, Q-verifiability can be thought of as a very specific form of robustness, where models are ONLY robust wrt the distractor in the input and NOT the signal, thus distinguishing it from classical robustness. Further, verifiability requires robustness wrt masking, as opposed to that additive noise, as is usual with the robustness literature. However, note that we do not know the signal and distractor apriori, which is why we use alternating minimization in VerT to alternately estimate the signal-distractor masks; and the resulting Q-verifiable models. We shall emphasize this point of view of verifiability as “robustness to masking on the distractors”, in the main paper. 

**(G2) How is this paper different from prior works on feature attribution?** Classical feature attribution methods (LIME, SHAP, smoothgrad, etc) work by attributing to black-box models. As we mention above, these may lead to erroneous results when the model is non-robust to the perturbations introduced by these methods. On the other hand, our method (VerT + QFA) involves adapting the model to the attribution method, making this fundamentally distinct from classic feature attribution methods. Thus our framework contains aspects of both inherently interpretable models and post-hoc explanations. 

Another fundamental contribution of this work lies in its conceptual framework. Classical feature attribution literature does not have a precise notion of “ground truth”, making it fundamentally unclear what quantities these are estimating in the first place. In this paper, **we introduce one notion of ground truth – the signal-distractor decomposition** that is a property of the dataset itself (and NOT the model!). Using this notion, we are able to verify that our procedures work as intended at least in settings where the ground truth is known, and when the model correctly identifies the entire signal component (i.e., for well-performing models). 


**(G3) Why is theorem 1 interesting?** Theorem 1 states that QFA (with some choice of Q) used with an appropriate Q-verifiable model (using the same Q distribution) is able to recover the underlying signal-distractor decompositions for optimal models on a dataset. This is interesting because this works with any choice of Q! This should intuitively make sense: if a model is able to be robust to masking with some distribution Q, then we can use this fact to do feature attribution, for any chosen Q.

**(G4) Does VerT only work in limited settings?** Not at all! Theorem 1 shows that when we are able to define the ground truth, i.e., the signal-distractor decomposition, this is recovered by VerT. This does not imply that VerT does not apply in other settings. In other settings where the signal-distractor distribution is unknown we only cannot evaluate the correctness of the QFA mask w.r.t. ground truth, but the theorem 1 guarantees correctness for optimal models. Non-optimal models may not identify the correct signal component (hence their non-optimality), making it such that the signal cannot be theoretically extracted from such models. However, in practice if a model is close-to-optimal, then it is reasonable to assume that it identifies close-to-correct signal components, and this is exactly what our experiments demonstrate. VerT models recover masks that are close to the ground truth, and our method (QFA) is able to identify this.

We aim to add discussions in the paper clarifying points (G1-G4).

**(G5) Additional Experiments:** Please refer to attached pdf.
As requested by reviewers QRQ5, 2ozo, and qGoL, we add SHAP as a baseline. We note that in general SHAP does not often perform as well as most gradient-based methods for image data. We find that it performs similarly to random attributions.

We also add B-CosNets as an inherently-interpretable model baseline, as suggested by reviewer qGoL. We find that B-CosNets performs comparably to our method for the IOU test on Hard MNIST. We were unable to train it to convergence on the Chest X-ray dataset. We also find that visualizations created by B-CosNets align with our expectations and appear visually interpretable. However, our method still significantly outperforms B-CosNets on the pixel perturbation test, showing that B-CosNets are not robust to perturbations of the distractor (i.e. are not verifiable). We also note that our method can be applied to a trained black-box model of any architecture and training procedure, whereas B-CosNets, like all inherently interpretable models, cannot.

We finally perform the ablation requested by reviewer QRQ5 to test the effect that the choice of Q has empirically. We consider various parameterizations of the normal distribution used for Q, and find that results are relatively consistent across the different choices of Q. 

Meta-review: The authors propose a framework for verifiable feature attribution, which, according to the authors, fixes some issues with previous definitions of feature attribution. In particular, the proposed definition depends on a counterfactual distribution that is used to replace features of interest, and quantify how the output changes. The paper demonstrates how verifiable feature attribution cannot be achieved in black box models. The authors propose a new method that transforms a black box model into one where feature attribution can be verified. The authors run experiments demonstrating the success of this method on synthetic and real world tasks. 

The authors study an important problem. This publication may spur further research in the direction of converting a black box model to an interpretable one. Drawbacks identified by the reviewers include: some assumptions about the data seem unrealistic; the method is developed to optimize a particular criteria and then it is compared against other methods that do not optimize for this criteria, thus resulting in a somewhat circular comparison scheme; as a result, the evaluation does not seem fair or appropriate; the algorithm cannot yet scale to larger models and datasets; there is some disagreement between what the authors claim what VerT does (the model, after "conversion" is equivalent to the original model in function space) and some empirical results showing that VerT produces models that are more robust. I would suggest discussing some of these drawbacks among limitations, and, when possible, address them directly when updating the paper.
Decision: Accept (poster)