SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Usha Bhalla*
usha_bhalla@g.harvard.edu
Suraj Srinivas*
ssrinivas@seas.harvard.edu
Himabindu Lakkaraju
hlakkaraju@hbs.edu
With the increased deployment of machine learning models in various real-world applications, researchers and practitioners alike have emphasized the need for explanations of model behaviour. To this end, two broad strategies have been outlined in prior literature to explain models. Post hoc explanation methods explain the behaviour of complex black-box models by identifying features critical to model predictions; however, prior work has shown that these explanations may not be faithful, in that they incorrectly attribute high importance to features that are unimportant or non-discriminative for the underlying task. Inherently interpretable models, on the other hand, circumvent these issues by explicitly encoding explanations into model architecture, meaning their explanations are naturally faithful, but they often exhibit poor predictive performance due to their limited expressive power. In this work, we identify a key reason for the lack of faithfulness of feature attributions: the lack of robustness of the underlying black-box models, especially to the erasure of unimportant distractor features in the input. To address this issue, we propose Distractor Erasure Tuning (DiET), a method that adapts black-box models to be robust to distractor erasure, thus providing discriminative and faithful attributions. This strategy naturally combines the ease of use of post hoc explanations with the faithfulness of inherently interpretable models. We perform extensive experiments on semi-synthetic and real-world datasets, and show that DiET produces models that (1) closely approximate the original black-box models they are intended to explain, and (2) yield explanations that match approximate ground truths available by construction. Our code is made public here.
An important desideratum of machine learning models is for their predictions to be explainable. This allows both human domain experts as well as laypeople to better understand and trust the decisions made by models, and furthermore, is also a regulatory requirement for high-stakes settings. For example, both the European General Data Protection Regulation (GDPR) [1] and the US AI Bill of Rights [2], require organizations to provide explanations for decisions made in high-stakes settings. A common approach to producing such explanations from black-box models in a post hoc manner is via feature attribution, which aims to identify important input features influencing a model prediction. These methods typically work by locally approximating non-linear models with linear ones [3] under some input perturbations such as feature erasure. Intuitively, if the underlying model is more sensitive
to the erasure of feature A than feature B, these methods aim to attribute a higher "importance" to feature A than feature B. A fundamental prerequisite for a feature to be considered important by a model is that it must first be useful in predicting the label, that is, it must be discriminative for the task. If a feature does not contain information relating to the output label, then it cannot be used to predict the label, and thus feature attribution methods must not consider them important. However, recent works [4, 5] have found this not to be case - feature attribution methods often highlight non-discriminative features. This motivates a natural question: what causes feature attributions to highlight such non-discriminative features, making them unfaithful?
Figure 1: Illustration of our method, Distractor Erasure Tuning. DiET models exhibit robustness to distractor erasure (non-discriminative features such as backgrounds), allowing for the recovery of discriminative attributions.
Answering this question has been hard because of a lack of theoretical understanding of the faithfulness of feature attributions. While these notions have been used empirically [4, 5] to assess the quality of attributions, the theoretical characterization of optimally faithful feature attributions is missing in the literature. In this work, we tackle this problem by proposing a framework for feature attribution methods emphasizing faithfulness and particularly discriminability, formalized via the signal-distractor decomposition for datasets. Essentially, the signal denotes the discriminative parts of the input (relative to a given task), while the distractor denotes the unimportant parts. Feature attribution methods are then evaluated on how well they are able to recover the signal, thus also providing a well-defined notion of a "ground-truth". We theoretically identify an important criterion to recover this ground-truth, that being the robustness of the model to the erasure of the input distractors. To enable black-box models to recover such ground truth attributions, we propose Distractor Erasure Tuning (DiET), a method that adapts models to be robust to the erasure of input distractors. Given that these distractor regions are not known in advance, our method works by alternating feature attribution and model adaptation. At a high level, our work still uses feature attribution methods while adapting black-box models to have faithful attributions. Thus, this strategy naturally combines the ease of use of post hoc explanation methods with the faithfulness benefits of inherently interpretable models by providing the best of both alternatives. Our contributions are the following:
1. We present a formalism for feature attribution that emphasizes discriminability and allows for a notion of well-defined ground truth, via the signal-distractor decomposition of a dataset. We show that it is necessary for models to be robust to distractor erasure for them to be able to recover this ground truth.
2. We propose distractor erasure tuning (DiET), a novel method that adapts black-box models to make them robust to distractor erasure.
3. We perform experiments on semi-synthetic and real-world computer vision datasets and show that DiET outputs models that are faithful to its original input models, and that their feature attributions are interpretable and match ground-truth feature attributions.
Post-Hoc Explainability. Post-hoc explainability methods aim to explain the outputs of fully trained black-box models either on an instance-level or global level. The most common post-hoc methods are feature attribution methods that rank the relative importance of features, either by explicitly producing perturbations [6, 7], or by computing variations of input gradients [8-10]. Perturbation-based methods are especially popular in computer vision literature [11-14], which use feature removal strategies adapted specifically for image data. However, these methods all assume a specific form for feature removal, and we show theoretically in Section 3 that this can lead to unverifiable attributions.
Inherently Interpretable Models. Inherently interpretable models are constructed such that we know exactly what they do, either through their weights or explicit modular reasoning. As such, the explanations provided by these models are more accurate than those given by post-hoc methods; however, the performance of interpretable models often suffers when compared to unconstrained black-box architectures. The most common inherently interpretable model classes include linear models, decision trees and rules with limited depth, GLMs, GAMs [15], JAMs [16-18], prototype-and concept-based models [19, 20], and weight-input aligned models [21]. While [19, 20] leverage the expressivity of deep networks, they constrain hypothesis classes significantly and still often suffer from a decrease in performance. Among these, our work most closely relates to JAMs, which amortise feature attribution generation using a learnt masking function to generate attributions in a single forward pass, and trains black-box models using input dropout. On other hand, JAMs (1) trains models from scratch, whereas DiET can interpret black-box models, (2) amortises feature attributions using a masking function resulting in less accurate attributions, (3) trains models to be robust to a large set of candidate masks via input dropout, leading to low predictive accuracy, whereas DiET trains models only to be robust to the optimal mask, leading to more flexibility and higher predictive accuracy.
Evaluating Correctness of Explanations. As explainability methods grow in number, so does the need for rigorous evaluation of each method. Research has shown that humans naively trust explanations regardless of their "correctness" [22], especially when explanations confirm biases or look visually appealing. Common approaches to evaluate explanation correctness rely on feature / pixel perturbation [23, 9, 24], i.e., an explanation is correct if perturbing unimportant feature results in no change of model outputs, whereas perturbing important features results in large model output change. Hooker et al. [5] proposed remove-and-retrain (ROAR) for evaluating feature attribution methods by training surrogate models on subsets of features denoted un/important by an attribution method and found that most gradient-based methods are no better than random. While prior works focused on developing metrics to evaluate correctness of explanations, our method DiET produces models that have explanations that are accurate by design, according to pixel-perturbation methods.
3 Theory of Discriminative Feature Attributions
In this section, we provide a theoretical framework for feature attribution, including a well-defined ground truth. We start by identifying a common feature of feature attributions, their reliance on perturbations or erasure. Intuitively, feature attribution methods work by simulating removing certain features and estimating how the model behaves when those features are removed: removing unimportant features should not change model behaviour. Typically, this erasure is implemented by replacing features with scalar values, such as the mean of the dataset [11, 12]. However, this can result in out-of-distribution inputs that can confuse a classifier, thus making it difficult to create meaningful attributions.
To ground this argument in an example, consider a model that classifies cows and camels. For an image of a camel, a feature attribution might note that only the hump of the camel and the sand it stands on are important for classification. As such, we would expect that the sky was irrelevant to the classifier's prediction, and we can concretely test this by altering it and creating a counterfactual sample. For example, we could mask the sky with an arbitrary uniform color; however, this may result in the sample being out-of-distribution for the model, and its prediction may change drastically even if the sky was not important for prediction. There are two strategies to overcome this problem. The first solution involves masking the sky in a manner that preserves the naturalness of the image,
but this solution involves using large-scale generative models, which themselves can contain biases and be uninterpretable. The second solution requires the classifier to be invariant to the erasure of the pixels corresponding to the sky, which is our solution in this paper. We formalize this argument below by defining an erasure-based feature attribution method called  $(\epsilon, \mathcal{Q})$ -feature attribution.
Notation. Throughout this paper, we shall assume the task of classification with inputs  $\mathbf{x} \sim \mathcal{X}$  with  $\mathbf{x} \in \mathbb{R}^d$  and  $y \in [1,2,\dots C]$  with  $C$ -classes. We consider the class of deep neural networks  $f: \mathbb{R}^d \to \triangle^C$  which map inputs  $\mathbf{x}$  onto a  $C$ -class probability simplex. This paper considers binary feature attributions, which are represented as binary masks  $\mathbf{m} \in \{0,1\}^d$ , where  $\mathbf{m}_i = 1$  indicates an important feature and  $\mathbf{m}_i = 0$  indicates an unimportant feature.
3.1 Feature Attributions with Input Erasure
We first define an erasure-based feature attribution such that the feature replacement method is explicit, and features are replaced with samples from a counterfactual distribution  $\mathcal{Q}$ . Particularly, we are interested in binary attributions (i.e., a feature is considered important or not) instead of real-valued ones.
Definition 1.  $(\epsilon, \mathcal{Q})$ -feature attribution (QFA) is a binary mask  $\mathbf{m}(f, \mathbf{x}, \mathcal{Q})$  that relies on a model  $f(\cdot)$ , an instance  $\mathbf{x}$ , and a  $d$ -dimensional counterfactual distribution  $\mathcal{Q}$ , and is given by
$$
\mathbf {m} (f, \mathbf {x}, \mathcal {Q}) = \arg \min  _ {\mathbf {m} ^ {\prime}} \| \mathbf {m} ^ {\prime} \| _ {0} s u c h t h a t \underset {q \sim \mathcal {Q}} {\mathbb {E}} \| f (\mathbf {x} _ {s} (\mathbf {m} ^ {\prime}, q)) - f (\mathbf {x}) \| _ {1} \leq \epsilon
$$
where  $\mathbf{x}_s(\mathbf{m},q) = \mathbf{m}\odot \mathbf{x} + (1 - \mathbf{m})\odot q$
Thus, an  $(\epsilon, \mathcal{Q})$ -feature attribution (henceforth,  $QFA$ ) refers to the sparsest mask that can be applied to an image such that the model's output remains approximately unchanged. QFA depends on the feature replacement distribution  $\mathcal{Q}$ , where  $\mathcal{Q}$  is independent of both  $\mathbf{x}$  and  $y$ . This generalizes the commonly used heuristics of replacing unimportant features with the dataset mean, in which case  $\mathcal{Q}$  is a Dirac delta distribution at the mean value. The choice of  $\mathcal{Q}$  is indeed critical, as an incorrect choice can hurt our ability to recover the correct attributions due to the resulting inputs being out-of-distribution and the classifier being sensitive to such changes. Specifically, an incorrect  $\mathcal{Q}$  can result in QFA being less sparse, as masking even a few features with the wrong  $\mathcal{Q}$  would likely cause large deviations in the model's outputs. As a result, given a model, we must aim to find the  $\mathcal{Q}$  that leads to the sparsest QFA masks. However, the problem of searching over  $\mathcal{Q}$  is complex, as it requires searching over the space of all  $d$ -dimensional distributions, and furthermore, if the underlying model is non-robust, there may not exist any  $\mathcal{Q}$  that leads to sparse attributions. To avoid this, we consider the inverse problem: given  $\mathcal{Q}$ , we find the class of models that have the sparsest QFAs w.r.t. that particular  $\mathcal{Q}$ . We call this the  $\mathcal{Q}$ -robust model class, which we define below:
Definition 2.  $\mathcal{Q}$ -robust model class  $\mathcal{F}_v(\mathcal{Q})$ : For some given distribution  $\mathcal{Q}$ , the class of models  $\mathcal{F}_v$  for which  $\mathcal{Q}$  has the sparsest QFA mask as opposed to any other  $\mathcal{Q}'$ , such that for all  $f \in \mathcal{F}_v(\mathcal{Q})$ ,
$$
\mathcal{Q} = \arg \min_{\mathcal{Q}^{\prime}}\underset {\mathbf{x}}{\mathbb{E}}\| \mathbf{m}(f,\mathbf{x},\mathcal{Q}^{\prime})\|_{0}
$$
Intuitively,  $\mathcal{Q}$ -robust models result in the sparsest QFA masks and can be thought of as being robust to the erasure of "irrelevant" input features. Recalling our example of the cows and camels, we would like models to be robust to the replacement of the pixels corresponding to the sky but not necessarily robust to pixels corresponding to the camel or the cow itself. This distinguishes it from classical robustness definitions, which require models to be robust to small perturbations (rather than erasure) at every feature uniformly. Thus  $\mathcal{Q}$ -robustness is equivalent to enforcing robustness to the erasure of distractor features, a notion that is central to this work. For the rest of this paper, we shall refer to QFA applied to a model from a  $\mathcal{Q}$ -robust model class as a "matched" feature attribution – the same  $\mathcal{Q}$  is used to both define the model class and the feature attribution.
3.2 Recovering the Signal-Distractor Decomposition
In the study of feature attribution, the 'ground truth' attributions are often unspecified. Here, we show that for datasets that are signal-distractor decomposable, formally defined below, there exists a ground truth attribution, and feature attributions for optimal verifiable models are able to recover it.
Intuitively, given an object classification dataset between cows and camels, the "signal" refers to the regions in the image that are discriminative, or correlated with the label, such as the cows or camels. The distractor refers to everything else, such as the background or sky. Note that if objects in the background are spuriously correlated with the label, i.e. sand or grass, those would be part of the signal, not the distractor. We first begin by formally defining the signal-distractor decomposition.
Definition 3. A labelled dataset  $D = \{(\mathbf{x},y)_{i = 1}^{N}\}$  has a signal-distractor decomposition defined by masks  $\mathbf{m}(\mathbf{x})\in \{0,1\} ^d$  for every input  $\mathbf{x}$ , where
1.  $\mathbf{x} \odot \mathbf{m}(\mathbf{x})$  is the discriminative signal, where  $p(y \mid \mathbf{x}) = p(y \mid \mathbf{x} \odot \mathbf{m}(\mathbf{x}))$
2.  $\mathbf{x} \odot (1 - \mathbf{m}(\mathbf{x}))$  is the non-discriminative  $distractor$ , where  $p(y \mid \mathbf{x} \odot (1 - \mathbf{m}(\mathbf{x}))) = p(y)$
3.  $\mathbf{m}(\mathbf{x})$  is the sparsest mask, i.e.,  $\mathbf{m}(\mathbf{x}) = \arg \min_{\mathbf{m}'(\mathbf{x})}\| \mathbf{m}'(\mathbf{x})\| _0$ , such that (1) and (2) are satisfied.
We propose to use the masks  $\mathbf{m}(\mathbf{x})$  implied by the signal-distractor decomposition as ground truth feature attributions. These are meaningful as they precisely highlight the discriminative components of the image and ignore the non-discriminative regions. Discriminability has previously been considered an important criterion to evaluate feature attributions [4, 5] however, we here take a step further and propose its usage as ground truth.
We observe first that the masks  $\mathbf{m}(\mathbf{x})$  of the signal-distractor decomposition always exist: setting  $\mathbf{m}(\mathbf{x})$  as an all-ones vector trivially satisfies conditions (1) and (2). When multiple masks exist, condition (3) requires us to choose the sparsest such mask  $\mathbf{m}(\mathbf{x})$ . Using the definitions provided, we show below an asymptotic argument stating that  $Q$ -robustness is a necessary condition to recover the optimal masks defined by the signal-distractor decomposition.
Remark: A dataset  $\mathcal{D}$  is said to have a "non-redundant signal" when the sparsest mask in condition (3) of the signal-distractor decomposition is equal to the sparsest mask when (1) alone is satisfied.
Theorem 1. QFA applied to  $\mathcal{Q}$ -robust models recover the ground-truth masks when applied to the Bayes optimal predictor  $f_v^* \in \mathcal{F}_v(\mathcal{Q})$ , for datasets  $\mathcal{D}$  with a non-redundant signal.
Proof Idea. We first note that the optimal  $\mathcal{Q}$  for QFA is equal to the ground truth distractor distribution, as this leads to the sparsest QFA. If a  $\mathcal{Q}$ -robust model aims to recover the sparsest masks, then its QFA mask must equal that obtained by setting  $\mathcal{Q}$  equal to the distractor. From the uniqueness argument in the definition of the signal-distractor decomposition, this is possible only when the optimal mask is recovered by QFA.
Corollary.  $QFA$  fails to recover the ground-truth masks when applied to predictors  $f \notin \mathcal{F}_v(\mathcal{Q})$ .
This follows from the fact that for any  $f \notin \mathcal{F}_v(\mathcal{Q})$ , there exists some other  $\mathcal{Q}'$  that results in a sparser mask, indicating that the ground truth masks are not recovered. Thus, this shows that feature attributions applied to the incorrect model class can be less effective - in this case, they fail to recover the ground truth masks. Further, the Bayes optimality is an important condition because it ensures that the resulting model is sensitive to all discriminative features in the input - sub-optimal models are sub-optimal precisely because they fail to capture the signal from all the discriminative components of the input, and this can interfere with such models being able to recover ground truth masks. In practice, if we expect our models to be highly performant, we can expect them to be sensitive to all discriminative parts of the input and thus approximately recover ground truth masks. Finally, in practice, we do not have access to the ground truth masks for natural datasets, as the discriminative and the non-discriminative regions are not known in advance. In order to use these notions of ground truth, it is thus vital to construct semi-synthetic datasets where the discriminative parts are known. Thus, one can use semi-synthetic datasets to validate a feature attribution approach and then apply it to gain insight into real datasets with unknown signal and distractor components.
To summarize, we have defined a feature attribution method with the feature removal process made explicit via the counterfactual distribution  $\mathcal{Q}$ . To minimize the sparsity of the attribution masks with a given  $\mathcal{Q}$ , we use models from the  $\mathcal{Q}$ -robust model class  $\mathcal{F}_v(\mathcal{Q})$ . Finally, we find that feature attributions derived from Bayes optimal models in the model class  $\mathcal{F}_v(\mathcal{Q})$  are able to recover the ground-truth masks and fail to do so otherwise.
4 DiET: Distractor Erasure Tuning
In the previous section we showed that given a  $\mathcal{Q}$ -robust model  $f_v \in \mathcal{F}_v(\mathcal{Q})$ , we are able to apply QFA to recover the ground truth masks. In this section, we shall discuss how to practically build such robust models, given a pre-defined counterfactual distribution  $\mathcal{Q}$  that defines the erasure method.
Relaxing QFA. We note that QFA as defined in definition 1 is difficult to optimize in its current form due to its use of  $\ell_0$  regularization and its constrained form. To alleviate this problem, we perform two relaxations: first, we relax the  $\ell_0$  objective into an  $\ell_1$  objective, and second, we convert the constrained objective to an unconstrained one by using a Lagrangian. The resulting objective function is given in equation 1. Assuming the model  $f_v$  is known to us, we can minimize this objective function to obtain  $(\epsilon, \mathcal{Q})$ -feature attributions for each point  $\mathbf{x} \in \mathcal{X}$ .
$$
\mathcal {L} _ {\mathrm {Q F A}} (\theta , \left\{\mathbf {m} (\mathbf {x}) \right\} _ {\mathbf {x} \in \mathcal {X}}) = \underset {\mathbf {x} \in \mathcal {X}} {\mathbb {E}} \left[ \underbrace {\| \mathbf {m} (\mathbf {x}) \| _ {1}} _ {\text {m a s k s p a r s i t y}} + \lambda_ {1} \underbrace {\| f _ {v} (\mathbf {x} ; \theta) - f _ {v} \left(\mathbf {x} _ {\mathrm {s}} (\mathbf {m} , q)\right) ; \theta) \| _ {1}} _ {\text {d a t a d i s t i l l a t i o n}} \right] \tag {1}
$$
Enforcing Model Robustness via Distillation. Assuming that the optimal masks denoting the signal-distractor decomposition are known w.r.t. every training data point (i.e.,  $\{\mathbf{m}(\mathbf{x})\}_{\mathbf{x}\in \mathcal{X}}$ ), one can project any black-box model into a  $\mathcal{Q}$ -robust model via distillation. Specifically, we can use equation 2 for this purpose, which contains (1) a data distillation term to enforce the  $\epsilon$  constraint in QFA, and (2) a model distillation term to enforce that the resulting model and original model are approximately equal. Accordingly, the black-box model  $f_{b}$  and our resulting model  $f_{v}$  both have the same model architecture, and we initialize  $f_{v} = f_{b}$ .
$$
\mathcal {L} _ {\text {t r a i n}} (\theta , \{\mathbf {m} (\mathbf {x}) \} _ {\mathbf {x} \in \mathcal {X}}) = \underset {\mathbf {x} \in \mathcal {X}} {\mathbb {E}} \left[ \underbrace {\| f _ {v} (\mathbf {x} ; \theta) - f _ {v} \left(\mathbf {x} _ {\mathrm {s}} (\mathbf {m} (\mathbf {x}) , q)\right) ; \theta) \| _ {1}} _ {\text {d a t a d i s t i l l a t i o n}} + \lambda_ {2} \underbrace {\| f _ {b} (\mathbf {x}) - f _ {v} (\mathbf {x} ; \theta) \| _ {1}} _ {\text {m o d e l d i s t i l l a t i o n}} \right] \tag {2}
$$