SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alternating Minimization between  $\theta$  and  $\mathbf{m}$ . We are interested in both of the above objectives: we would like to recover the optimal masks from the dataset, as well as use those masks to enforce  $(\epsilon, \mathcal{Q})$  constraints via distillation to yield our  $\mathcal{Q}$ -robust models. We can thus formulate the overall optimization problem as the sum of these terms, as shown in equation 3. Notice that both these objectives assume that either the optimal masks, or the robust model is known, and in practice, we know neither. A common strategy in cases that involve optimizing over multiple sets of variables is to employ alternating minimization [25], which involves repeatedly fixing one of the variables and optimizing the other. We handle the constrained objective on the mask variables via projection, i.e., using hard-thresholding / rounding to yield binary masks.
$$
\theta^ {*}, \left\{\mathbf {m} ^ {*} (\mathbf {x}) \right\} = \arg \min  _ {\theta , \mathbf {m}} \left(\mathcal {L} _ {\text {t r a i n}} (\theta , \left\{\mathbf {m} (\mathbf {x}) \right\}) + \mathcal {L} _ {\mathrm {Q F A}} (\theta , \left\{\mathbf {m} (\mathbf {x}) \right\})\right) \tag {3}
$$
such that  $\mathbf{m}(\mathbf{x})\in \{0,1\} ^d\quad \forall \mathbf{x}\in \mathcal{X}$
Iterative Mask Rounding with Increasing Sparsity. In practice, mask rounding makes gradient-based optimization unstable due to sudden jumps in the variables induced by rounding. This problem commonly arises when dealing with sparsity constraints. To alleviate this problem, use a heuristic that is common in the model pruning literature [26] called iterative pruning, which involves introducing a rounding schedule, where the sparsity of the mask is gradually increased during optimization steps. Inspired by this choice, we employ a similar strategy over our mask variables.
Practical Details. We implement these objectives as follows. First, we initialize the robust model to be the same as the original model,  $f_{v} = f_{b}$ , and the mask to be all ones,  $D_{s} = m\odot D_{d},m = 1$ . We then iteratively (1) simplify  $D_{s}$  by optimizing  $\mathcal{L}_{QFA}$  until  $\mathbf{m}$  converges, (2) round  $\mathbf{m}$  such that it is binary (i.e.  $D_{s}$  is a subset of features in  $D_{d}$  rather than a weighting of them), and (3) update  $f_{v}$  by minimizing  $\mathcal{L}_{train}$  such that  $D_{s}$  is equally as informative as  $D_{d}$  to  $f_{v}$  and  $f_{v}$  is functionally equivalent
to  $f_{b}$ . As per Definition 2, we replace masked pixels in  $D_{s}$  with a pre-determined counterfactual distribution  $\mathcal{Q}$ . This ensures that the given  $\mathcal{Q}$  is the optimal counterfactual distribution for  $f_{v}$ , meaning  $f_{v}$  comes from the  $\mathcal{Q}$ -robust model class  $\mathcal{F}_{v}(\mathcal{Q})$ . The pseudocode is given in Algorithm 1.
Algorithm 1 Distractor Erasure Tuning
Input: Dataset  $D_{d}\coloneqq (x,y)$  , model  $f_{b}$  , hyperparameter  $k$  rounding steps Hyperparameters:  $k$  rounding steps,  $u$  mask scaling factor,  $s(t)$  sparsity at step t  $\{\mathbf{m}(\mathbf{x})\}$  s.t.  $\mathbf{m}(\mathbf{x})\gets$  ones with shape  $\mathbb{R}^{d / u}$  Init mask  $\mathbf{m}(\mathbf{x})$  to ones  $f_{v}\leftarrow f_{b}$  Init robust model  $f_{v}$  to  $f_{b}$  for  $k$  rounding steps do while  $\mathcal{L}_{\mathrm{QFA}}$  not converged do  $\{\mathbf{m}(\mathbf{x})\} \gets \{\mathbf{m}(\mathbf{x})\} +\nabla_{\mathbf{m}}\mathcal{L}_{\mathrm{QFA}}$  end while m  $\leftarrow$  round(m,s(t))  $\forall \mathbf{m}\in \{\mathbf{m}(\mathbf{x})\}$  while  $\mathcal{L}_{\mathrm{train}}$  not converged do  $f_{v}\leftarrow f_{v} + \nabla_{\theta}\mathcal{L}_{\mathrm{train}}$  end while end for return  $\{\mathbf{m}(\mathbf{x})\} ,f_v$
Mask Scale. In order to encourage greater "human interpretability," we increase the pixel size of the masks by lowering their resolution. We do this by downscaling the masks before optimization. Concretely, we initialize the masks to be of size  $m_d = x_d / u$ , where  $x_d$  is the dimension of the image  $\mathbf{x}$  and  $u$  is the pixel size we wish to consider. We then upsample the mask to be of dimension  $x_d$  before applying it to  $\mathbf{x}$ . The more we downscale the mask by (i.e. the greater  $u$  is), the more interpretable and visually cohesive the distilled dataset  $\mathbf{x}_s$  is.
In this section, we present our empirical evaluation in detail. We consider various quantitative and qualitative metrics to evaluate the correctness of feature attributions given by DiET models as well as the faithfulness of DiET models to the models they are meant to explain. We also evaluate DiET models' ability to explain models manipulated to have arbitrary uninformative input gradients. Finally, we analyze the effect of the mask downscaling hyperparameter on attributions. Comparisons to additional baselines beyond those shown in Figure 2 are given in 8.
Datasets. Hard MNIST: The first is a harder variant of MNIST where the digit is randomly placed on a small subpatch of a colored background. Each sample also contains small distractor digits and random noise patches. For this dataset, we consider the signal to be all pixels contained within the large actual digit, and the distractor to be all pixels in the background, noise, and smaller digits. Chest X-ray: Second, we consider a semi-synthetic chest x-ray dataset for pneumonia classification [27]. To control exactly what information the model leverages such that we can create ground truth signal-distractor decompositions, we inject a spurious correlation into this dataset. We randomly place a small, barely perceptible noise patch on each image in the "normal" class. We confirm that the model relies only on the spurious signal during classification by testing the model's drop in performance when flipping the correlation (adding the noise patches to the "pneumonia" class) and seeing that the accuracy goes from  $100\%$  to  $0\%$ . As such, for this dataset, the signal is simply the noise patch and the distractor is the rest of the xray. CelebA: The last dataset is a subset of CelebA [28] for hair color classification with classes {dark hair, blonde hair, gray hair}. We correlate the dark hair class with glasses to allow for qualitative evaluation of each methods' ability to recover spurious correlations. This dataset does not have a ground truth signal distractor decomposition, as there are many unknown discriminative spurious correlations the model may rely upon.
Models. For all experiments, we use ImageNet pre-trained ResNet18s for our baseline models,  $f_{b}$ . All models achieve over 96% test accuracy. We train DiET models with  $Q \sim \mathbb{1}_{d*d} * \mathcal{N}(\mu(D_d), \sigma^2(D_d))$ , meaning that each image is masked with a uniform color drawn from a normal distribution around the mean color of the dataset. For all evaluation, we use  $Q \sim \mathbb{1}_{d*d} * \mathcal{N}(\mu(D_d), 0)$
Figure 2: Visualization of datasets and attribution methods considered in this work. Row 1: Raw data samples, Row 2: Ground truth attributions, Row 3: DiET attributions, Rows 4-7: Baseline methods.
(the dirac delta of the dataset mean) to ensure that masked samples are minimally out-of-distribution for the baseline models  $f_{b}$ .
5.1 Evaluating the Correctness of Feature Attributions
Pixel Perturbation Tests. We test the faithfulness of our explanations via the pixel perturbation variant proposed in [9, 5], where we mask the  $k$  least salient pixels as determined by any given attribution method and check for degradation in model performance with the mean of the dataset. This metric evaluates whether the  $k$  masked pixels were necessary for the model to make its prediction. As mentioned in previous works, masked samples come from a different distribution than the original samples, meaning poor performance after pixel perturbation can either be a product of the model's performance on the masks or the feature attribution scores being incorrect. To disentangle the correctness of the attributions from the robustness of the model, we perform pixel perturbation tests on ground truth feature attributions, with results reported in the appendix. Note that our method returns binary masks, but this metric requires continuous valued attributions to create rankings. As such, for this experiment we use the attributions created by our method before rounding.
Results are shown in Figure 3. We find that the attributions produced by DiET, used in conjunction with DiET models outperform all baselines. Furthermore, DiET attributions tested on the baseline model also generally perform better than gradient-based attributions.
Intersection Over Union. We further evaluate the correctness of our attributions by measuring their Intersection Over Union (IOU) with the "ground truth" attributions. We use the signal from the ground truth signal-distractor decomposition as described in 5 for the ground truth attributions. For each image, if the ground truth attribution is comprised of  $n$  pixels, we take the intersection over union of the top  $n$  pixels returned by the explanation method and the  $n$  ground truth pixels, meaning an IOU of 1 corresponds to a perfectly aligned/correct attribution. Results are shown in 1, where our method performs the best for both datasets. We report mean and standard deviation over the dataset for each method.
5.2 Evaluating the Faithfulness of DiET Models
To ensure that the DiET model  $(f_v)$  returned by our method is faithful the original model  $(f_b)$  it approximates, we test the accuracy of DiET models with respect to the predictions produced by the original model. Specifically, we take the predictions of the original model  $f_b$  to be the labels of the dataset. We evaluate DiET models on both the original dataset  $(f_v(D_d) \approx f_b(D_d))$  and the simplified dataset  $(f_v(D_s) \approx f_b(D_d))$ , with results shown in 2. We see that DiETmodels are highly faithful to the baseline models they approximate.
Figure 3: Pixel perturbation tests (higher is better) for MNIST (left), Chest X-ray (middle), and CelebA (right) datasets. DiET's recommended mask sparsity is shown as a vertical dashed line. We observe that DiET performs the best overall. Refer to Section 5.1 for details.
Table 1: Intersection Over Union Results
Hard MNIST	Chest X-ray
DiET (Ours)	0.461 ±0.08	0.821± 0.05
SmoothGrad	0.252±0.05	0.045 ±0.05
GradCAM	0.295 ±0.09	0.000 ±0.00
Input Grad	0.117 ±0.05	0.017±0.03
FullGrad	0.389 ±0.10	0.528 ±0.12
Table 2: Faithfulness of DiET to Original Model
Hard MNIST	Chest X-ray	CelebA
Faithfulness on Original Data	0.996	1.00	0.995
Faithfulness on Simplified Data	0.987	1.00	0.975
5.3 Qualitative Analysis of Simplified Datasets
We first explore how well DiET recovers signal-distractor decompositions. For CelebA, we see that all methods except for input gradients correctly recover the spurious correlation for the "dark hair/glasses" class, however only our method provides useful insights into the other two classes. We see that our method correctly identifies hair as the signal for the "blonde hair" class, whereas other methods simply look at the eyes, which are not discriminative. Furthermore, we see that for the "gray hair" class, our method picks up on hair, as well as initially unknown spurious correlations such as wrinkles and bowties. For Hard MNIST, we see that our method clearly isolates the signal and ignores the distractor. FullGrad and GradCAM suffer from a locality bias and tend to highlight the center of each digit. SmoothGrad and vanilla gradients are much noisier and highlight edges and many random background pixels. For the Chest X-ray dataset, we see that our method and FullGrad perfectly highlight the spurious signal. GradCAM again suffers from a centrality bias, and cannot highlight pixels on the edge. SmoothGrad and gradients appear mostly random to the human eye.
We also consider the visual quality of our attributions compared with the baselines (examples are shown in 2). We find that our method, FullGrad, and GradCAM appear the most interpretable, as opposed to SmoothGrad and vanilla gradients, because they consider features at the superpixel level rather than individual pixels. We also see that GradCam and FullGrad seem relatively class invariant, and tend to focus on the center of most images, rather than the discriminative features for each class, providing for less useful insights into the models and datasets.
5.4 Robustness to Adversarial Manipulation of Explanations
In this section, we highlight our method's robustness to adversarial explanation manipulation. To this end, we follow the manipulation proposed in [29], which adversarially manipulates gradient-based explanations. This is achieved by adding an extra term to the training objective that encourages input gradients to equal an arbitrary uninformative mask of pixels in the top left corner of each image. Note that model accuracy on the classification task is the same as training with only cross-entropy loss.
We repeat experiments for all evaluation metrics on these manipulated models, with pixel-perturbation shown below 6, and IOU, model faithfulness, and model robustness in the appendix. We see that gradient-based methods perform significantly worse on manipulated models; however, our method
remains relatively invariant. We also note while the models are only manipulated to have arbitrary input gradients, SmoothGrad and GradCAM are also heavily affected such that their attributions are entirely uninterpretable as well, as shown below.
Figure 4: Pixel perturbation test and example images for models trained with gradient manipulation on Hard MNIST. Results for other datasets are in the appendix. Refer to Section 5.4 for details.
5.5 Attribution Sensitivity to Hyperparamaters
We conduct an ablation study on the choice of how much to downscale the mask by. The less we downscale by, the more fine-grained the mask is, allowing for optimization over a larger set of masks. However, the more we downscale by, the visually cohesive or "interpretable" to humans the masks are. We evaluate the trade-off between these two via pixel perturbation tests over multiple downscaling factors and with qualitative evaluations of the final masks in 5. We see that a downscaling factor of 8 performs the best on pixel perturbation tests. Increased factors of downscaling impose a greater locality constraint that results in informative pixels being masked, as shown in the visualization.
Figure 5: Ablation study of pixel perturbation test with varying levels of mask downscaling for MNIST and CelebA. Example images for each factor are shown on the right.
In this paper, we seek to build machine learning models such that their feature attributions remain discriminative. In particular, we propose DiET, a method that adapts black-box models into those that are robust to distractor erasure. We empirically evaluate DiET and find that the resulting models are highly faithful to the original and produce interpretable attributions that closely match the ground truth ones. Limitations of DiET include requiring full access to the training dataset and the baseline model. Furthermore, while it produces verifiable feature attributions that tell us how important each feature is to the model's prediction, it does not tell us what the relationship between important features and the output/label is, as is true with all feature attributions.
This work is supported in part by the NSF awards IIS-2008461, IIS-2040989, IIS-2238714, Kempner Institute Graduate Fellowship, and research awards from Google, JP Morgan, Amazon, Harvard Data Science Initiative, and the Digital, Data, and Design  $(\mathrm{D}^{3})$  Institute at Harvard. The views expressed here are those of the authors and do not reflect the official policy or position of the funding agencies.
[17] Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invasive: Instance-wise variable selection using neural networks. In International Conference on Learning Representations, 2019.
[18] Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their interpretations. In International Conference on Artificial Intelligence and Statistics, pages 1459-1467. PMLR, 2021.
[19] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[20] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[21] Moritz Böhle, Mario Fritz, and Bernt Schiele. B-cos networks: Alignment is all we need for interpretability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10329-10338, 2022.
[22] Himabindu Lakkaraju and Osbert Bastani. "how do i fool you?" manipulating user trust via misleading black box explanations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 79-85, 2020.
[23] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2660-2673, 2016.
[24] Chirag Agarwal, Satyapriya Krishna, Eshika Saxena, Martin Pawelczyk, Nari Johnson, Isha Puri, Marinka Zitnik, and Himabindu Lakkaraju. Openxai: Towards a transparent evaluation of model explanations. Advances in Neural Information Processing Systems, 35:15784-15799, 2022.
[25] Prateek Jain and Purushottam Kar. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning, 10, 2017. URL all_papers/JainK17_FTML.pdf.
[26] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[27] Daniel Kermany, Kang Zhang, Michael Goldbaum, et al. Labeled optical coherence tomography (oct) and chest x-ray images for classification. Mendeley data, 2(2):651, 2018.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[29] Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. Advances in Neural Information Processing Systems, 32, 2019.
7 Proofs
Theorem 2.  $\mathcal{Q}$ -feature attributions applied to  $\mathcal{Q}$ -robust models  $(\mathcal{F}_v(\mathcal{Q}))$  recover the signal-distractor decomposition for Bayes-optimal predictor  $f_v^* \in \mathcal{F}_v(\mathcal{Q})$ .
Proof. Consider QFA with  $\epsilon = 0$ . Let  $f^{*}$  be the Bayes-optimal model, which implies that  $f_{y}^{*}(\mathbf{x}) = p_{bayes}(y \mid \mathbf{x}) = p(y \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid y)}{\sum_{i} p(\mathbf{x} \mid y = i)}$ , i.e., the model estimates the correct conditional probabilities from the data given the class-conditional generative probabilities. Note that this is only defined for inputs  $\mathbf{x} \in \mathcal{X}$  in the support of the data distribution and not outside, i.e.,  $p_{bayes}(y \mid \mathbf{x}) = p(y \mid \mathbf{x})$  only when  $\mathbf{x} \in \mathcal{X}$ .
Let us define  $\mathbf{x} \odot (1 - \mathbf{m}) = \mathbf{x}_{\mathrm{distractor}} \sim \mathcal{X}_{\mathrm{distractor}}$ . This is the distribution of distractor images, which (recall) are independent of the label  $y$ . Using this, we consider an idealized version of QFA, with  $\mathcal{Q}_{ideal} = \mathcal{X}_{\mathrm{distractor}}$ . From Definition 1, this results in generation of simplified inputs  $\mathbf{x}_s = \mathbf{x} \odot \mathbf{m} + q \odot (1 - \mathbf{m})$ . For  $q \sim \mathcal{Q}_{ideal} = \mathcal{X}_{\mathrm{distractor}}$ , we observe that  $\mathbf{x}_s \in \mathcal{X}$ , the data distribution. Recall the Bayes optimal classifier is defined across the data distribution  $\mathcal{X}$ , and applying QFA results in the sparsest  $\mathbf{m}$  such that  $p(y | \mathbf{x}) = p_{bayes}(y | \mathbf{x}) = p_{bayes}(y | \mathbf{x}_s) = p(y | \mathbf{x} \odot \mathbf{m} + q \odot (1 - \mathbf{m})) = p(y | \mathbf{x} \odot \mathbf{m})$ . The last equality holds because  $q \odot (1 - \mathbf{m})$  is independent of  $y$ . This corresponds to the definition of the signal-distractor, and thus it implies that QFA with  $\mathcal{Q}_{ideal}$  recovers the mask defined by the signal-distractor decomposition.
For any other value of  $\mathcal{Q} \neq \mathcal{Q}_{ideal}$ , we first consider the  $\mathcal{Q}$ -robust Bayes optimal predictor  $p_{bayes}^{\mathcal{Q}}(y \mid \mathbf{x})$ . This has the property that  $p_{bayes}^{\mathcal{Q}}(y \mid \mathbf{x}) = p_{bayes}(y \mid \mathbf{x})$  for  $\mathbf{x} \in \mathcal{X}$ . We now compare mask  $\mathbf{m}_{\mathcal{Q}}$  derived from applying QFA on  $p_{bayes}^{\mathcal{Q}}$  and mask  $\mathbf{m}_{\mathcal{Q}_{ideal}}$  from applying QFA with  $\mathcal{Q}_{ideal}$  on  $p_{bayes}^{\mathcal{Q}}$ . From the previous paragraph, we know that  $\mathbf{m}_{\mathcal{Q}_{ideal}} = \mathbf{m}_{ideal}$  is the ideal sparsest mask. However from Definition 2,  $\mathbf{m}_{\mathcal{Q}}$  is the sparsest mask. Thus it is the case that  $\mathbf{m}_{\mathcal{Q}} = \mathbf{m}_{ideal}$ , proving our overall result.
We now present proof for an additional statement not described in the main text, where we connect QFA to other commonly used feature attributions via the local function approximation framework [3] as follows.
Theorem 3. QFA is an instance of the local function approximation framework (LFA), with (1) random binary perturbations, and (2) an interpretable model class consisting of linear models with binary weights
Proof. Assume a black box model given by  $f_{b}(\mathbf{x};\mathbf{m}) = \mathbb{1}\left(\mathbb{E}_{q}\| f(\mathbf{x}_{s}(\mathbf{m},q)) - f(\mathbf{x})\|_{2}\leq \epsilon\right)$ , loss function  $\ell (f,g,x,\xi) = (f(x;\xi) - g(\xi))^2$ , neighborhood perturbation  $Z = \mathrm{Uniform}(0,1)^d$ , and an interpretable model family  $\mathcal{G}$  being the class of binary linear models.
For these choices, it is easy to see that
$$
\begin{array}{l} \operatorname *{arg  min}_{g\in \mathcal{G}}\ell (f,g,\mathbf{x},\xi) \\ = \arg \min  _ {g \in \mathcal {G}} \mathbb {E} _ {\xi} \left(f _ {b} (\mathbf {x}; \xi) - g ^ {\top} \xi\right) ^ {2} + \lambda \| g \| _ {0} \\ = \arg \min  _ {g \in \mathcal {G}} \mathbb {E} _ {\xi} \left(\mathbb {1} \left(\mathbb {E} _ {q} \| f (\mathbf {x} _ {s} (\xi , q)) - f (\mathbf {x}) \| _ {2} \leq \epsilon\right) - g ^ {\top} \xi\right) ^ {2} + \lambda \| g \| _ {0} \\ \end{array}
$$
This above objective is minimized when  $g = \mathbf{m}^*$ , i.e., the ideal  $\epsilon \mathcal{Q}$ -FA mask, because this sets the first term to be zero by definition, and the second sparsity term ensures the minimality of the mask.
Model Verifiability. We further test the verifiability of our models by evaluating how the model's performance changes when performing the pixel perturbation test on groundtruth attributions. This enables us to disentangle the verifiability of the model from the correctness of the attributions, as we know that our attributions are correct. We use the same groundtruth attributions as in 5.1. We report the  $\ell_1$  norm between predictions made on the original samples and predictions made on the masked samples. We compare our verifiable models to the baseline models they approximate, as well as models trained with input dropout, which [18] proposes as their verifiable model class. Training with input dropout is equivalent to training  $f_v$  with random masks and cross-entropy loss rather than optimized masks and  $f_b$  prediction matching. Results are shown in 3, where we see that our model performs similarly for masked and normal samples, whereas the other models do not.
Hard MNIST	Chest X-ray
DiET Model (fv)	0.027	0.0009
Original Model (fb)	0.107	0.0032
fb + Input Dropout	0.167	0.0536
Table 3: Model Verifiability