SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robustness to Explanation Attacks. We report additional results on Chest X-ray and CelebA for the pixel perturbation tests, IOU tests, and model faithfulness tests for baseline models trained with manipulated gradients, as outlined in 5.4. We see that DiETmodels are still highly faithful and produce correct explanations even when derived from models adversarially trained to have manipulated explanations.
Table 4: Faithfulness of DiETModel for Manipulated Models
Hard MNIST	Chest X-ray	CelebA
Accuracy on Original Data	0.990	1.00	0.970
fv(Dd) = fb(Dd)
Accuracy on Simplified Data	0.980	1.00	0.946
fv(Ds) = fb(Dd)
Table 5: Intersection Over Union Results for Manipulated Models
MNIST (manipulated)	Chest X-ray (manipulated)
Method	DiET(Ours)	0.454 ±0.08	0.631±0.12
SmoothGrad	0.158 ±0.07	0.000 ±0.00
GradCAM	0.040±0.06	0.000 ±0.00
Input Grad	0.002±0.01	0.000 ±0.00
FullGrad	0.333 ±0.12	0.004 ±0.04
Signal vs Distractor Masking. For our CelebA experiments, there may have been unintended spurious correlations that we did not foresee and that our method did not recover, leading to a vacuous decomposition. To further support the results in Figure 3, we also perform experiments on masking the signal instead of the distractor, by masking out "important" pixels as opposed to the unimportant ones. The results for this experiment are shown in 6.
Figure 6: Pixel perturbation tests for models trained with gradient manipulation for Chest X-ray (middle) and CelebA (right).
Table 6: Signal Masking experiments
Fraction Masked	Masked Distractor Acc	Masked Signal Acc	Random Mask Acc
0.2	0.971	0.772	0.965
0.4	0.967	0.569	0.935
0.6	0.967	0.392	0.826
0.8	0.960	0.339	0.587
These results indicate that for the DiETmodels, there do not exist pixels outside the shown signal regions in Figure 2 that contain information about the label. These results show that the signal-distractor decomposition for CelebA is very likely non-vacuous.
Finetuning on less data. We recognize that a limitation of this work is that it requires further training of the given models on their training datasets. To address this issue, we briefly explore whether DiET can be finetuned on a small subset of the data distribution (without labels) as follows for MNIST. The original model is trained on 8835 samples from the train split. DiETs finetuned on 1500 samples from a separate unlabeled validation split. We perform pixel perturbation on the remaining 8500 test samples. We compare this to DiET models finetuned on the full train set and note a minimal change in performance in 7.
Table 7: Small Scale Finetuning
Percent Masked	DiET(Finetuned on Train Split)	DiET(Finetuned on Val Split)
90	0.977	0.962
95	0.966	0.937
98	0.738	0.691
99	0.234	0.288
Overall, this shows that there exist ways to decrease the computational complexity of our procedure when applied to larger datasets, which we haven't fully investigated yet.
Effect on standard robustness. We conduct additional robustness experiments for Gaussian and Bernoulli noise with varying standard deviations and probabilities. Results are shown in 8 and 9. We find that DiETmodels are generally more robust than regular models. Note that neither DiETmodels nor the original models are explicitly trained or tuned for robustness on these distributions.
Additional Baselines: SHAP and BCosNets. We consider two additional explanation baselines: SHAP [7] and BCosNets [21]. Results for pixel perturbation tests and visualizations are shown in 7. Results for IOU tests are shown in 10.
Table 8: Robustness experiments (Gaussian noise)
Hard MNIST	CelebA
STD	Ours	Original	Ours	Original
0.2	0.974	0.976	0.959	0.939
0.4	0.961	0.892	0.675	0.537
1.0	0.61	0.176	0.335	0.438
Table 9: Robustness experiments (Bernoulli noise)
Hard MNIST	CelebA
p	Ours	Original	Ours	Original
0.9	0.182	0.155	0.348	0.356
0.75	0.718	0.3	0.393	0.364
0.5	0.922	0.748	0.528	0.453
0.1	0.962	0.969	0.867	0.807
We note that in general SHAP does not often perform as well as most gradient-based methods for image data. We find that it performs similarly to random attributions.
We add B-CosNets as an inherently-interpretable model baseline. We find that B-CosNets performs comparably to our method for the IOU test on Hard MNIST. We were unable to train it to convergence on the Chest X-ray dataset. We also find that visualizations created by B-CosNets align with our expectations and appear visually interpretable. However, our method still significantly outperforms B-CosNets on the pixel perturbation test, showing that B-CosNets are not robust to perturbations of the distractor (i.e. are not verifiable). We also note that our method can be applied to a trained black-box model of any architecture and training procedure, whereas B-CosNets, like all inherently interpretable models, cannot.
Table 10: SHAP and B-Cos IOU. Note that B-CosNets did not converge for the Chest X-ray dataset.
Hard MNIST	Chest X-ray
DiET	0.461 ± 0.08	0.821 ± 0.05
B-CosNets	0.465 ± 0.07	-
SHAP	0.036 ± 0.02	0.016 ± 0.05
Effect of Choice of Q. We perform an ablation to test the effect that the choice of Q has empirically. We consider various parameterizations of the normal distribution used for Q, and find that results are relatively consistent across the different choices of Q, as shown in 8.
9 Additional Implementation and Computation Details
Models were trained on the original train/test split given by https://github.com/jayaneetha/ colorized-MNIST for Hard MNIST and [27] for the Chest X-ray dataset and with a random 80/20 split for CelebA. Baseline models were trained with Adam for 10 epochs with learning rate  $1e - 4$  and batch size 256. All hyperparameters are included in the code for this paper. The model distillation and data distillation terms are weighted with  $\lambda_{1} = \lambda_{2} = 1$ . We learn our masks with SGD (lr=300, batch size = 128) and our robust models with Adam (lr=1e - 4, batch size = 128). We ran all experiments on a single A100 80 GB GPU with 32 GB memory.
Figure 7: Left: Pixel perturbation tests with added baselines of SHAP (orange) and B-CosNets (blue). Right: Visualization of SHAP and B-CosNets explanations on Hard MNIST and CelebA. Results for SHAP are shown at varying feature sizes. We note that SHAP explanations appear relatively random at all levels of granularity.
Figure 8: Pixel perturbation tests on Hard MNIST for DiET models trained with various Qs,  $Q \sim \mathcal{N}(\mu, \lambda \sigma)$ , for  $\lambda$  in  $\{0.01, 0.1, 1, 2\}$ .
10 Broader Impact
Our method, DiET, aims to transform black-box models into distractor robust, interpretable models, which produce easily verifiable feature attributions. As such, if applied correctly, it can help users and stakeholders of machine learning models better understand a model's predictions and behavior by isolating the features necessary for each prediction, which can help highlight biases, overfitting, mistakes, and more. It can also help to identify spurious correlations that naturally exist in datasets and are leveraged by models through identification of the signal-distractor decomposition. However, even if DiET does not identify a spurious correlation, that does not mean that further dataset cleaning, processing, or curation is not needed, as a different model may still learn a spurious correlation that was not leveraged by the original model. Furthermore, feature attributions often do not constitute a
complete explanation of a model. For instance, while an attribution tells us what was important, it does not tell us how it was important or how the model uses that feature. In all high stakes applications, it is still imperative that stakeholders think critically about each prediction and explanation, rather than blindly trusting either.