Missingness Bias in Model Debugging
Authors: Saachi Jain
Hadi Salman
Eric Wong
Pengchuan Zhang
Vibhav Vineet
Sai Vemprala
Aleksander Madry
Venue: ICLR 2022 Poster (2022)
Keywords: model debugging
vision transformers
missingness

Abstract: Missingness, or the absence of features from an input, is a concept fundamental to many model debugging tools. However, in computer vision, pixels cannot simply be removed from an image. One thus tends to resort to heuristics such as blacking out pixels, which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-based architectures can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice.


Published as a conference paper at ICLR 2022
MISSINGNESS BIAS IN MODEL DEBUGGING
Saachi Jain", Hadi Salman", Eric Wong', Pengchuan Zhang",
Vibhav Vineet", Sai Vemprala', Aleksander Madry'
Massachusetts Institute of Technology
*Microsoft Research
[saachij, hady, wongeric, madry ) @mit.
2 [penzhan, vivineet, sai. vemprala] @mi crosoft.ccom
ABSTRACT
Missingness, or the absence of features from an input, is concept fundamental to many model debugging tools. However, computer vision, pixels cannot simply be removed from image. One thus tends to resort heuristics such blacking out pixels. which may in turn introduce bias into the debugging process. We study such biases and, in particular, show how transformer-ban can enable a more natural implementation of missingness, which side-steps these issues and improves the reliability of model debugging in practice.
architectures
INTRODUCTION
Model debugging aims to diagnose model's failures. For example, researchers can identify global biases of models via the extraction of human-aligned concepts (Bau et al., 2017; Wong et al., 2021), or understand the texture bias by analyzing the models performance on synthetic datasets (Geirhos et al., 2019; Leclere et al., 2021). Other approaches aim to highlight local features to debug individ- ual model predictions (Simonyan et al., 2013; Dhurandhar et al., 2018; Ribeiro et al., 2016a; Goyal al., 2019).
A common theme in these methods is to compare the behavior of the model with and without certain individual features (Ribeiro et al., 2016a; Goyal al., 2019; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Dhurandhar et al., 2018: Chang et al., 2019). For example, in- terpretability methods such LIME (Ribeiro al., 201 and integrated gradients (Sundararajan et al., 2017) use the predictions when certain features are removed from the input to attribute differ- ent regions of the input the decision of the model. Dhurandhar al. (2018) find minimal regions in radiology images that are necessary for classifying a person as having autism. Fong & Vedaldi (2017) propose learning image masks that minimize a class score achieve interpretable explama- tions. Similarly, in natural language processing, model designers often remove individual words to understand their importance the output (Mardaoui Garreau, 2021; Li al., 2016). The absence of features from an input, concept sometimes referred to as missingness (Sturmfels et al., 2020), is thus fundamental to many debugging tools.
However, there is problem: while we can easily remove words from sentences, removing objects from images is not as straightforward. Indeed, removing a feature from an image usually requires approximating missingness by replacing those pixel values with something else, e.g., black color. However, these approximations tend not to be perfect (Sturmfels et al., 2020). Our goal thus to give holistic understanding of missingness and, specifically, to answer the question:
How do missingness approximations affect our ability to debug ML models?
Equul contribution.
Our code is available htwp::thhh...cooo/too.o.b.bbbaaa innnes..

---

Published as a conference paper at ICLR 2022
(a) Original image
(b) Masking human
(c) Masking the dog s snout
Consider an image of a dog being held by its owner. By removing the owner from the image, we can study how much our model's prediction depends on the presence of human. In identify which aspects of the dog (head, body, paws) are most critical for classifying the image by ablating these parts.
Figure 1:
similar vein,
OUR CONTRIBUTIONS
In this paper, we investigate how current missingness approximations, such as blacking out pixels, can result what we call missingness bias. This bias turns out to hinder our ability to debug models. We then show how transformer- architectures can enable a more natural implementation of missingness, allowing us side-step this bias. More specifically, our contributions include:
Pinpointing the missingness bias. We demonstrate multiple granularities how simple approxi- mations, such as blacking out pixels, can lead to missingness bias. This bias skews the overall output distribution toward unrelated classes, disrupts individual predictions, and hinders the model's use of the remaining (unmasked) parts the image.
Studying the impact of missingness bias on model debugging. We show that missingness bias negatively impacts the performance of debugging tools. Using LIME- common feature attribution method that relies on missingness -ss case study, we find that this bias causes the corresponding explanations to be inconsistent and indistinguishable from random explanations.
Using vision transformers to implement more natural form of missingness. The token- centric nature of vision transformers (VIT) (Dosovitskiy et al., 2021) facilitates a more natural im- plementation of missingness: simply drop the corresponding tokens of the image subregion we want to remove. We show that this simple property substantially mitigates missingness bias and thus enables better model debugging.
MISSINGNESS
Removing features from the input is an intuitive way to understand how a system behaves (Sturmfels al., 2020). Indeed, by comparing the system's output with and without specific features, infer what parts the input led a specific outcome (Sundararajan et al., 2017) see Figure 1. The absence of features from input sometimes referred to missingness (Sturmfels al., 2020).
The concept of missingness commonly leveraged in machine learning, especially for tasks such model debugging. For example, several methods for feature attribution quantify feature impor- tance by studying how the model behaves when those features are removed (Sturmfels et al., 2020; Sundararajan al., 2017; Ancona al., 2017). One commonly used method, LIME (Ribeiro al., 2016 iteratively turns image subregions on and off in order to highlight its important parts. Simi- larly, integrated gradients (Sundararajan al., 2017), a typical method for generating saliency maps, leverages a "baseline image" to represent the "absence" of features in the input. Missingness- tools are also often used domains such as natural language processing (Mardaoui & Garreau, 2021; Li et al., 2016) and radiology (Dhurandhar et al., 2018).
Challenges of approximatig missingness in computer vision. While ignoring parts of an image simple for humans, removing image features is far more challenging for computer vision models (Sturmfels et al., 2020). After all, convolutional networks require a structurally contiguous image input. We thus cannot leave "hole" in the image where the model should ignore the input.
2

---

Published as a conference paper at ICLR 2022
Original
Random
Least Salient
Most Salient
GT: Ilstworm
ResNet-50: flttorrm VIT-S: flalworm
ResNet-50 60: crossword VIT-S: flatworm
ResNet-50: jigsaw puzzle VIT-S: flatworm
ResNet-50 -0: dwelling VIT-S: sea slug
remove various regions of the original image; mask- ing for ResNet, and dropping tokens for ViT. (Section 2.1): Irrespective of what subregions of the image are removed (least salient, most salient, random), ResNet-50 outputs the wrong class (crossword, jigsaw puzzle, cliff dwelling). Taking a closer look at the randomly masked image of Figure 2, we notice that the predicted class (crossword puzzle) not totally unreasonable given the masking pattern. The model seems to be relying on the masking pattern to make the prediction, rather than the remaining (unmasked) portions of the image. (Section 2.2): The VIT-S on the other hand either maintains its original prediction predicts reasonable label given remaining image subregions.
Figure 2: Given an image of flatworm
Consequently, practitioners typically resort to approximating missingness by replacing these pixels with other, intended to be "meaningless", pixels.
Common missingness approximations include replacing the region of the image with black color, random color, random noise, a blurred version of the region, and forth (Sturmfels et al., 2020; Ancona et al., 2017: Smilkov et al., 2017; Fong & Vedaldi, 2017; Zeiler & Fergus, 2014; Sundarara- jan al., 2017). However, there is clear justification for why any of these choices is good approximation
missingness. For example, blacked out pixels are an especially popular baseline, motivated by the implicit heuristic that near zero inputs are somehow neutral for simple model (An- cona et al., 2017). However, if only part of the input is masked or the model includes additive bias terms, the choice of black is still quite arbitrary. In (Sturmfels al., 2020), the authors found that saliency maps generated with integrated gradients are quite sensitive to the chosen baseline color, and thus can change significantly based on the (arbitrary) choice of missingness approximation.
2.1 MISSINGNESS BIAS
What impact do these various missingness approximations have on models? We find that current approximations can cause significant bias the model's predictions. This causes the model to make errors based on the "missing" regions rather than the remaining image features, rendering the masked image -itto---disttbution..
Figure depicts an example of these problems. If we mask a small portion of the image, irrespective of which part of the image that is, convolutional networks (CNNs) output the wrong class. fact, CNNs seem to be relying on the masking pattern make the prediction, rather than the remaining (unmasked) portions the image. This type of behavior can be especially problematic for model debugging techniques, such as LIME, that rely on removing image subregions to assign importance to input features. Further examples can be found in Appendix C.I.
There seems be an inherent bias accompanying missingness approximations, which we refer as the missingness bias. In Section 3, we systematically study how missingness bias can affect model predictions at multiple granularities. Then in Section 4, we find that missingness bias can cause undesirable effects when using LIME by causing its explanations to be inconsistent and indistin- guishable from random explanations.
2.2 A MORE NATURAL FORM OF MISSINGNESS VIA VISION TRANSFORMERS
The challenges of missingness bias raises important question: what constitutes correct notion of missingness? Since masking pixels creates biases in our predictions, we would ideally like to remove those regions from consideration entirely. Because convolutional networks slide filters across the
3

---

Published as a conference paper at ICLR 2022
image, they require spatially contiguous input images. We are thus limited to replacing pixels with some baseline value (such as blacking out the pixels), which leads to missingness bias.
Vision transformers (ViTs) (Dosovitskiy et al., 2021) use layers of self-attention instead of convo- lutions to process the image. Attention allows the network to focus on specific sub-regions while ignoring other parts of the input (Vaswani et al., 2017; Xu et al., 2015); this allows ViT's to be more robust to occlusions and perturbations (Naseer et al., 2021). These aspects make ViTs especially appealing for countering missingness bias in model debugging.
In particular, we can leverage the unique properties of ViTs to enable a far more natural implementa- tion of missingness. Unlike CNNs, ViTs operate on sets of image lokens, each of which correspond to positionally encoded region of the image. Thus, in order to remove a portion of the image, we can simply drop the tokens that correspond to the regions of the image we want "delete." " Instead of replacing the masked region with other pixel values, we can modify the forward pass of the ViT to directly remove the region entirely.
We will refer to this implementation of missingness as dropping tokens throughout the paper (see Appendix B for further details). As we will see, using ViTs to drop image subregions will allow us to side-step missingness bias (see Figure 2), and thus enable better model debugging".
3 THE IMPACTS OF MISSINGNESS BIAS
Section 2.1 featured several qualitative examples where missingness approximations affect the model's predictions. Can we get precise grasp on the impact of such missingness bias? In this section, pinpoint how missingness bias can manifest at several levels of granularity. We fur- ther demonstrate how, by enabling a more natural implementation missingness through dropping tokens, VITs can avoid this bias.
Setup. To systematically measure the impacts of missingness bias, we iteratively remove subre- gions from the input and analyze the types of mistakes that our models make. See Appendix for experimental details. We perform an extensive study across various: architectures (Appendix C.3), missingness approximations (Appendix C.4), subregion sizes (Appendix C.5), subregion shapes: patches vs superpixels (Appendix C.6), and datasets (Appendix E).
Here we present our findings on a single representative setting: removing 16 x 16 patches from ImageNet images through blacking out (ResNet-50) and dropping tokens (ViT-S). The other settings lead to similar conclusions as shown in Appendix C. Our assessment of missingness bias, from the overall class distribution to individual examples, is guided by the following questions:
To what extent do missingness approximations skew the model's overall class distribution? We find that missingness bias affects the model's overall class distribution (i.e the probability of predicting any one class). In Figure 3, we measure the shift in the model's output class distribution before and after image subregions are randomly removed. The overall entropy of output class dis- tribution degrades severely. In contrast, this bias is eliminated when dropping tokens with the ViT. The ViT thus maintains high class entropy corresponding to a roughly uniform class distribution. These findings hold regardless of what order we remove the image patches (see Appendix C.2).
Does removing random or unimportant regions flip the model's predictions? We now take closer look at how missingness approximations can affect individual predictions. In Figure 4, we plot the fraction of examples where removing a portion of the image flips the model's prediction. We find that the ResNet rapidly flips its predictions even when the less relevant regions are removed first. This degradation is thus more likely due to missingness bias rather than the removal of individual regions. In contrast, the VIT maintains its original predictions even when large parts of the image are removed.
Do remaining unmasked regions produce reasonable predictions? When removing regions of the image with missingness, we would hope that the model makes a "best-effort" prediction
Unless otherwise specified, we drop tokens for the vision transformers when analyzing missingness bias on ViTs. An analysis of the missingness bias for ViTs when blacking out pixels can be found in Appendix C.7.
4

---

Published as a conference paper at ICLR 2022
Pasher:50 50
Un ttem
Stedked
Elacked
Class Entropy Degradation
Frodunal
7-
6-
Entropy
3-
deserin of 1ondoaa
ResNet-50 VIT-S
0-
peaneti pece caltion
150
200
Number of Patches Remaved
Figure 3: We measure the shift in output class distribution after applying missingness approxima- tions. Left: Fraction of images predicted as each class (on a log scale) before and after randomly removing 50% of the image. We display the most frequently predicted 30 classes after applying the missingness approximations. Right: Degradation in overall class entropy as subregions are removed. As patches are blacked out, the ResNet's predictions skew from uniform distribution toward a few unrelated classes such as maze, crossword puzzle, and carton. On the other hand, the ViT maintains a uniform class distribution with high class entropy.
ResNet-50
Frecton of Predictions Unchenged
Frection of Predictions Unchenged
0.8-
0.8
Random
Least Salient First
0.4
0.4
ResNet-50
0.2
Original
Retrained
0.0
0.0
0
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Number of Features Removed
Number Features Removed
Figure 4: We plot the fraction of images where the prediction does not change image regions are removed. The ResNet flips its predictions even when unrelated patches removed, while the ViT maintains its original prediction.
Figure 5: We repeat the experiment in Figure with models retrained with missingness aug- mentations. Applying missingness approxima- tions during training mitigates missingness bias for ResNets.
given the remaining image features. This assumption is critical for interpretability methods such as LIME (Ribeiro et al., 20) where crucial features are identified by iteratively masking out image subregions and tracking the model's predictions.
Are models actually using the remaining uncovered features after missingness approximations are applied though? To answer this question, we measure how semantically related the model's predictions are after masking compared to its original prediction using similarity metric the WordNet Hierarchy (Miller, 1995) as shown in Figure 6. By the time we mask out 25% of the image, the predictions of the ResNet largely become irrelevant to the input. ViTs the other hand continue to predict classes that are related to the original prediction. This indicates that ViTs successfully leverage the remaining features in the image to provide reasonable prediction.
Can we remove missingness bias by augmenting with missingness approximations? One way remove missingness bias could be to apply missingness approximations during training. For
5

---

Published as a conference paper at ICLR 2022
WuP Sim From Original Prodlction
Random
Original Prediction
Loast Salient
WUP Sim From From Original Prodiction
Most Salient
0.8
RedVet-50
0.8
ResNet-50
0.8
ResNet-50
ViT-S
0.7-
0.7
0.7
0.6-
0.6
0.5-
0.5
0.5
50
100
150
200
50
100
150
200
50
100
150
200
Number of Features Removad
Number of Features Removed
Number of Features Removed
Figure 6: We iteratively remove image regions the order of random, most salient, and least salient. We then plot the average WordNet similarity between the original prediction and the new prediction the predictions differ. We find that ViT-S, even when the prediction changes, continues to predict something relevant to the original image.
example, in RemOve and Retrain (ROAR), Hooker et al. (2018) suggest retraining multiple copies of the model by blacking out pixels during training (see Appendix F for an overview on ROAR).
To check if this indeed helps side-step the missingness bias, retrain our models by randomly removing 50% of the patches during training, and again measure the fraction of examples where removing image patches flips the model's prediction (see Figure 5). While there a significant gup in behavior between the standard and retrained CNNs, the ViT behaves largely the same. This result indicates that, while retraining important when analyzing CNNs. is unnecessary for ViTs when dropping the removed tokens: we can instead perform missingness approximations directly on the original model while avoiding missingness bias for free. See Appendix for more details.
4 MISSINGNESS BIAS IN PRACTICE: A CASE STUDY ON LIME
Missingness approximations play a key role several feature attribution methods. One attribution method that fundamentally relies missingness is the local interpretable model-agnostic explana- tions (LIME) method (Ribeiro et al., 2016a). LIME assigns a score to each image subregion based on its relevance to the model's prediction. Subregions of the image with the top scores referred to as LIME explanations. A crucial step of LIME is "turning off" image subregions, usually by replacing them with some baseline pixel color. However, as found in Section missingness approximations can cause missingness bias, which can impact the generated LIME explanations.
We thus study how this bias impacts model debugging with LIME. To this end, we first show that missingness bias can create inconsistencies in LIME explanations, and further cause them to be indistinguishable from random explanations. In contrast, by dropping tokens with ViTs, we can side-step missingness bias in order to avoid these issues, enabling better model debugging.
Figure depicts an example of LIME explanations. Qualitatively, we note that explanations gen- crated for standard ResNets seem to be less aligned with human intuition than ViTs or ResNets retrained with missingness augmentations".
Missingness bias creates inconsistent explanations. Since LIME uses missingness approxima- tions while scoring each image subregion, the generated explanations can change depending on which approximation used. How consistent are the resulting explanations? We generate such explanations for a ViT and a CNN using 8 different baseline colors. Then, for each pair of colors, we measure how much their top-k features agree (see Figure 8). We find that the ResNet produces explanations that are almost as inconsistent as randomly generated explanations. The explanations of the VIT, however, are always consistent by construction since the ViT drops tokens entirely. For further comparison, we also plot the consistency of the LIME explanations of ViT-S where we mask out pixels instead of drop the tokens.
Missingness bias renders different LIME explanations indistinguishable. Do LIME explana- tions actually reflect the model's predictions? A common approach to answer this is to remove the
Seee Appendix D.I for more details on this. We also include an overview of LIME and detailed experimental setup for this section in Appendix A, and further experiments using superpixels in Appendix D.2.

---

Published as a conference paper at ICLR 2022
LIME Explenation
Maskinng Top Fealures
LIME Explanation
Masking Top. Features
VT-88
VIT-B
Oligial Image
ReeeNee-SS 50
Flee PheeNee-Se 50
Original
NeeeNe--50 (with iitoc. sugmentzationsl
Ree (with micos aagght Auggmentatiions
Figure 7: Examples of generated LIME explanations and masking the top 20 features. Since LIME requires removing image features, can be subject missingness bias. We note that LIME expla- nations generated for standard ResNets seem to be less aligned with human intuition than ViTs or ResNets retrained with missingness augmentations (See Appendix D.I for more examples).
jaccard Smilarty Top K Protuue
RedNet-50
MT-S (dropping tokers)
MT-5 (masking picek)
0.4
62-
0.0
Features
Figure 8: We plot the agreement (using Jaccard similarity) of top-k features across LIME explana- tions of 28 pairs of baseline colors. The result is averaged over the 28 pairs, and we display the 95% confidence interval over the pairs colors. ResNet-50's explanations are almost as consistent as random explanations. For ViT with dropping tokens, explanations are naturally always consistent.
top-k subregions (by masking using missingness approximation), and then check if the model's prediction changes (Samek et al., 2016). This is sometimes referred to as the top- ablation test (Sturmfels et al., 2020). Intuitively, explanation is better if it causes the predictions flip more rapidly. We apply the top-k ablation test four different LIME explanations on a ResNet-50 and a VīT-S as shown in Figure 9. Specifically, for each model evaluate: 1) its generated explana- tions, 2) the explanations of an identical architecture trained with different seed 3) the explanations of the other architecture and 4) randomly generated explanations.
For CNNs (Figure 9-1eft), all four explanations (even the random one) flip the predictions at roughly an equal rate the top-K ablation test. In these cases, the bias incurred during evaluation plays larger role in changing the predictions than the importance of the region being removed, rendering the ViT however (Figure 9-right), the LIME explana-
the four explanations indistinguishable. tion generated from the original model outperforms all other explanations (followed by an identical model trained with different seed). As we would expect, the ResNet and the random explanations cause minimal prediction flipping, which indicates that these explanations do not accurately capture feature importance for the VIT. Thus, unlike for the CNNs, the different LIME explanations for the VIT are distinguishable from random (and quantitatively better) via the top-k ablation test.
What happens if we retrain our models with missingness augmentations? As in Section 3, repeat the above experiment on models where 50% the patches are removed during training. The results reported in Figure 10. We find that the LIME explanations evaluated with the retrained CNN are now distinguishable, and the explanation generated by the same CNN outperforms the other explanations. Thus, retraining with missingness augmentation "fixes" the CNN and makes the

---

Published as a conference paper at ICLR 2022
Evaluating Model: ResNet-50
Evaluating Model:
1.0
1.0
Fraction of Predictions Unchangen gei
Fraction of Predictions Unchanged
0.8
0.8
LIME Explanation From
0.6
0.6
ResNet-50
VIT-S
Random
0.4
0.4
Original
Different Seed
0.2
0.2
0.0
0.0
40
50
10
20
30
40
Number of Features Removed
Number of Features Removed
Figure 9: We evaluate LIME explanations using the top-K ablation test on a ResNet and ViT by measuring the fraction of examples who keep their original prediction after removing the Top-K features. sharper degradation indicates a more appropriate explanation for that model. While the LIME scores on the ResNet are largely indistinguishable, the VIT shows clear differentiation between the different explanations.
Evaluating Model: ResNet:50
Evaluating Model: VIT-S (With Miss. Augmentation)
(With Miss. Augmentation)
1.0
Fraction of Prodictions Unchanged
Fraction of Predictions Unchanged
0.8
0.8
LIME Explanation From
0.6
0.6
ResNet-50
VIT-5
Random
Original
Different Seed
0.2
0.2
0.0
40
0.0
100
Number of Features Removed
Number of Features Removed
Figure 10: We replicate the experiment in Figure 9, but instead models where missingness approximations were introduced during training. This procedure fixes evaluation issues for ResNets, but does not substantially change the evaluation picture of ViTs.
top-k ablation test more effective by mitigating missingness bias. On the other hand, since the ViT already side-steps missingness bias by dropping tokens, change when using the retrained model. We can thus evaluate LIME explanations directly on the original ViT without resorting surrogate models.
the top-k ablation test does not substantially
RELATED WORK
Interpretability and model debugging. Model debugging is a common goal in interpretability research where researchers try justify model decisions based either local features (i.e., specific to given input) or global ones (i.e., general biases of the model). Global interpretability methods include concept-based explanations (Bau et al., 2017; Kim et al.. 2018: Yeh al., 2020; Wong al., 2021). They also encompass many of the robustness studies including adversarial (Szegedy et al., 2014; Goodfellow et al., 2015; Madry et al., 2018) or synthetic perturbations (Geirhos et al., 2019; Engstrom et al., 2019; Kang et al., 2019; Hendryeks & Dietterich, 2019; Xiao et al., 2020; Zhu et al., 2017) which can be cast as uncovering global biases of vision models (e.g. the model is biased to textures (Geirhos et al., 2019)). Local explanation methods, also known as feature attribution methods, aim to highlight important regions the image causing the model prediction. Many of these methods gradients to generate visual explanations, such saliency maps (Simonyan al., 2013; Dabkowski & Gal, 2017; Sundararajan et al., 2017). Others explain the model predictions by studying the behavior of models with and without certain individual features (Ribeiro et al., 201
8

---

Published as a conference paper at ICLR 2022
Goyal et al., 2019; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Dhurandhar et al., 2018; Chang et al., 2019; Hendricks et al., 2018; Singla et al., 2021). They focus on the change in classifier outputs with respect to images where some parts are masked and replaced with various references such as random noise, mean pixel values, blur, outputs of generative models, etc.
Evaluating feature attribution methods. It is important for feature attribution methods to truly reflect why the model made decision. Unfortunately, evaluating this is hard since we lack the ground truth of what parts of the input are important. Several works showed that visual assessment fails to evaluate attribution methods (Adebayo al., 2018; Hooker al., 2018; Kindermans al., 2017: Lin et al., 2019; Ych et al., 2019; Yang & Kim, 2019; Narayanan et al., 2018), and instead proposed several qualitative tests as replacements. Samek al. (2016) proposed the region pertur- bation method which removes pixels according to the ranking provided by the attribution maps, and measures how the prediction changes i.e., how the class encoded the image disappears when we progressively remove information from the image. Later on, Hooker et al. (2018) proposed remove and retrain (ROAR) which showed that in order for region perturbation method to be more informa- tive, the model has to be trained with those perturbations. Kindermans et al. (2017) posit that feature attribution methods should fulfill invariance with respect to some set of transformations, e.g. adding constant shift to the input data. Adebayo et al. (2018) proposed several sanity checks that feature attribution methods should pass. For example, a feature attribution method should produce different attributions when evaluated on trained model and randomly initialized model. Zhou et al. (2021) proposed a modifying datasets such that a model must rely on a set of known and well-defined features to achieve high performance, thus offering ground truth for feature attribution methods. The notion of missingness. The notion of missingness commonly used in machine learning. especially for tasks such as feature attribution (Sturmfels et al., 2020; Sundararajan et al., 2017; Hooker et al., 2018; Ancona et al., 2017). Practitioners leverage missingness in order to quantify the importance of specific features, by evaluating the model when removing the feature in the input (Ribeiro et al., 20; Goyal et al., 2019; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017: Zintgraf et al., 2017; Dhurandhar et al., 2018; Chang et al., 2019; Sundararajan et al., 2017; Carter et al., 2021; Covert et al., 2021). For example, in natural language processing, model designers often remove tokens to assign importance to individual words (Mardaoui & Garreau, 2021; Li et al., 2016). In computer vision, missingness is foundational to several interpretability methods. For example, LIME (Ribeiro et al., 2016a) iteratively turns image features and off in order to learn the importance of each image subregion. Similarly, integrated gradients (Sundararajan et al., 2017) requires "baseline image" that is used represent "absence" of feature in the input. It is thus important to study missingness and how to properly approximate it for computer vision applications. Vision transformers. Our work leverages the vision transformer (ViT) architecture, which was first proposed by Dosovitskiy et al. (2021) as a direct adaptation of the popular transformer architec- ture used in NLP applications ( (Vaswani et al., 2017) for computer vision. In particular, ViTs do not include any convolutions, instead they tokenize the image into patches which are then passed through several full layers of multi-headed self-attention. These help the transformer globally share infor- mation between all image regions at every layer. While convolutions must iteratively expand their receptive fields, vision transformers can immediately share information between patches throughout the entire network. ViTs recently got lot of attention in the research community from works rang- ing from efficient methods for training ViTs (Touvron et al., 2020) to studying the properties of ViTs (Shao et al., 2021; Mahmood et al., 2021; Naseer et al., 2021; Salman et al., 2021). The work of Naseer et al. (2021) is especially related to our work it studies the robustness of ViTs to number of input modifications including occlusions. They find that the overall accuracy of CNNs degrades more quickly than ViTs when large regions of the input is masked. These findings complement our intuition that ResNets can experience bias when pixels are replaced with a variety of missingness approximations.
CONCLUSION
In this paper, we investigate how current missingness approximations result in missingness bias. We also study how this bias interferes with our ability to debug models. We demonstrate how transformer-based architectures are one possible solution, as they enable a more natural (and thus less biasing) implementation of missingness. Such architectures indeed side-step missingness bias and are more reliable to debug in practice.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENTS
Work supported in part by the NSF grants CCF-1553428 and CNS-1815221. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001 120C0015.
Research sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 1-8750011-22-0866
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrrunk Slic superpixels. Technical report, 2010.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems (NeurIPS), 2018.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint 1oXiv:171..0611 2017.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition (CVPR), 2017.
Emanuel Ben-Baruch, Tal Ridnik, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. arXiv preprint arXiv:2009.14119 2020.
Brandon Carter, Siddhartha Jain, Jonas W Mueller, and David Gifford. Overinterpretation reveals image classification model pathologies. Advances in Neural Information Processing Systems, 34, 2021.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference on Learning Representations (ICLR), 2019.
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90, 2021.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Neural Information Processing Systems (NeurIPS), 2017.
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. arXiv preprint 1rXiv:1802.07622 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Ex- ploring the landscape of spatial robustness. In International Conference Machine Learning (ICML), 2019.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In International Conference on Computer Vision (ICCV) 2017.
10

---

Published as a conference paper at ICLR 2022
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICIR), 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint arXiv: / 990400451, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog-
nition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual expla- nations. Proceedings of the European Conference on Computer Vision (ECCV), pp. 264-279, 2018.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. In International Conference on Learning Representations (ICLR), 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. benchmark for interpretabil- ity methods in deep neural networks. arXiv preprint arXiv: 10758, 2018.
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardr. Testing robustness against unforeseen adversaries. In ArXiv preprint rrxiiv1908 08016, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning (ICML), 2018.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In arXiv preprint arXiv:1711.0066, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
Guillaume Leclerc, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang, et al. 3db: A framework for debugging computer vision models. In arXiv preprint arXiv:2106.0380 2021.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. In Proceedings of NAACL-HLT, pp. 68 18669,1,22 2016.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision (ECCV), 2014.
Zhong Qiu Lin, Mohammad Javad Shafiee, Stanislav Bochkarev, Michael St Jules, Xiao Yu Wang, and Alexander Wong. Do explanations reflect decisions? a machine-centric strategy to quantify the performance of explainability algorithms. arXiv preprint
2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations (ICLR), 2018.
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. 2021.
Dina Mardaoui and Damien Garreau. An analysis lime for text data. In International Conference on Artificial Intelligence and Statistics, 2021.
George A Miller. Wordnet: lexical datahase for english. Communications of the ACM, 1995.
11

---

Published as a conference paper at ICLR 2022
Menaka Narayanan, Emily Chen, Jeffrey He, Been Kim, Sam Gershman, and Finale Doshi- Velez. How do humans understand explanations from machine learning systems? an evaluation of the human- ciinan-itterprttablittyy of explanation. arXiv preprint arXlv:1802.00682, 2018.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. arXiv preprint TrXiv:2105.10497 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016a.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. International Conference on Knowledge Discovery and Data Mining (KDD), 2016b.
Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. In International Journal of Computer Vision (JCV), 2015.
Hadi Salman, Saachi Jain, Eric Wong, and Aleksander Mądry. Certified patch robustness via smoothed vision transformers. arXiv preprint urXiv:2110.07719, 2021.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Rober Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, (8(11):2600267.7 2016.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robust- ness visual transformers. 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR), 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.6604. 2013.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862, 2021.
D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg. SmoothGrad: removing noise by adding noise. /CML workshop on visualization for deep learning, 2017.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020. doi:
baselines.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning (ICML). 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Computer Vision and Pattern Recog- nition (CVPR), 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar 2xXiv:20/2.12877 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2022
Ross Wightman. Pytorch image models.
http:s///ggtaab..mmm//wwtttttttt
pytorch-imagee iaazch--maae-mmdeele 2019.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debug- gable deep networks. In International Conference Machine Learning (ICML), 2021.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006.09900 2020.
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2057.2007. PMLR, 2015.
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701, 2019.
Chih- Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar, On the (in) fidelity and sensitivity of explanations. Advances Neural Information Processing Systems, 32:10967-1097 2019.
Chih-Kua Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aweee aware concept-based explanations in deep neural networks. Advances Neural Information Processing Systems (NeurIPS), 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? arXiv preprint arXiv:2104. 14403, 2021.
Zhuotum Zhu, Lingxi Xie, and Alan Yuille. Object recognition without and without objects. In International Joint Conference Artificial Intelligence, 2017.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 170.:51702.04595, 2017.
13

---

Published as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS.
A.1 MODELS AND ARCHITECTURES
We use two sizes of vision transformers: ViT-Tiny (ViT-T) and ViT-Small (VIT-S) (Wightman, 2019; Dosovitskiy et al., 2021). We compare to residual networks of similar size: ResNet-18 and ResNet- 50 (He et al., 2016), respectively. These architectures and their corresponding number of parameters summarized in Table 1.
Table 1: collection neural network architectures we use in our paper.
Architecture Params
VIT-T 5M
ResNet-18 12M
VIT-S
ResNet-50 26M
22M
A.2 TRAINING DETAILS
We train our models on ImageNet (Russakovsky al., 2015), with custom (research, non-commercial) license, as found here sttts::/paperee thoode com/ hom//ataset/ imagenet. For all experiments in this paper, we consider 10,000 image subset of the original ImageNet validation set (we take every 5th image).
For ResNets, we train using SGD with batch size of 512, momentum of 0.9, and weight decay of le-4. We train for 90 epochs with initial learning rate of 0.1 that drops by factor of 10 every 30 epochs.
2. For ViTs, we use the same training scheme as used Wightman (2019).
Note that we use the same (basic) data- iitt---ugmetaatioon techniques for both ResNets and ViTs. Specif- ically, we only use random resized crop and random horizontal flip (no RandAug, CutMix, MixUp, etc.).
We attach all our model weights to the submission.
Models trained with missingness augmentations. In Sections 3 and 4, we also consider models that were augmented with missingness approximations during training (inspired by ROAR (Hooker et al.. 2018), see Appendix F for further discussion). We retrain our models by randomly removing 50% of the patches (by blacking out for ResNet and dropping the respective tokens for ViT). The other training hyperparameters are maintained the same as the standard models above.
Infrastructure and computational time. For ImageNet, we train our models on 4 V100 GPUs each, and training took around 12 hours for ResNet-18 and ViT-T, and around 20 hours for ResNet- 50 and VIT-S.
For CIFAR-10, fine-tune pretrained ViTs and ResNets on single V100 GPU. Fine-tuning ViT-T and 18 took around hours, and fine-tuning ViT-S and ResNet-50 took around 1.5 hours.
All of our analysis can be run on single 1080Ti GPU, where the time for one forward pass with batch size of 128 is reported in Table 2.
Table 2: A collection of neural network architectures we use in our paper.
Architecture
ViT-T 0.031 0.018
ResNet-18 0.033 0.013
ViT-S 0.041 + 0.016
ResNet-50 0.039 0.015
Inference time (sec)
A.3
EXPERIMENTAL DETAILS FOR SECTION 3
For the experiments in Section 3, we iteratively remove subregions from the input. In the main paper, consider removing 16 x 16 patches: black patches for the ResNet-50 and drop
14

---

Published as a conference paper at ICLR 2022
the corresponding token for the ViT-S. We consider other patch sizes well superpixels in Ap- pendix C.
We consider removing patches in three orders: random, most salient, and least salient. We saliency as a rough heuristic for relevance to the image (typically, more salient regions tend to in the foreground and less salient regions in the background). For all models, of an image subregion as the mean value of that subregion for standard ResNet-50's saliency map (the order of patches removed is thus the same for both the ResNet and the ViT).
determine the salience
A.4 EXPERIMENTAL DETAILS FOR SECTION
Overview on LIME Local interpretable model-agnostic explanations (LIME) Ribeiro et al. (2016b) is a common method for feature attribution. Specifically, LIME proceeds by generating perturbations of the image, where in each perturbation the subregions are randomly turned on or off. For ResNets, we turn off subregions by masking them with some baseline color, while for ViTs drop the associated tokens. After evaluating these perturbations with the model, fit classifier using Ridge Regression to predict the value of the logit of the original predicted class given the presence of each subregion. The LIME explanation is then the weight of each subregion in the ridge classifier (these are often referred to as LIME scores). We perform LIME with 1000 perturbations, and include implementation of LIME in our attached code.
Implementation details for LIME consistency plots. For the experiment in Figure 8, we evaluate LIME using 8 different baseline colors (the colors are generated by setting the R, G, and B values as either 0 or 1). Then, for each pair of colors, we measure the similarity of their top-k feature sets according to their LIME scores for varying (using Jaccard similarity) averaged over 10,000 examples. We plot the average over the 28 pairs of colors.
15

---

Published as a conference paper at ICLR 2022
B IMPLEMENTING MISSINGNESS BY DROPPING TOKENS IN VISION TRANSFORMERS
As described in Section 2.2, the token-centric nature of vision transformers enables natural implementation of missingness: simply drop the tokens that correspond to the removed image sub- regions. In this section, we provide more detailed description of dropping tokens, well as few implementation considerations.
Recall that ViT has stages when processing input image x.
Tokenization: is split into 16 16 patches and positionally encoded into tokens.
Self-Attention: The set of tokens is passed through several self-attention layers and pro- duces a class label.
After the initial tokenization step, the self-attention layers of the transformer deal solely with of tokens, rather than a constructed image. This set is not constrained to a specific size. Thus, after the patches have all be tokenized, we can remove the tokens that correspond removed regions of the input before passing the reduced set to the self-attention layers. The remaining tokens retain their original positional encodings.
Our attached code includes an implementation of the vision transformer which takes in an optional argument of the indices of tokens drop. Our implementation can also handle varying token lengths in batch (we use dummy tokens and then mask the self- self-attention layers appropriately).
Dropping tokens for superpixels and other patch sizes the main body of the paper, we deal with 16 x 16 image subregions, which aligns nicely with the tokenization of vision transformers. In Appendix C, we consider other patch sizes that do not align along the token boundaries, as well as irregularly shaped superpixels. In these cases, conservatively drop the token if any portion of the token was supposed to be removed (we thus remove a slightly larger subregion).
16

---

Published as a conference paper at ICLR 2022
C ADDITIONAL EXPERIMENTS (SECTION 3)
C.I ADDITIONAL EXAMPLES OF THE BIAS (SIMILAR TO FIGURE 2).
In Figure 11, we display more examples that qualitatively demonstrate the missingness bias.
daddy lego
Original
VIT-S: daddy long
WTT5: fotworm
VIT-S catemoran
VIT-S: vnlccnt
TTT buckoga
Fleshee-dD daddy long lage
Reeher-00: Tnlworm
achoom
Reehha-t0 voliano
Redfeed0: buckeps
Rendom
VIT-S: boods long Postit arostword
VIT 5: Retwo
VIT-S catamastn
VIT S: vokano
VTT buckeye
Rechie-00 crossword
RosMitt S0 aroswoord
Rechie maze
Fechen16: mede
Leoot
VIE-S civldy long laga
W11-0: Fotwes
VIT-S cetemaran
VIT-8: vnlieno
buckey
Jganw
Sallent
PRaaNet-50 hook
Fleahiet CD
Reshan envorope
FasaNes-SS sconaboard
Figure 11: Further examples of removing 75 16 x 16 patches from ImageNet images. The im- ages are blacked out for ResNet-50, and the corresponding tokens are dropped for ViT-S. While ResNet-50 skews toward classes that unrelated to the remaining image features (i.e crossword, jigsaw puzzle), the ViT-S either maintains its original prediction or predicts a reasonable label given remaining image features.
17

---

Published as a conference paper at ICLR 2022
C.2 BIAS FOR REMOVING PATCHES IN VARIOUS ORDERS
In this section, we display results for the experiments in Section 3 where we remove patches in 1) random order 2) most salient first and 3) least salient first (Figure 12). We find that missingness approximations skew the output distribution for ResNets regardless of what order we remove the patches. Similarly, we find that the ResNet's predictions flip rapidly in all three cases (though to varying extents). Finally, the ViT mitigates the impact of missingness bias in all three cases.
13% Raow
Tota Sercosd
WAB Sater Tres So
Selent
Bucked
(a) The shift in the output class distribution after applying missingness approximations in different orders.
Class Entropy Degradation
Accurscy Degracation
WerdNet Siilssrty origiaal Prediction
1.0-
INPP Sm Origiaal Prodiction
0.8
0.8
0.7-
ResNet-50 50 VIT-S
4-
Fraidlen Labele
0.6
Random Order
Least Salient First
0.5
50
100
150
200
50
100
350
200
0
50
100
150
200
Number of Patches Removed
Number
Patcches Removed
Number of Patcches Removed
(b) Degradation in class entropy (left), the fraction of predictions that change (middle), and the average Word- Net similarity if the prediction changes after masking (right) as we remove patches from the image in different orders.
Figure 12: Full experiments for removing 16 x 16 patches by blacking out (ResNet-50) or dropping tokens (VTT-S).
18

---

Published as a conference paper at ICLR 2022
C.3 RESULTS FOR DIFFERENT ARCHITECTURES
In this section, we repeat the experiments in Section with several other training schemes and types of architectures. Our results parallel our findings in the main paper.
C.3.1
VIT-T AND RESNET-18
Class Entropy Degradstion
Accurscy Degrasation
Wordhiet Simllarity to Original Prediction
1.0-
From Predictio
undhanged
0.8
0.7-
ResNet-18
0.6
0.6
VIT-T
3-
0.4
Most Salient First
Random
0.4
Salient First
0.2
0.3
0.0
160
200
Number Romoved
Number
Remeved
Figure 13: Bias experiments as in Section 3, with a ViT-T and ResNet-18.
C.3.2 VIT-S AND ROBUST RESNET-50
We consider a ViT-S and an L2 adversarially robust ResNet-50 (e = 3).
Entrapy Degradation
Accursey
Smiarity Original
1.0
WIF 5m From FromOrgissl Protiction
0.8
Labes Unchanges
0.8
0.7-
Robust Resiiet 50
Eneogy
0.6
0.6
Random Order
0.4
Least Sa lient First
0.2
0.5
50
100
150
100
150
200
150
200
Number of Patchees Removed
Number Patches Removed
Number Pati Removed
Figure 14: Bias experiments as Section 3, with ViT-S and robust ResNet-50.
C.3.3 VIT-S AND INCEPTIONV3
We consider a ViT-S and an Inception (Szegedy et al. (2016)) model.
Entropy Degradation
diitt Similarkyy Prediction
1.0
0.8
Froctton Lab-bl Lichangged
0.8
0.7
Inceptioniv3
Entrpy
From Oriinal
WIT-S
4-
Random Order
0.5
Least Salient First
0.4
50
100
150
200
50
100
3500
200
0
50
100
150
200
Number of Patches Removed
Number Patcche Removed
Number of Patchee Removed
Figure 15: Bias experiments in Section with ViT-S and Inception V3,
C.3.4 VIT-S AND VGG-16
We consider a ViT-S and a VGG-16 with BatchNorm (Simonyan & Zisserman (2015)).
19

---

Published as a conference paper at ICLR 2022
Class Entropy Degradation
Accurscy Degradation
Werdheet Siillarty Prediction
5im Original Prodictin
0.8
VGG16
VIT-S
Labels
0.6
0.4
0.6
Most Salient First
Random Order
Lesst Salient First
0.5
0.0
50
100
150
200
50
100
150
200
0
50
100
150
200
Number of Patches Removed
Number of Patches Ramoved
Number of Patches Removed
Figure 16: Bias experiments in Section with VIT-S and VGG16.
C.4 RESULTS FOR DIFFERENT MISSINGNESS APPROXTMATIONS
In this section, consider missingness approximations other than blacking out pixels. In Figure 17, we use three baselines: a) the mean RGB value of the dataset, b) a randomly selected baseline color for each image, and c) randomly selected color for each pixel Sturmfels al. (2020); Sundararajan et al. (2017). Since we drop tokens for the vision transformers, changing the baseline color does not change the behavior for the ViTs. Our findings in the main paper for blacking out patches closely mirror the findings for other baselines.
Class Entropy Degt adltion
Accuracy Degranationn
Werdhie: Similerly ky Driginal Prediciion
1.0
0.8
0.8
0.7
Entroy
From Original
VTT-S
0.6
3-
0.4
0.6
Most Salient First
Random Order
2-
0.2
0.5
Salient
0.0
50
100
150
200
100
250
200
100
150
200
Number
Removed
Number
Number Bemovedd
(a) Using the mean ImageNet RGB value for baseline color
Class Entropy Degradation
Accuracy Degracation
WerdNet Simllarity Original Prediction
1.0
Predidion
0.80
0.8
0.75
Entrop
0.70
VIT-S
0.6
Random Order
of
0.4
0.60
Most Salient First
0.55
0.2
0.50
0.0
0.45
50
100
150
100
250
200
100
1500
200
Number
Removed
Number
Reenoowd
Number Patches Berroved
(b) Picking a random baseline color each image,
Class Entropy Degradation
Accuracy Degracation
WerdNet Similarity Original Prodiction
Prodectonn
0.80
of Undranged
0.8
0.75
ResNet-50
0.70
VIT-S
0.6
Random Order
0.4
0.60
Most Salient First
0.55
0.2
0.50
0.0
0.45
50
100
150
200
100
150
200
50
100
150
200
Number of Patches Removed
Number Potches Removed
Number Potches Removed
(c) Picking random baseline color for each pixel.
Figure 17: Using different baseline colors for masking pixels.
20

---

Published as a conference paper at ICLR 2022
We also consider blurring the removed features, as suggested in Fong & Vedaldi (2017). We use gaussian blur with kernel size 21 and o = 10. Examples of blurred images can be found in Figure 18a. Unlike the previous missingness approximations, this method does not not fully remove subregions of the imput; thus, blurring the pixels can still leak information from the removed regions, which can then influence the model's prediction. Indeed, find that, by visual inspection, we can still roughly distinguish the label of images that are entirely blurred (as in Figure 18a).
(a) Examples of blurring ImageNet images.
Class Entropy Degt adaiion
Accuracy Degradaiin adation
Wer dhtt: Similarky Original Prediction
0.80
ResNet-50
0,70
VIT-5
0.6
0.65
Random Order
0.4
0.60
Most Salient first
2-
0.55
Least Salient First
0.2
0.50
0.45
50
100
150
200
100
150
200
100
150
200
Number of Patches Removed
Number Potchhe Removed
Number Potches Removed
(b) Repeating the experiments in Section 3 using the blurred ImageNet image.
Figure 18: Using the blurred image for the missingness approximation.
For completeness, we repeat the experiments in Section 3 using the blurred image as the missingness approximation (Figure 18b). We find that ResNets still experience missingness bias, though the bias is reduced compared to using image-independent baseline color.
C.5 USING DIFFERENTLY SIZED PATCHES
In the main body of the paper, we consider image subregions of 16 x 16. In this section, we consider subregions of other patch size: 14 14, 28 28, 32 32, and 56 56. As mentioned in Appendix Section B, when dropping tokens for the ViT, we conservatively drop the token if any part of the corresponding image subregion being removed. Thus, the ViT removes slightly more area than the ResNet for patch sizes that are not multiples of 16. We find that the ResNet is impacted by missingness bias regardless of the patch size, though the effects of the bias is reduced for very large patch sizes (see Figure 19).
21

---

Published as a conference paper at ICLR 2022
Entropy Degradation
Accurscy Degracation
Simllantt Original
1.0
From Ongiral Protiction
Unchanngd
0.8-
ResNet-50 50
0.6
0.6
Random Order
0.4
0.5
Most Salient first
Least Salient First
1-
200
400
600
200
400
600
800
200
600
800
Number Patches Removed
Number Patchee Removed
Number Patchee Removed
(a) 8 8 patches
Entropy Degradation
Accuracy Diege
Similarkty Original
1.0
from Ongiaa Prediction
unchanged
0.8
0.7-
ResNet-50 50
Entrpy
0.6
Random Order
0.4
Most Salient first
0.5
Least Salient First
0.2
1-
0.4
100
200
100 Number Patches Removed
100
Number Patches Removed
Number Patches Removed
(b) 14
14 patches
Entropy Degradation
Accuracy Degg adution
Similarky Original
1.0
Predetton
Fraction ofLat da unchanged
0.8
0.75
ResNet-50 50
0,70
0.6
0.65
0.4
0.60
Most Salient first
Random
0.55
Least Salient First
0.2
1-
0,50
0.0
0.45
40
20
20
60
Number Putches Removed
Putches
Putches
(c) 28
28 patches
Entropy Degradation
Accuracy Degi adation
Similarky Original Prediction
1.0
Unchanges
0.8
0.75
0.70
ResNet-S0
0.6
0.65
0.4
0.60
Most Salient First
0.55
Least Salient First
0.2
0,50
0.0
Putches Removed
Removed
(d) 32
32 patches
Class Entropy Degradation
Accurscy Degracation
Werdlet Simllartt Original Prediction
1.0
0.8
0.70
0.75
ResNet-S0
0.6
0.65
3-
0.4
0.60
Most Saliant First
Order
0.55
Salient
0.2
0.50
0.0
calch Remnooed
Putches
Itmrved
56
56 patches
Figure 19; Using different baseline colors for masking pixels.
22

---

Published as a conference paper at ICLR 2022
C.6 USING SUPERPIXELS INSTEAD OF PATCHES
Thus far, we have used square patches. What if we instead use superpixels? In this section, we compute the SLIC segmentation of superpixels (Achanta al., 2010). In order keep the super- pixels roughly the same size (and of a more similar size as the patches in the paper), we consider the images having more than 130 superpixels. We display examples of the superpixels in Figure 20a.
We then repeat our experiments from Section 3, and analyze our models' predictions after removing 50 superpixels in different orders (blacking out for ResNets and dropping tokens for ViTs). As described in Section B, we conservatively drop all tokens for ViTs that contain any pixels that should have been removed. We find that ResNets are significantly impacted by missingness bias when masking out superpixels. As in the case of patches, dropping tokens through the VIT substantially mitigates the missingness bias.
(a) Examples of superpixel segmentations.
Eniropy Degradetion
Accurary Degradation
Deiginal
6.5
1.0
0.80
6.0
0.8
ResNet 50 ViT-s
0.70
5.5
0.6
Random Order
0.4
Salient First
5.0
0.53
Least Sallent First
0.2
4.5-
0.45
40
20
40
20
Number of Superpizels Removed
Number of Superpteets Removed
Number Superpiees kods Removed
(b) Mcasuring the impact of removing superpixels through missingness approximations.
Figure 20: Using SLIC superpixels instead of patches
C.7 COMPARISON OF DROPPING TOKENS VS BLACKING OUT PIXELS FOR VITS
We compare the effect of implementing missingness by dropping tokens to simply blacking out pixels for ViTs. Figure 21, shows condensed version of the experiments we did previously this section, but now including an additional baseline which is a ViT-S with blacking out pixels instead of dropping tokens. We find that using either of the ViTs instead of the ResNet significantly mitigates missingness bias on all three metrics. However, dropping tokens for ViT's mitigates missingness bias more effectively than simply blacking out pixels.
Class Entropy Degradation
Accurboy Degradation
WordNet Sim larity to Original Prediction
WIP 5m From OHignal Predi-tom
0.8
0.8
ResNet-50
0.7
0.6
VIT-S (8 [Blacking Piceis)
3-
0.4
0.6
Random Order
Most Salient first
0.2
Least Salient First
1-
0
50
100
150
200
50
100
150
200
50
160
150
200
Number of Patches Remaved
Number of Patches Removed
Number
Patches Removed
Figure 21: We compare dropping tokens vs blacking out pixels for ViTs.
23

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTS (SECTION 4)
D.1 EXAMPLES OF LIME
In this section, display further examples of LIME explanations for VIT-S, ResNet-50, and ResNet-50 retrained with missingness augmentations (Figure 22). As we explain in Section 4, LIME relies heavily on the notion of missingness, and can be subject to missingness bias. We note that the ViT-S and retrained ResNets qualitatively have more human-aligned LIME explanations (by highlighting patches the foreground over patches in the background) compared to standard ResNet. While human-alie alimameen does not guarantee that the LIME explanation is good (the model might be relying on non-aligned features), we do see substantial difference in the explanations of models robust to the missingness bias (ViTs and retrained ResNet) and models suffering from this bias (standard ResNet).
ResNet-50 (with miss. augmentations)
ResNet-50 (with miss. augmentations)
VIT-S
ResNet-50
VTT-S
ResNet-50
Figure 22: Examples of LIME explanations
24

---

Published as a conference paper at ICLR 2022
D.2 TOP-K K ABLATION TEST WITH SUPERPIXELS.
We repeat the top-k ablation test in Section 4, using superpixels instead of patches (Figure 23). The setup for superpixels is that same that described in Appendix C.6. After generating LIME expla- nations for ViT and ResNet-50, we evaluate these explanations using the top-k ablation test. As we found for 16 16 patches, the explanations when evaluating with ResNet than when evaluating for VIT: even masking features according to the random explanations rapidly flips the predictions. We do find that masking random superpixels seems to have greater effect on ViTs than masking random 16 x 16 patches: this is likely because the superpixels are on average larger.
less distinguishable
Evaluating Net-Ne
Evaluating
1.0
1.0-
Fraction of Predictions Unchanged
Fraction of Predictions Unchanged
Explanation From
0.6
0.6
Res Res-et-50
0.4
0.4
Random
0.2
0.2
0
10
30
40
50
10
20
40
50
Number of Features Removed
Number of Features Removed
Figure 23: Top ablation test using superpixels instead of patches.
25

---

Published as a conference paper at ICLR 2022
D.3 EFFECTS OF MISSINGNESS BIAS ON LEARNED MASKS
In this section, we consider different model debugging method (Fong & Vedaldi (2017)) which also relies missingness. In this method, a minimal mask directly optimized for each image. We implement this method at the granularity of 16 x 16 patches.
Model Debugging through a learned mask (Fong & Vedaldi (2017)) Specifically x be the input image, .M be the model, be the model's original prediction æ, and be baseline image (in our case, blacked out pixels). The method optimizes for m, a 14 x 14 grid of weights between and which assigns importance to each patch. We define the perturbation via of the input image and the baseline image for each patch (plus some normally distributed noise e):
linear combination
fa, m) x upsample(m) b (1 upsample(m.)
Then the optimal m is computed as:
= arg min A || - m|[1 + do bellm + (M((s,m))).
with A 0.01, 0.2, 3, (0,00.4))
The optimal m is computed through backpropagation, and can then be treated as a model ex- planation. Parameters and implementation were adapted from the implementation https: ///ithub.cco com/ htt.://.ccoo/accgg///oo/.// http://yttoooeepplppeeeo black-ll------ box.
Assessing the impact of missingness bias. Since m must backpropagated, we cannot leverage the drop tokens method for ViTs (which would require for each patch to be either 0 or 1). How- ever, we generate explanations for the ResNet-50 in order to examine the impact missingness bias. As in Section 4, evaluate the explanation generated by this method alongside random baseline and the LIME explanation for that ResNet using the top-K ablation test (Figure 24). As is the case for the LIME explanations, missingness bias renders the explanations generated by this method indistinguishable from random.
Evaluating Model: ResNet-50
1.0
Explanation From LIME
Fraction of Predictions Unchanged
0.8
Fong & Fedaldi, 2017
Random
0.6
0.4
0.2
0.0
10
20
30
40
50
Number of Features Removed
(a) Top-K ablation test
(b) Examples of the generated expla- nations
Figure 24: (a) Top K ablation test evaluated on a ResNer-50. We evaluate explanations generated by LIME, Fong Vedaldi (2017), and random baseline. Due missingness bias, the explanations are indistinguishable from random (b) Examples of explanations generated by Fong & Vedaldi (2017).
26

---

Published as a conference paper at ICLR 2022
E OTHER DATASETS
While our main paper largely considers the setting of ImageNet, we include here few results on other datasets.
E.I MS-COCO
MS-COCO (Lin al. (2014)) is object recognition dataset with 80 object recognition categories that provides bounding box annotations for each object. We consider the multi-label task of object tagging, where the model predicts whether each object class present in the dataset. We train object tagging models using a ResNet-50 and ViT-S, with an Asymmetric Loss as nen-Baruch al. (2020). An object predicted "present" the outputted logit is above some threshold.
We study the setting of removing people from images using missingness. In particular, we seek remove the image regions contained inside "person" bounding box that is not contained in bounding box for another non-person object. Examples of removing people from the image can be found in Figure 25. We then seek to check whether removing the person affected the model predictions for other, non-person object classes.
Specifically, we consider the 21,634 images in the MS-COCO validation set that contain people. For each image, we evaluate our model on the original image and the image with the person re- moved (blacking out for the ResNet-50 and dropping the token for the ViT). Then, measure the consistency of the non-person predictions, we compute the Jaccard similarity for the set of predicted objects (excluding the person class) before and after masking. We plot the average similarity over all the images for different prediction thresholds in Figure 25.
We find that the VIT more consistently maintains the predictions of the non-person object classes when masking out people. In contrast, masking out people for the ResNet is more likely to change the predictions for other object classes.
0.70
ResNet-50
before/after removing person
0.65
VIT-S
Jaccard sim of predictions
0.60
0.55
0.50
0.45
0.40
0.2
0.4
0.6
0.8
1.0
Threshold
(a) Consistency of non-person prediction after removing people from images.
(b) Examples of removing people from MS COCO images
Figure 25: (a) Average Jaccard similarity of the set of non-person predictions before and after re- moving all people from the image. We plot over prediction thresholds for the tagging task. (b) Examples of removing people from MS-COCO images.
27

---

Published as a conference paper at ICLR 2022
E.2 CIFAR-10
We consider the setting of CIFAR-10 (Krizhevsky (2009)). Specifically, we train ResNet-50 and VīT-S the CIFAR-10 dataset upsampled to 224x224 pixels. We start training from the ImageNet checkpoints used throughout this paper. This step is necessary for ensuring high accuracy when training ViTs on CIFAR-10 (Touvron al. (2020)). We then consider how randomly removing 16x16 patches from the upsampled CIFAR images changes the prediction. Similarly to the case of ImageNet, we find that the ResNet-50 more rapidly changes its prediction as random parts of the image are masked, while the ViT-S maintains its prediction even as large parts of the image are removed.
Accuracy Degradation
Fraction of Labels Unchanged
0.8
ResNet-50
0.6
VIT-S
0.4
0.2
50
100
150
200
Number of Patches Removed
Figure 26: We plot the fraction of images where the prediction does not change as image regions are removed for the CIFAR-10 dataset.
28

---

Published as a conference paper at ICLR 2022
F RELATIONSHIP TO ROAR
Here, we present more details about the ROAR experiment of Section 3.
F.1 OVERVIEW ON ROAR
Evaluating feature attribution methods requires the ability to remove features from the input to assess how important these features are to the model's predictions. To do so properly, Hooker et al. (2018) argue that re-training (with removing pixels) is required so that images with removed features stay in-distribution. Their argument holds since machine learning models typically assume that the train and the test data comes from a similar distribution.
So, they propose RemOve and Retrain (ROAR) where models (of the exact same architecture) are retrained such that random pixels are blacked out during training. The intuition that this way, removing pixels do not render images aut-of-distribution. Overall, they were able to better assess how much removing information from the image affects the predictions of the model using those retrained surrogate models.
The authors of ROAR list several downsides for their approach though. In particular, retraining models can be computationally expensive. More pressingly, the retrained model is the same model that they analyze, but instead a surrogate with a substantially different training procedure: any feature attribution or model debugging result inferred from the retrained model might not hold for the original model. Given these downsides, is retraining always necessary?
F.2 VITS DO NOT REQUIRE RETRAINING
Here, we show that retraining is not always necessary: indeed for ViTs, we do not need to retrain- ing to be able properly evaluate feature attribution methods. While ROAR in (Hooker et al., 2018) dealt with blacking out features on a per-pixel level, we adapt their approach for masking out larger contiguous regions (like patches). If we apply missingness approximations during training as in ROAR, missingness approximations are now in-distribution, and thus would likely mitigate the observed biases.
We retrain a ResNer-50 and a ViT-S by randomly removing 50% of patches during training (through blacking pixels for the ResNet-50 and dropping tokens for the ViT-S). Our goal is to compare the behavior of each model to its retrained counterpart. If retraining does not change the model's s behavior when missingness approximations are applied, retraining would be unnecessary. and we can instead confidently use the original model. In Figure 5, we measure the fraction of images where the model changes its prediction as we remove image features for both the standard and retrained models. We find that, while there is significant gap in behavior between the standard and retrained CNNs, the standard and retrained ViTs behave largely the same..
This result indicates that, while retraining is important when analyzing CNNs, it is unnecessary for ViTs: we can instead intervene on the original model. We thus avoid the expense of training the augmented models, and can perform feature attribution the actual model instead of proxy.
29


Reviews:
[{"review_id": "sghrtUx-CkS", "reviewer": "Reviewer_hiFn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\ni) The authors have analyzed a very relevant problem in vision, especially explainable AI as applied in vision. Perturbing inputs (images) can be a strong tool towards finding importance of regions of input, however the authors showed through a number of well designed experiments and evaluation metrics that, if wrongly, applied this can be harmful too.\nii) The experiments are well thought and executed. A natural progress is seen in going about the experiments starting with trying to find if masking image regions make the classifier look for only a few classes of images, flipping class decision is good if important regions are masked, but whether class decisions are flipped even with random or unimportant region removal, taking LIME as a case study, the authors also tried to see if changing baseline colors in the mask region results in consistent explanations and so on.\niii) This work definitely does open more doors into research of better debugging tools rather than removing regions from images.\niv) The paper is clearly written.\n\nWeaknesses:\ni) The authors have shown all their results in section 3 by only blackenning rather than any other baseline value. Blackenning is majorly known to be an incorrect replacement for the removed pixels [a, b]. Other common methods are blurring, graying (using the mean values) or random noise. It would be interesting to see if at all these baselines also show the same effects of missingness bias.\nii) The authors have used LIME as the case study for generating explanations [sec 4]. Some relevant approaches (e.g., [a, b, c]) provide much smoother saliency maps which might be worth trying especially to check inconsistency and indistinguishability of explanations as well as the effect dependence on maze/crossword like pattern [Figs. 2 and 3].\niii) It would have been good if authors could show the results on more models like EfficientNet or even VGG instead of only ResNet. This would show that this problem exists in CNNs in general and is not specific to only ResNet.\niv) (Fig. 8) Does it make sense to find a measure of agreement for different baseline colors in ViT. These baseline colors are not used at all while making ViT prediction for these masked images. So, it is quite natural that all these so called ViT variations will give exactly the same features with same importance for all these baseline methods. Also it would be good to know what are the 8 different baseline colors?\nv) Instead of being an weakness, this is rather an academic query: There is no major alternative to the current method followed for removal of regions in images for CNNs. This problem is not talked of much.  After analyzing an already existing problem, it would have been great if the authors would have talked about any method to overcome it. Shifting to transformer based architecture may not be a solution for explaining and debugging CNNs.\n\n[a] Petsiuk et. al., RISE: Randomized Input Sampling for Explanation of Black-Box Models, BMVC 2018\n[b] Fong et. al., Interpretable Explanations of Black Boxes by Meaningful Perturbations, ICCV 2017\n[c] Fong et. al., Understanding Deep Networks via Extremal Perturbations and Smooth Masks, ICCV 2019", "summary_of_the_paper": "The paper discusses that it is common in Computer Vision debugging and explainability techniques to remove image regions to attribute different regions of the image to the decision of a classification model. Although such removal (of words) can be beneficial for Natural Language model debugging, it adds an additional bias in Computer Vision. This is because removing regions implies replacing the corresponding pixels with some baseline values e.g., black color, random intensities, average pixel values etc. The paper shows that irrespective of which part of the region is being masked (i.e., removing original image pixel values and replacing with baseline values), masking small portions of image can lead to CNNs predicting incorrectly and unreliably. The authors show that the CNN based classifiers seem to rely on the “masking pattern” to make the prediction, rather than the remaining (unmasked) portions of the image. In fact, even after removing some image regions randomly, the output distribution gets highly skewed towards a few classes e.g., crossword, jigsaw puzzle etc. Using LIME as a case study, it shows that missingness bias can lead to inconsistent and indistinguishable explanations. Moreover, this bias can be overcome if the model is trained with suitable augmentations that remove regions while training.\nThe paper illustrates how using Visual Image Transformers is a better natural choice as these models allow actual removal of image regions rather than replacement with baseline values. Hence, these issues seen in CNNs due to missingness bias is not prevalent in ViTs.", "summary_of_the_review": "Overall, this work is good showing crisp thinking and analysis of the experiment design. However, few more experimentation would have been better to have. Some such experiments are listed in detail in the weaknesses section above. The approximation of “missingness of pixels” by supposed to be “meaningless pixels” is a longstanding problem in vision and researchers are aware of this. This work definitely will help progress the research in this. However, I would like to see the authors to discuss some possible solutions to this problem so that the work does not look like an illustration of an already known problem in Computer Vision only.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XS6eS5JLUo", "reviewer": "Reviewer_SQBL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of this paper is interesting, the paper is well-written and it uses various visualizations to demonstrate the issue on CNNs/ResNets. However, my concerns are the following: \n\n-  The paper claims that a ‘hole’ cannot be left in the image in convolutional networks. This is not clear why it is the case, since a convolutional network uses a sliding window to convolve each pixel with the kernel of the layer. Then, each output in the feature map is exactly the output that corresponds to a specific positioning of the kernel. So, similarly to the way that we can ‘leave out’ patches of the image, we could implement the sliding window to skip some patches, so that the feature map (and every next feature map) would be exactly zero in those dimensions. \n\n- The previous point of skipping some parts of the image with the sliding window, makes me wonder what the results on ResNet would be in this case; currently, the paper showcases poor results on ResNet, but it seems plausible that it was not optimized for this task. \n\n- The paper experiments with patches that are only rectangular and size of 14x14 or bigger (appendix C.6). How would it work in the case of non-rectangular patches and or small ones? \n\n- I am not convinced about the **utility of the proposed method**; ViT is a state-of-the-art method only when it comes to extremely large-scale image recognition; in fact, the authors of ViT recognize that if only trained on datasets like ImageNet it can perform similarly to ResNet. In other words, to learn ViT, we need a huge annotated dataset, which practically means that it can be used for debugging imagenet (or a similar dataset) but it is hard to be used for any other dataset. This calls into question whether it can be a general tool, or something useful only for a handful of ImageNet-like datasets. For instance, **can the proposed tool be used for debugging classifiers on medical imaging or similar constrained-size datasets** that are not related to ImageNet? \n\n- Another limitation of the method is that practically only a single task and mainly ImageNet (?) datasets are used for the evaluation. The contribution is based on the empirical validation across a board of tasks and across datasets, which at the moment is not the case. How can the proposed method generalize to different datasets, e.g., ChestRay or food101 datasets? \n\n- Currently, only ResNet is used as a baseline, while previous work [Sturmfels, 2020] used Inception v4 instead. Also, EfficientNet or similar networks have recently been performing well in terms of image recognition. It would have been better to showcase that those also suffer from the same issue as ResNet. \n", "summary_of_the_paper": "The authors focus on the problem of model debugging (for image recognition). They identify that the proposed tools (that rely on CNNs and ResNets) might suffer from the ‘missingness’ issue, i.e., the absence of features due to masking objects of interest. The authors exhibit how the method of masking pixels/patches can lead to the missingness bias and this can happen even if the masked values are replaced with some other 'dummy' values. To mitigate that, they propose to use a recent transformer since it does not use convolutions, but linearized patches as inputs. Then, they demonstrate that the proposed method can simply 'skip' those patches that are masked and it does not result in a skewed result. \n", "summary_of_the_review": "The paper makes an observation on the absence of features for debugging purposes and argues that ViT should be preferred over standard CNNs. The paper exhibits the use case on ImageNet, but it does not demonstrate how this can be useful for diverse datasets, or even the utility of the method in various cases (e.g. non-rectangular, small patches) and tasks. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7AMalxzdlyF", "reviewer": "Reviewer_sYZg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is well written and the authors provide clear explanations at each stage whilst analyzing the effect of various missingness methods across convolutional neural networks and the Vision Transformer architecture. The authors build on the compelling case that when using debugging methods such as LIME, that iteratively block out features (or regions of the input image in the vision case) to analyze the behavior of convolutional neural networks with and without those features, the bias introduced by the blocked out regions (via semi-arbitrary manipulation such as blackout, blurring, etc) can dramatically impact the behavior of the model. As a solution to this problem, the authors analyze (and compare with CNNs) the use of a VIT architecture to simply drop target regions instead of blocking them out. For VIT architectures, dropping target regions is a clever solution that was demonstrated to produce superior explainability and debugging results when compared with a CNN and blackout or blurring.\n\nWhile overall I found the paper to provide meaningful insight and analysis, the paper could be improved through further analysis of adversarial robustness of CNNs vs VITs. CNNs have been demonstrated to overfit non robust features in the feature space (features that allow the classifier to achieve high accuracy but have no or little relation to the object class), whereas VITs have as of late been shown to be more robust to such noise. While some analysis was conducted when analyzing the retraining of models with missingness augmentation, given that retraining proved to significantly improve the results from the CNN, I would encourage the authors to more thoroughly analyze whether the superior capability of the proposed VIT token drop method is a result of this behavior rather than blackout and blurring being inferior/problematic methods.", "summary_of_the_paper": "This paper analyzes the bias introduced when regions of images are modified in order to measure the effect of a model with and without those features present in the image, for model debugging purposes. The authors compare different removal methods, and subsequently analyze how a Vision Transformer architecture might be used instead of a CNN to actually drop features rather than artificially removing them.", "summary_of_the_review": "Overall the work provides meaningful analysis and insight into the use of debugging methods such as LIME with CNN and VIT architectures, and shows the superiority of token dropping in VIT to blackout, blurring or other feature removal methods in CNNs. Additional analysis by the authors into feature robustness and retraining with CNNs vs VIT to determine whether the superior explainability is due to the regions not being considered in the token drop method as argued by the paper, or whether the propensity for more robust features in VITs is the reason.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work identifies an interesting bias that can occur when applying occlusion based interpretability methods to debug image classifiers. For context, the motivation behind many of these methods is that by occluding various parts of the image, one can ask counterfactuals such as "what would the model have predicted if this object were not present in the image"? However, the authors note that when occluding pixels, classifiers are still functions of the occlusions themselves, so this process may introduce a bias as a result. This is most clearly demonstrated in Figure 2 where a convolutional architecture classifies various occluded images as "jigsaw" or "crossword puzzle", arguably due to the fact that scattered patch based occlusions resemble crossword puzzles. The authors then demonstrate that ViT models can be modified in a way to ask the above counter-factual in a more principled manner---namely by dropping image tokens within the transformer model, the resulting function doesn't take any occluded pixels as input. Reviewers all found the analysis quite insightful, and did not find any significant flaws in the experiments. During the rebuttal, the authors added numerous experiments to address concerns raised by reviewers regarding lack of datasets for which the method was run on. Unfortunately, only one of the reviewers acknowledged the rebuttal and did not raise their score citing doubts that the method may not work well on datasets with differing image statistics (e.g. medical imaging). After reading all of the reviews and rebuttal, the AC feels the authors have adequately addressed the most pressing reviewer concerns, and finds the presented analysis sufficient to warrant acceptance.
Decision: Accept (Poster)