Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the bias introduced when regions of images are modified in order to measure the effect of a model with and without those features present in the image, for model debugging purposes. The authors compare different removal methods, and subsequently analyze how a Vision Transformer architecture might be used instead of a CNN to actually drop features rather than artificially removing them.
main_review: Overall the paper is well written and the authors provide clear explanations at each stage whilst analyzing the effect of various missingness methods across convolutional neural networks and the Vision Transformer architecture. The authors build on the compelling case that when using debugging methods such as LIME, that iteratively block out features (or regions of the input image in the vision case) to analyze the behavior of convolutional neural networks with and without those features, the bias introduced by the blocked out regions (via semi-arbitrary manipulation such as blackout, blurring, etc) can dramatically impact the behavior of the model. As a solution to this problem, the authors analyze (and compare with CNNs) the use of a VIT architecture to simply drop target regions instead of blocking them out. For VIT architectures, dropping target regions is a clever solution that was demonstrated to produce superior explainability and debugging results when compared with a CNN and blackout or blurring.

While overall I found the paper to provide meaningful insight and analysis, the paper could be improved through further analysis of adversarial robustness of CNNs vs VITs. CNNs have been demonstrated to overfit non robust features in the feature space (features that allow the classifier to achieve high accuracy but have no or little relation to the object class), whereas VITs have as of late been shown to be more robust to such noise. While some analysis was conducted when analyzing the retraining of models with missingness augmentation, given that retraining proved to significantly improve the results from the CNN, I would encourage the authors to more thoroughly analyze whether the superior capability of the proposed VIT token drop method is a result of this behavior rather than blackout and blurring being inferior/problematic methods.
summary_of_the_review: Overall the work provides meaningful analysis and insight into the use of debugging methods such as LIME with CNN and VIT architectures, and shows the superiority of token dropping in VIT to blackout, blurring or other feature removal methods in CNNs. Additional analysis by the authors into feature robustness and retraining with CNNs vs VIT to determine whether the superior explainability is due to the regions not being considered in the token drop method as argued by the paper, or whether the propensity for more robust features in VITs is the reason.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors focus on the problem of model debugging (for image recognition). They identify that the proposed tools (that rely on CNNs and ResNets) might suffer from the ‘missingness’ issue, i.e., the absence of features due to masking objects of interest. The authors exhibit how the method of masking pixels/patches can lead to the missingness bias and this can happen even if the masked values are replaced with some other 'dummy' values. To mitigate that, they propose to use a recent transformer since it does not use convolutions, but linearized patches as inputs. Then, they demonstrate that the proposed method can simply 'skip' those patches that are masked and it does not result in a skewed result. 

main_review: The motivation of this paper is interesting, the paper is well-written and it uses various visualizations to demonstrate the issue on CNNs/ResNets. However, my concerns are the following: 

-  The paper claims that a ‘hole’ cannot be left in the image in convolutional networks. This is not clear why it is the case, since a convolutional network uses a sliding window to convolve each pixel with the kernel of the layer. Then, each output in the feature map is exactly the output that corresponds to a specific positioning of the kernel. So, similarly to the way that we can ‘leave out’ patches of the image, we could implement the sliding window to skip some patches, so that the feature map (and every next feature map) would be exactly zero in those dimensions. 

- The previous point of skipping some parts of the image with the sliding window, makes me wonder what the results on ResNet would be in this case; currently, the paper showcases poor results on ResNet, but it seems plausible that it was not optimized for this task. 

- The paper experiments with patches that are only rectangular and size of 14x14 or bigger (appendix C.6). How would it work in the case of non-rectangular patches and or small ones? 

- I am not convinced about the **utility of the proposed method**; ViT is a state-of-the-art method only when it comes to extremely large-scale image recognition; in fact, the authors of ViT recognize that if only trained on datasets like ImageNet it can perform similarly to ResNet. In other words, to learn ViT, we need a huge annotated dataset, which practically means that it can be used for debugging imagenet (or a similar dataset) but it is hard to be used for any other dataset. This calls into question whether it can be a general tool, or something useful only for a handful of ImageNet-like datasets. For instance, **can the proposed tool be used for debugging classifiers on medical imaging or similar constrained-size datasets** that are not related to ImageNet? 

- Another limitation of the method is that practically only a single task and mainly ImageNet (?) datasets are used for the evaluation. The contribution is based on the empirical validation across a board of tasks and across datasets, which at the moment is not the case. How can the proposed method generalize to different datasets, e.g., ChestRay or food101 datasets? 

- Currently, only ResNet is used as a baseline, while previous work [Sturmfels, 2020] used Inception v4 instead. Also, EfficientNet or similar networks have recently been performing well in terms of image recognition. It would have been better to showcase that those also suffer from the same issue as ResNet. 

summary_of_the_review: The paper makes an observation on the absence of features for debugging purposes and argues that ViT should be preferred over standard CNNs. The paper exhibits the use case on ImageNet, but it does not demonstrate how this can be useful for diverse datasets, or even the utility of the method in various cases (e.g. non-rectangular, small patches) and tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper discusses that it is common in Computer Vision debugging and explainability techniques to remove image regions to attribute different regions of the image to the decision of a classification model. Although such removal (of words) can be beneficial for Natural Language model debugging, it adds an additional bias in Computer Vision. This is because removing regions implies replacing the corresponding pixels with some baseline values e.g., black color, random intensities, average pixel values etc. The paper shows that irrespective of which part of the region is being masked (i.e., removing original image pixel values and replacing with baseline values), masking small portions of image can lead to CNNs predicting incorrectly and unreliably. The authors show that the CNN based classifiers seem to rely on the “masking pattern” to make the prediction, rather than the remaining (unmasked) portions of the image. In fact, even after removing some image regions randomly, the output distribution gets highly skewed towards a few classes e.g., crossword, jigsaw puzzle etc. Using LIME as a case study, it shows that missingness bias can lead to inconsistent and indistinguishable explanations. Moreover, this bias can be overcome if the model is trained with suitable augmentations that remove regions while training.
The paper illustrates how using Visual Image Transformers is a better natural choice as these models allow actual removal of image regions rather than replacement with baseline values. Hence, these issues seen in CNNs due to missingness bias is not prevalent in ViTs.
main_review: Strengths:
i) The authors have analyzed a very relevant problem in vision, especially explainable AI as applied in vision. Perturbing inputs (images) can be a strong tool towards finding importance of regions of input, however the authors showed through a number of well designed experiments and evaluation metrics that, if wrongly, applied this can be harmful too.
ii) The experiments are well thought and executed. A natural progress is seen in going about the experiments starting with trying to find if masking image regions make the classifier look for only a few classes of images, flipping class decision is good if important regions are masked, but whether class decisions are flipped even with random or unimportant region removal, taking LIME as a case study, the authors also tried to see if changing baseline colors in the mask region results in consistent explanations and so on.
iii) This work definitely does open more doors into research of better debugging tools rather than removing regions from images.
iv) The paper is clearly written.

Weaknesses:
i) The authors have shown all their results in section 3 by only blackenning rather than any other baseline value. Blackenning is majorly known to be an incorrect replacement for the removed pixels [a, b]. Other common methods are blurring, graying (using the mean values) or random noise. It would be interesting to see if at all these baselines also show the same effects of missingness bias.
ii) The authors have used LIME as the case study for generating explanations [sec 4]. Some relevant approaches (e.g., [a, b, c]) provide much smoother saliency maps which might be worth trying especially to check inconsistency and indistinguishability of explanations as well as the effect dependence on maze/crossword like pattern [Figs. 2 and 3].
iii) It would have been good if authors could show the results on more models like EfficientNet or even VGG instead of only ResNet. This would show that this problem exists in CNNs in general and is not specific to only ResNet.
iv) (Fig. 8) Does it make sense to find a measure of agreement for different baseline colors in ViT. These baseline colors are not used at all while making ViT prediction for these masked images. So, it is quite natural that all these so called ViT variations will give exactly the same features with same importance for all these baseline methods. Also it would be good to know what are the 8 different baseline colors?
v) Instead of being an weakness, this is rather an academic query: There is no major alternative to the current method followed for removal of regions in images for CNNs. This problem is not talked of much.  After analyzing an already existing problem, it would have been great if the authors would have talked about any method to overcome it. Shifting to transformer based architecture may not be a solution for explaining and debugging CNNs.

[a] Petsiuk et. al., RISE: Randomized Input Sampling for Explanation of Black-Box Models, BMVC 2018
[b] Fong et. al., Interpretable Explanations of Black Boxes by Meaningful Perturbations, ICCV 2017
[c] Fong et. al., Understanding Deep Networks via Extremal Perturbations and Smooth Masks, ICCV 2019
summary_of_the_review: Overall, this work is good showing crisp thinking and analysis of the experiment design. However, few more experimentation would have been better to have. Some such experiments are listed in detail in the weaknesses section above. The approximation of “missingness of pixels” by supposed to be “meaningless pixels” is a longstanding problem in vision and researchers are aware of this. This work definitely will help progress the research in this. However, I would like to see the authors to discuss some possible solutions to this problem so that the work does not look like an illustration of an already known problem in Computer Vision only.
review_title: null

Meta-review: This work identifies an interesting bias that can occur when applying occlusion based interpretability methods to debug image classifiers. For context, the motivation behind many of these methods is that by occluding various parts of the image, one can ask counterfactuals such as "what would the model have predicted if this object were not present in the image"? However, the authors note that when occluding pixels, classifiers are still functions of the occlusions themselves, so this process may introduce a bias as a result. This is most clearly demonstrated in Figure 2 where a convolutional architecture classifies various occluded images as "jigsaw" or "crossword puzzle", arguably due to the fact that scattered patch based occlusions resemble crossword puzzles. The authors then demonstrate that ViT models can be modified in a way to ask the above counter-factual in a more principled manner---namely by dropping image tokens within the transformer model, the resulting function doesn't take any occluded pixels as input. Reviewers all found the analysis quite insightful, and did not find any significant flaws in the experiments. During the rebuttal, the authors added numerous experiments to address concerns raised by reviewers regarding lack of datasets for which the method was run on. Unfortunately, only one of the reviewers acknowledged the rebuttal and did not raise their score citing doubts that the method may not work well on datasets with differing image statistics (e.g. medical imaging). After reading all of the reviews and rebuttal, the AC feels the authors have adequately addressed the most pressing reviewer concerns, and finds the presented analysis sufficient to warrant acceptance.
Decision: Accept (Poster)