Removing input features via a generative model to explain their attributions to classifier's decisions
Authors: Chirag Agarwal
Dan Schonfeld
Anh Nguyen
Venue:  (2020)
Keywords: attribution maps
generative models
inpainting
counterfactual
explanations
interpretability
explainability

Abstract: Interpretability methods often measure the contribution of an input feature to an image classifier's decisions by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples. Instead, we propose to integrate a generative inpainter into three representative attribution map methods as a mechanism for removing input features. Compared to the original counterparts, our methods (1) generate more plausible counterfactual samples under the true data generating process; (2) are more robust to hyperparameter settings; and (3) localize objects more accurately. Our findings were consistent across both ImageNet and Places365 datasets and two different pairs of classifiers and inpainters.

Under review as a conference paper at ICLR 2020
REMOVING INPUT FEATURES VIA A GENERATIVE MODEL TO EXPLAIN THEIR ATTRIBUTIONS TO AN IM- AGE CLASSIFIER'S S DECISIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Interpretability methods often measure the contribution of an input feature to an image classifier's decisions by heuristically removing it via e.g. blurring, adding noise, or graying out, which often produce unrealistic, out-of-samples. Instead, we propose to integrate generative inpainter into three representative attribution methods to remove an input feature. Compared to the original counterparts, our methods (1) generate more plausible counterfactual samples under the true data generating process; (2) are more robust to hyperparameter settings; and (3) local- ize objects more accurately. Our findings were consistent across both ImageNet and Places365 datasets and two different pairs of classifiers and inpainters.
INTRODUCTION
Explaining classifier's outputs given certain input is increasingly important, especially for life- critical applications (Doshi-Veler Kim 2017). A popular means for visually explaining an image classifier's decisions is an attribution map i.c. heatmap that highlights the input pixels that are the evidence for and against the classification outputs (Montavon et al. 2018). To construct an attribution map, many methods approximate the attribution value of input region by the classi- fication probability change when that region is absent i.e. removed from the image. That is, most perturbation-base based attribution methods implement the absence of an input feature by replacing with (a) mean pixels; (b) random noise; or (c) blurred versions of the original content. While re- moving an input feature to measure its attribution is a principle method in causal reasoning, the existing removal (i.e. perturbation) techniques often produce out-of of-dissributtin images (Fig. Ip.d) i.c. a type of adversarial example, which (1) we found produce heatmaps that are sensitive to hyperparameter settings; and (2) questions the correctness of the heatmaps (Adebayo et al. 2018
To combat these two issues, we propose to harness a state-of-the-art generative inpainting model (hereafter, inpainter) to remove features from an input image and fill in with content that is plausible under the true data distribution. We test our approach on three representative attribu- tion methods of Sliding-Pate Mcaningful-Perrtnn Perturbation (MP) (Fong Vedaldi 2017)) across two large-scale datasets ImageNet (Russakovsky al.) 2015) and Places365 (Zhou et al. 2017). For each dataset, pair of pre-trained image classifiers and inpainters. Our main findings are:
(SP) (Zeiler & Fergus 2014). LIME (Ribeiro et al.) (016), and
separate
1. Blurring graying out the object in photo yields images that are up 3 times more recognizable to classifiers and remain more similar to the original photo (via MS-SSIM and LPIPS) than the images whose objects were removed via inpainting (Sec. 4.2
2. Attribution methods with an inpainter produces (1) more plausible perturbation samples; (2) attribution maps that perform on par better than their original counterparts three existing benchmarks: object localization, Insertion, and Deletion (see Sec. 4.4); (3) expla- nations that are more robust to hyperparameter changes (Sec. 4.3
1 All our code will be available on github.
Our positive results here and contrast with a previous result by Chang et al. 2019

---

Under review as a conference paper at ICLR 2020
(a) Real
(b) SP
(c) SP-G
(d) LIME
LIME-G
(f) MP
MP-G
samples
freight car 0.929
0.391
0.840
0.003
0.898
0.002
0.002
heatmaps
Figure 1: Three attribution methods, SP (Zeiler & Fergus 2014). LIME (Ribeiro et al. 2016). and MP (Fong Vedaldi 2017). often produce unrealistic, -fitt--diittibuttion perturbation samples. Top row: SP slides 29 x 29 gray patch across the image (b). LIME grays out set of random superpixels (d). MP blurs out almost the entire image (f). contrast, learned inpainter integrated into these methods produces realistic samples for the same perturbation masks, here, completing the freight car (c), completing the background (e), and removing the car from the scene (g). Note that the freight car class probability is reduced by 57% (i.e. from 0.929 to 0.391) when only the top of the car is occluded (b). However, is reduced by ~000 down to 0.003 when the car is still present but the background is unnaturally masked out (d). Since the inpainted samples are more realistic, the probability drops are often less (c & e) and substantial only when the object is removed completely (g). Bottom row: the inpainted samples yield heatmaps that, in overall, outperform the original methods on the object localization task. Here, our heatmaps (SP-G, LIME-G, and MP-G) are less noisy and more focused on the object.
RELATED WORK
Attribution methods can be categorized into two main classes: (1) white-box and (2) black-box.
White-box Given the access to the network architecture and parameters, attribution maps can be constructed analytically from (a) the gradients of the output w.r.t. the input image (Simonyan al.) 2013). (b) the class activation map in cully-convolutiooal neural networks (Zhou et al.) 2016). (c) both the gradicnts and activations (Selvaraju et al. 2017). or (d) the gradient times the input image (hhrikumar al. 2017). However, these heatmaps can be too noisy be human- interprrrrreeeeaallee because the gradients in the pixel space are often local. Importantly, some gradient-bast attribution maps can be unfaithful explanations e.g. acting like edge detectors (Adebayo al. 2018).
To make a gradient-based heatmap more robust and smooth, a number of methods essentially aver- age out the resultant heatmaps across large set of perturbed inputs that created via (a) adding random noise to the input image (Smilkov et al. 2017 Fong & Vedaldi 2017), (b) blurring or jitter- ing the image (Fong& & Vedaldi (0177) or (c) linearly interpolating between the input and a reference "bascline" image (Sundararajan et al. 2017).
Black-box In addition to the white-box setting, perturbation techniques are even more important in approximating attributions under the black-box setting i.e. when we do not have access to the network parameters. Black-box methods often iteratively remove (i.e. occlude or perturb) an input region and take the average resultant classification probability change to be the attribution value for iirvrenttiontttaiinn an object out of a scene (revealing the content behind it) while keeping the rest of the scene intact-are impractical for most real-world applications.
that region. While the idea is principle in causal reasoning, the physical
Feature removal Therefore, the absence of an input region is often implemented by replacing it with (a) mean pixels (Zeiler Fergus 2014 Ribeiro et al. 2016); (b) random noise (Dabkowski & Gal 2017: Lundberg & Lee| 2017); (c) blurred versions of the original content (Fong & Vedaldi 2017) ). However, these removal techniques often produce unrealistic, oft-of-samples (Fig. which
raise huge concerns the sensitivity and faithfulness of the explanations.
2

---

Under review as a conference paper at ICLR 2020
Do explanations become more robust and faithful if input features are removed via learned, natural image prior? Here, we systematically study that question across three representative attri- bution methods: two black-box (i.e. SP and LIME) and one white-box (i.e. MP). We chose these three methods because they represent diverse approaches and perturb different types of input fea- tures: pixels (i.e. MP), superpixels (i.e. LIME); and square patches (i.e. SP). Similar us, Chang et al. (2019) and Uzunova et al. (2019) also harnessed image generative models to remove input fea- tures. However, their findings were (I) both only within the MP framework: (2) either for grayscale, medical-image datasets (Uzunova et al. 2019) or based on unrealistic samples (see comparison in Sec. 4.1). Furthermore, their results were not relevant to our question of whether integrating an inpainter helps attribution maps become more robust to hyperparameters. Note that Chang et al. (2199) found negative results for integrating the inpainter whereas our methods improved both the robustness of attribution maps (Sec. 4 and their ability to localize objects (Sec.
Counterfactual explanations A task that is related but not the same as ours to generate a textual explanation for why an image is predicted as class instead of some other class e' (Anne Hendricks ct al., 2018). For visual explanations, Goyal et al. (2019) proposed to find a minimal input region such that when exchanged with another region in reference image would change the classification for the original image into some target class. However, their counterfactual sample was generated by swapping patches between two images rather than by generative model.
METHODS
3.1 DATASETS AND NETWORKS
Classifiers Our experiments were conducted separately with each of the two ResNet-50 image classifiers (He et al. 2016) that were pre-trained on the 1000-class ImageNet 2012 and Places365, respectively. The two models were officially released by the Torch (2019) model zoo and by the authors (CSAILVision) 2019), respectively.
Datasets We chose these two datasets because they are large-scale, natural-image sets and cover a wide range of images from object-centric (i.c. ImageNet) to scenery (i.e. Places365). While state-of-the-ar image synthesis has advanced rapidly, unconditionally inpainting a large free-form mask in an arbitrary photo remains challenging (Yu et al., 2018a). Therefore, we ran our study on two subsets called ImageNet-S and Places365-S of the original validation sets of ImageNet and Places365, respectively, after filtering out semantically complex images. That is, we filtered out images that YOLO-v3 object detector (Redmon Farhadi 2018) found more than one object. For ImageNet-S, also filtered out images with more than one ImageNet bounding boxes. In total, ImageNet-S and Places365-S contains 15,082 and 13,864 images respectively (see Figs. & S6 for example images).
Inpainter We used two TensorFlow DeepFill-v1 models pre-trained by Yu et al. (2018b) for Im- ageNet and Places365, respectively. DeepFill-vl takes input color image and binary mask, both at resolution 256 x 256, and outputs an inpainted image of the same size.
3.2 PROBLEM FORMULATION
Let Dx + E be an image classifier that maps square, color image of spatial dimension D x D onto a softmax probability of a target class. An attribution map A E [-1, DxD associates each input pixel æ, to scalar A, [- which indicates how much æ; contributes for against the prediction score sax). We will describe below three different methods for generating attribution maps together with our three respective proposed variants which harness inpainter.
3.3
SLIDINGG-PACC
SP
Zeiler & Fergus 2014) proposed to slide a gray, occlusion patch across the image and record the probability changes attribution values in corresponding locations in the heatmap. That is, given a binary occlusion mask m E 10, (here, 1's inside the patch region and 0's otherwise) and filler image f E DxD33, a perturbed image DDD3 (see Fig. [p) is given by:
3

---

Under review as a conference paper at ICLR 2020
a == == m) f
(I)
where O denotes the Hadamard product and f is zero image i.e. a gray image before input pre-processing. For every pixel D; one generate perturbation sample Æ' (i.e. by setting the patch center at æ;; and compute the attribution value A; =( However, sliding the patch densely across the 224 224 input image of ResNet-50 computationally expensive. Therefore, we chose a 29 x 29 occlusion patch size with stride 3, which yields a smaller heatmap A' of size 66 x 66. We bi-linearly upsampled A' to the image size to create the full-resolution A. We implemented SP from scratch in PyTTorch following MATLAB implementation (Math Works, 2019).
SP-G Note that the stride, size, and color of the SP sliding patch are three hyperparameters that often chosen heuristically, and varying them can change the final heatmaps radically. To ameliorate the sensitivity to hyperparameter choices, we propose variant called SP-G by only replacing the gray filler image of SP with the output image of an inpainter (described in Sec. S.c.3.1 i.e. f G(m, æ) while keeping the rest of SP the same (Fig. D vs. e; top row). That is, at every location of the sliding window, SP-G queries the inpainter for content to fill in the window.
3.4
LOCAL INTERPRETABLE MODEL-AGNOSTIC EXPLANATIONS
LIME While SP occludes one square patch in the input at a time, LIME (Ribeiro et al.] 2016) occludes random-shaped region. The algorithm first segments the input image into S non- overlapping superpixels (Achanta et al. 2012). Then, we generate a perturbed image T by graying random set of superpixels among 2d possible sets. That is, LIME follows Eq. I where the pixel-wise mask m is derived from a random superpixel mask m' E (0, S For each sample ø', measure the output score s(z') and evenly distribute it among all occluded superpixels in x'. Each superpixel's attribution is then inversely weighted by the L2 distance || x''l via an expo- nential kernel and then averaged out across N samples. The resultant attribution as of superpixel k is finally assigned to all pixels in that group in the full-resolution heatmap A.
In practice, Ribeiro al. (2016) iteratively optimized for {ax}s via LASSO for 1000 steps to also maximize the number of zero attributions i.c. encouraging simpler, sparse attribution map. We used the implementation provided by the authors (Ribeiro 2019) and their default hyperparameters of S = 50 and N = 1000.
LIME-G While avoiding the bias given by the SP square patch, LIME perturbation samples re- main unrealistic. Therefore, we propose LIME-G. G. variant of LIME, by only changing the gray image f to a synthesized image G(m, a)) as in SP-G while keeping the rest of LIME unchanged.
3.5
MEANINGFUL PERTURBATION
MP Because SP and LIME gray out patches and superpixels in the input image, they generate unrealistic counterfactual samples and produce coarse heatmaps. ameliorate these issues, FFng & Vedaldi (2017) proposed the MP algorithm i.e. learning a minimal, continuous mask m [0, 1 that blurs out the input image in a way that would minimize the target-class probability. That is, they attempted to solve the following optimization problem:
arg min A/Im|]1 + s(x)
where T given by Eq. 1 but with f = (xr i.c. the input image blurred by a Gaussian kernel (.) of radius = 10. Note that, in MP, the attribution map A is also the learned mask m.
However, solving Eq. directly often yields heatmaps that are noisy and sensitive to hyperparameter changes. Therefore, Fong & Vedaldi (2017) only learned small 28 28 mask and upsampled it to the image size in every optimization step. addition, they also encouraged the mask to be smooth and robust to input changes by changing the objective function to the following:
m arg min \ [m]]1 + X,TV (m) Er-z((0,4) ( (z, T))]
(3)
'The ImageNet mean pixel gray (0.485, 0.456, 0.406).

---

Under review as a conference paper at ICLR 2020
where (m) = E, I1 i.e. a rotal-variation norm that acts as a smoothness prior over the mask. The third term is the expectation over a batch of randomly jittered versions of the blurred image æ. That is, (.) is the jitter operator that translates an image 2 vertically or horizontally by pixels where F is drawn from a discrete uniform distribution (1( 4).
We randomly initialized the mask from continuous uniform distribution (()0, 1) and minimize the objective function in Eq. B via gradient descent for 300 steps. Our MP implementation is in PyTorch and followed all the hyperparameters described in Fong & Vedaldi (2017).
MP-G We propose to create an MP variant called MP-G by only changing the filler image f B, (a:) that is used in Eq. ] to an inpainted image i.e. f = G(m, 2) where my E {0, D is the result of binarizing m at a threshold of 0.5. Note that this binarization step is necessary because G() expects a binary mask; however, we are still learning a continuous mask m E DRD
4 EXPERIMENTS AND RESULTS
4.1 INPAINTER FAILED TO SYNTHESIZE BACKGROUNDS GIVEN ONLY OBJECTS
Chang et al. 2019) proposed to find minimal set of input pixels that would keep the classification outputs unchanged even when all other pixels are removed via an inpainter i.e. the "preservation" objective (Fong& & Vedaldi 2017). They used the same DeepFill-v1 inpainter as in our work; how- ever, their "preservation" objective tends to force the inpainter to predict the missing background pixels conditioned on the remaining foreground object a task that DeepFill-vI was not trained to do and thus produced unrealistic samples as in Chang et al. (2019). In contrast, our MP-G method harnesses the dual "deletion" objective (Fong & Vedaldi 2017) i.e. finding the smallest set of input pixels such that when inpainted would minimize the target-class probability-hiie intuitively asks the inpainter to replace the main object with some generated content similar to the real background.
To compare these two objectives, we randomly chose 50 validation-set images from 52 ImageNet bird classes and computed their segmentation masks via a pre-trained DeepLab model (Chen et al., 2017]. We found that using the DeepFill-vl to inpaint the foreground region (i.e. our "deletion" task) yields realistic samples where the object is removed. In contrast, using the inpainter to fill in the missing background area (i.e. the "preservation" task) yields unrealistic images whose backgrounds contain features (e.g. bird feathers or beaks) unnaturally pasted from the object (see a side-by-side comparison in Fig. S7). This result motivated to integrate DeepFill-vl into attribution methods and to study MP-G with the "deletion" objective which has not been explored in Chang et al. (2019)
4.2 INPAINTER IS EFFECTIVE IN REMOVING DISCRIMINATIVE FEATURES
While removing objects from an image via DeepFill-v1 qualitatively yields realistic samples, here, quantitatively test how effective this strategy is in removing target-class discriminative features in comparison with three existing alternative filling methods: (1) zero pixels; (2) random noise; or (3) blurred versions of the original content.
Using the same procedure as described in Sec. 4.1 we randomly sampled 1000 bird images and segmented out the bird in each image. We filled the object mask in each image via all four methods (Fig. [ and compare the results (Table D. Surprisingly, the blurred and grayed-out images are still correctly classified 26.4% and 13.3% (Table respectively, by a pre-trained Inception-v3 classifier (Szegedy et al. 2016). That is, these perturbed images still contain much discriminative features relevant to the target class. In contrast, only 8.9% of the inpainted images were correctly classified suggesting that the inpainter was more effective in removing the discriminative features.
After the main subject (here, birds) has been removed from an image, intuitively, one would expect the modified image to be perceptually different from the original image (where the bird exists). Here, we evaluate how each of the four in-filled images E (where the bird has been removed) is perceptually dissimilar to the original image by measuring the MS-SSIM and LPIPS (Zhang al.) 2018) scores between every pair (x, σ). Across both metrics, the inpainted images are consistently the most dissimilar from the real images compared to the blurred and grayed-out images. Note that in all three quantitative metrics, the inpainted images are the closest to the noise-filled filled images (Table Ti-e) despite being substantially more realistic (Fig.
5

---

Under review as a conference paper at ICLR 2020
(a) Real
(b) Blur
(c) Gray
(d) Inpaint
(e) Random-noke noise
bustard 0.996
0.020
0.050
0.001
0.001
The results of filling the object mask in the real image (a) via four different filling methods. The shape of the bird is still visible even after blurring (b), graying out (c) or adding noise (e) to the bird region. The inpainter removes the bird and fills in with some realistic background content (d). Here, the bustard bird class probability for the inpainted image (d) at random chance (0.001) suggesting that the Inception-v3 classifier does not detect any remaining bustard features.
Figure 2:
Filling methods
Metrics
(a) Real
(b) Blur
(c) Gray
(d) Inpaint
(c) Random
Inception Accuracy
92.30%
26.40%
13.30%
8.90%
4.40%
MS-SSIM (lower is better)
1.000
0.941
0.731
0.707
0.692
LPIPS
(higher is better)
0.000
2.423
3.186
3,208
3.639
Table I: Evaluation of four different filling methods on 1000 random bird images. The Inception- v3 accuracy scores (Salimans et al.] 2016) suggest that inpainting the object mask (d) removes substantially more discriminative features relevant to the removed object compared to blurring (b) or graying out (c). Perceptually, the inpainted images are also more dissimilar to the corresponding real images according two similarity metries: MS-SSIM and LPIPS (Zhang al. 2018). See Fig. for examples of all filling results.
4.3 ARE GENERATIVE ATTRIBUTION METHODS MORE ROBUST TO HYPERPARAMETERS?
Porturbation-basee based attribution methods have many hyperparameters that are often heuristically tuned, which when varied can change the explanations radically. That sensitivity poses a huge challenge to (1) evaluating the explanations; and (2) building trust with end users (Doshi-Veld Velez & Kim 2017). Our hypothesis is that heuristically -ertturre samples are often far from the true data distribution and thus might contribute the sensitivity of heatmaps hyperparameters. Here, test whether our attribution methods which incorporate an inpainter, i.e. SP-G, LIME-G, and MP-G (hereafter, G-methods), are more robust hyperparameter changes than their original counterparts.
4.3.1 EXPERIMENT SETUP
Similarity metries and Image sets Following Adebayo al. (2018), used three metries from scikit-image (van der Walt et al. 2014)) to measure the similarity of heatmaps: the structural similar- ity index (SSIM), the Pearson correlation of the histograms of gradients (HOGs), and the Spearman rank correlation. We upsampled all heatmaps to the full image size before feeding them into the similarity metrics. We performed the test on a set of 1000 random images from ImageNet-S and also replicated the experiments on Places365-S.
SP sensitivity across patch sizes It remains question how choose the patch size in the SP al- gorithm because changing it can change the explanation radically (Zintgraf et al.) 2017). Therefore, we compare the sensitivity of SP and SP-G when sweeping across patch sizes p x with stride where p E {5, 17, 29, 41, 53). We chose this set to cover the common sizes that have been used in the literature. For each input image, we obtained = heatmaps (i.c. each corresponds to a patch size) and then measured the similarity among all k<K 1)/2 10 possible pairs.
LIME sensitivity across random batches of samples LIME randomly samples N perturbed im- ages {æ'},, L.v and uses them to fit heatmap. We found that the heatmap for an input can vary across different batches of random perturbation samples. Therefore, we compared the sensitivity LIME and LIME-G across 5 random batches of N 500 perturbation samples. That is, for each input

---

Under review as a conference paper at ICLR 2020
SP-
5P-6
5P-6
UME (50) LIMS-G (50)
UIEE
LIME (50) LIME-G (50)
LIME-6 (10)
MP
MP-G
MP-G
(a) SSIM
(b) Pearson correlation of HOG features (c) Spearman rank correlation
Figure 3: Error plots for SSIM (a), Pearson correlation of HOG features (b), and Spearman rank
correlation (c) scores obtained from 1000 random ImageNet-S images (higher is better). LIME-G is consistently more robust than LIME on both low and high resolutions i.c. S E {50. 150} (green and blue bars). The same trends were also observed on the Places365-S dataset (Fig. S4 numbers are reported in Table S3
The exact
image among the 1000, we generated & = 5 heatmaps and computed the similarity among all 10 possible pairs. We ran this experiment for small and a large heatmap resolution i.c. two numbers of superpixels S E (50, 150} while keeping all other hyperparameters constant.
MP sensitivity across random initializations Similar to the LIME sensitivity experiment, here, we compared the variations of MP vs. MP-G heatmaps of size 28 x 28 across 5 random initializations. We re-ran each algorithm 5 times on each input image and computed the similarity among all 10 possible pairs. Here, all hyperparameters are the same as in Sec. 3.5
4.3.2 RESULTS
First, we found that all 6 algorithms produce inconsistent explanations across the controlled hyper- parameters (Fig. 3 all scores are below 1). That is, LIME and MP heatmaps can change as one simply changes the random seed! However, LIME-G is consistently more robust than LIME across all metrics (Fig. 3) and across both superpixel settings. Across the patch sizes, SP-G is also con- sistently more robust than SP (Fig. bb: light vs. dark yellow). SP-G and SP performed on par with high standard deviations under the Spearman rank correlation (Fig. 3b). Across the random initial- izations. MP- G is consistently less robust than MP (Fig. 3 light vs. dark red). The above trends were also observed on Places365-$ (Fig. S4) suggesting that the findings are statistically significant.
4.4 ARE EXPLANATIONS BY GENERATIVE ATTRIBUTION METHODS MORE FAITHFUL?
While there are currently no established ground ground-truth truth datasets to evaluate an attribution map, prior research often assessed heatmaps via three proxy metries: (1) the object localization task (Zhou et al. 2016); (2) Insertion and Deletion task (Petsiuk et al., 2018). Here, we ran all 6 algorithms on the NetgeNet-S and Places365-$ datasets using the default hyperparameters in Sec. The heatmaps are then upsampled to the full image resolution for evaluation on all three benchmarks above.
Object localization Zhou et al. (2016) proposed to evaluate heatmaps by its ability to segment out objects in the ImageNet images, which often contain single object of a known class. We applied the localization procedure in Fong & Vedaldi 2017) for the ImageNet-S dataset. That is, for each algorithm, we derived multiple bounding boxes per heatmap by thresholding at different values of t = apfmaz where jmar is the maximum intensity in the heatmap and a [0 0.05 0.95). For each a, we computed the Intersection over Union (IoU) score between a derived bounding box and the ImageNet ground-truth. The object localization error was calculated by thresholding each IoU score at 0.5 and averaging them across the number of images. For each method, we chose the best a" that yielded the lowest error held-out set of 1000 ImageNet-S images (Table 2
We found that LIME-G and SP-G outperformed their respective counterparts while MP-G was on par with MP (Table 2 Among the 6 methods, LIME-G obtained the lowest error of 27. 12%. For qualitative evaluation, FSg S10 show set of heatmaps and derived bounding boxes for the cases where G-methods outperformed the original methods the most (in terms of IoU) and vice versa.
Deletion and Insertion metrics The Deletion metric measures the target-class probability changes we gradually zero out the input pixels of the highest attributions one-by-one in the descending
7

---

Under review as a conference paper at ICLR 2020
baseline
SP
SP-G
LIME
LIME-G
MP
MP-G
threshold
0.5
0.05
0.05
0.15
0.15
0.20
0.50
Error (%)
38.56%
39.17%
34.87% %
29.28%
27.12 %
29.98%
30.43%
Table 2: Localization errors (lower is better) for three G-methods and three originals on ImageNet- Naively taking the whole image as a bounding box yields an error of 38.56% (baseline). We also found SP-G to outperform SP on replicated test with 53 x 53 patches (data not shown).
order. We evaluated all methods via the code released by Petsiuk et (2018) on both the Deletion and also the related Insertion metric (Petsiuk et al., 2018 Samek et al.) 2016). However, did not find any consistent trends across both ImageNet-S and Places365-$ datasets and both metrics. For example, SP-G consistently outperformed SP under the Deletion metric on both datasets, but not under the Insertion metric. See Sec. forr more details. Note that these metrics (1) assume that input pixels are independent; however, zero-ing out pixels one-by-one almost always yields unrealistic images; (2) the probability changes are computed from those unrealistic, out-of-distribution inputs and therefore be misleading the exact problem that our paper attempts solve.
THE INNER-WORKING OF GENERATIVE ATTRIBUTION METHODS
Here, used SP-G and LIME-G the studies to explain why G-methods are both more (1) robust to hyperparameter changes (Sec. 4.3) and (2) accurate in localizing objects (Sec. 4.4).
5.1 MORE ACCURATE OBJECT LOCALIZATION: A CASE STUDY OF SP-G
We found that as the gray patch of SP is slided from left to right across the input image (Fig. 4a top), the target-class probability gradually decreases and approaches 0 when the patch occludes most of the object (Fig. 4b red line). Note that the probability can drop even when the patch is far outside the object region (Fig. 4b red line within 00, 24)) due to SP unrealistic grayish samples. As the
result, the probability distributions by SP often yield a large blob of high attributions around the object in the heatmap (Fig. 4a top).
In contrast, the inpainted samples of SP-G often keep the probability variance low except when the patch overlaps with the object (Fig. 4b blue vs. red), yielding heatmaps that are more localized towards the object (Fig. 4aa bottom). Across 1000 random ImageNet-S images, found that the average probability change when the SP 53 x 53 patch is outside the object bounding box is ~-2 times higher than that of SP-G (i.e. 0.09 0.04). In sum, our observations here consistent with the findings that SP-G and LIME-G G obtained lower localization errors than the original counterparts.
24
36
44
53
1.0
0.8
0.932
0.699
0.069
0.151
0.403
02-
0.0
36
48
53
38
0.987
0.987
0.913
0.907
0.972
Patch locations when siding from right acrous one
(a) Perturbation samples and heatmaps (rightmost)
(b) Target-class probability over patch locations
Figure 4: We ran SP and SP-G using 53 x 53 patch on a nail class image. Here are the perturbation samples derived for both methods when the patch is slided horizontally across one row 5 locations
(9. 24, 36, 44, 53} (a); and their respective target-class probability scores (b). SP-G samples are more realistic than those by SP and yielded a heatmap that localizes the object more accurately (a). That is. the probabilities for SP-G samples are more stable and only substantially drop when the patch covers the object (blue vs. red). See Fig, SII for more examples.
8

---

Under review as a conference paper at ICLR 2020
5.2 MORE ROBUST HEATMAPS: A CASE STUDY OF LIME-G
Here, we provide insights for why LIME-G produced heatmaps that are more consistent than LIME across 5 random batches of samples (i.e. controlled, in practice, by 5 random seeds). We observed that the top-1 predicted labels of 20.5% of the LIME grayish perturbation samples (e.g. Fig. 5ap were from only three classes { jigsaw puzzle, maze, hen-of-the-wood } whereas the same top-1 label distribution for LIME-G samples was almost uniform (see Fig. S21 for more details). That is, across different objects, many LIME samples are still given the same label due to their similar grayish, puzzle-like patterns. Relatedly, we observed that LIME perturbation sample is often given near- zero probability score regardless of what input feature is being masked out (Fig. Sap. Therefore, when fitted N samples, where N is often too smal w.r.t. the total 2 possible samples, the heatmap appears random and changes upon a new set of random masks (Fig. 5b).
LIME
kuvasz
0,001
0,000
0.008
0.001
0.001
kuvesz
SSIM: 0.465
(a) LIME perturbation samples from one trial
(b) LIME heatmaps using 5 random seeds
LIME-G
kuvasz
0.528
0.944
0.793
0.704
0.071
kuvasz
SSIM: 0.895
LIME-G perturbation samples from one trial
(d) 5 LIME-G heatmaps using random seeds
Figure 5: In Sec. 4.3 we compared the robustness of LIME vs. LIME-G heatmaps when running using different random seeds. This is an example where LIME-G heatmaps are more consistent than LIME's s (d vs. b). While LIME grayish samples (a) are given near-zero probabilities. LIME-G samples (here, inpainted using the same masks as those in the top row) are often given high proba- bilities except when the kuvasz dog's eye is removed (c). LIME-G consistently assign attributions to the dog's s eye (d) while LIME heatmaps appear random (b). The top-1 predicted labels for out of 5 LIME samples (a) here are paper towel.
In contrast, for LIME-G samples, the probabilities consistently drop when some discriminative fea- tures (e.g. the kuvasz dog's eye in Fig, 5c) are removed. This phenomenon yields heatmaps that are more consistently localized around the same input features across different random seeds (Fig. (dd.. Our explanation also aligns with the finding that when the number of superpixels S increases from 50 to 150 (while the sample size remains at N 500), the sensitivity gap between LIME vs. LIME-G increases by ~3 times (Fig- 3. gap between green bars vs. gap betwen blue bars).
See Figs. S12 ST9 for qualitative examples of when LIME-G more robust than LIME and vice versa. Quantitatively, we found that the image distribution where LIME-G showed superior ro- bustness over LIME across all three similarity metrics mostly contains images of scenes, close-up or tiny objects. In contrast, LIME is robust than LIME-G on images of mostly birds and medium-sized objects (see Sec. f more details).
DISCUSSION AND CONCLUSION
4.4 is that the original formulation (1) optimizes a tiny 28 x 28 mask; and (2) already strongly regularizes it to be sparse and smooth. A non-mutually exclusive hypothesis is that since MP operates in the pixel space, the learned masks can be of any shape. However, DeepFill-vl was not designed for inpainting free-form masks Yu et 2018a). We also found that blur mask learned by MP often covers 999 of the input image. That is, the MP attribution map is explaining the target-class probability drops when the entire image is modified- unrealistic causal intervention. In contrast, MP-G perturbations are more localized and only changes ~40% of the input image (see Sec
A hypothesis for why we did not find MP-G outperform MP (Secs. 4.3
Because sample size N is often too small, for both LIME and LIME-G, G, the low-variance sample probability distribution (e.g, all near-zero scores in Fig. 5a) for an image is highly correlated with high cedds--radoo-seed sensitivity (Fig. (20 Pearson correlation 0.856).

---

Under review as a conference paper at ICLR 2020
In sum, integrating state-of-the-arr, unconditional inpainter into three existing attribution methods can yield visual explanations that (1) localize objects better; (2) are more robust to hyperparameter changes; and (3) are based on more plausible counterfactuals. Our results suggest that harnessing generative models to synthesize synthetic interventions (here, removal of input features) might be a promising direction for future causal explanation methods.
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Paseal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274-2282, 2012.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, pp- 9505 9515, 2018.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual expla- nations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 264-279, 2018.
Hho Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference Learning Representations, 2019.
Liang-Chie Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4): 834- 848,848,222 2017.
CSAILVision. Csallvision/placee::::: The places365-on for scene classification. 2019. (Accessed on 09/21/2019).
https://
github. wwt.ub.com/caaaa.. http://////aA..isoo/oooooaoom
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 011.77022200000 2017.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint (tt.:199440745/11 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-47744 2017.
Math Works.
Network
visualization
based
on
occlusion sensitivity.
https: ogs. me mathworr rks http:///eeeeeeeeeeeeeee///////// network-vi sua zat on cessed on 09/17/2019).
2019. (Ac-
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
10

---

Under review as a conference paper at ICLR 2020
PyTorch. torchvision.model. pytorch master documentation.
org/
mot.oo/mmoolls...m mtt.s/oooooo...ammmmma 2019. (Accessed on 09/21/2019).
Joseph Redmon and Ali Farhadi.
Yolov3: An incremental improvement.
arXiv preprint
104::08004027677, 2018.
Marco Ribeiro. Lime: Explaining the predictions of any machine learning classifier.
https:
2019. (Accessed 09/17/2019).
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. ACM, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)211-225 2015.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization what a deep neural network has Iearned. IEEE transactions on neural networks and learning systems, 28(1 (1 (111):666002666
2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-base baased local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3145-3153, International Convention Centre, Sydney, Australia, 06- 11 Aug 2017. PMLR. URL /RRpattpp//prrccedig.s mlr html
wttp://..gg........ss///0o// wwwpppwwssseeepppppppppppp shr kumarl7
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
networs: Vi-
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.0388 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution deep networks. In Proceedings of the 34th International Conference Machine Learning-Volle Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Hristina Uzunova, Jan Ehrhardt, Timo Kepp, and Heinz Handels. Interpretable explanations of black box classifiers applied on medical images by meaningful perturbations using variational autoen- coders. In Medical Imaging 2019: Image Processing, volume 10949, pp. 1094911. International Society for Optics and Photonics, 2019.
Stéfan van der Walt, Johannes L. Schönberger, Juan Nunez-Iglesias. François Boulogne, Joshua D. Warner, Neil Yager, Emmanuelle Gouillart, Tony Yu, and the scikit-image contributors. scikit- image: image processing in Python. PeerJ, 2:c453, 6 2014. ISSN 2167-8359. doi: 10.7717/peerj. 453. URL htt.t:ttt::////tt
org/10. 771 peer .43
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. arXiv preprint arXiv:/800 035.03589, 2018a
11

---

Under review as a conference paper at ICLR 2020
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5505-5514, 2018b.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833. Springer, 2014.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586-595, 2018
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 452-:1442- 2017.
Luisa M Zintgraf, Taco Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 17Xiv:1702.04555, 2017.
12

---

Under review as a conference paper at ICLR 2020
APPENDIX
A MASKS LEARNED MP PERTURB ALMOST THE ENTIRE IMAGE
For ImageNet images where numan-annotated bounding boxes only cover less than 50% of the image, MP perturbs the entire image to generate its final attribution maps. Hence, the heatmap does not completely reflects the target object class the drop in probability is due to the perturbation across the whole image (Fig. ST). On the other hand for MP-G, as discussed in Sec. 3.5 we binarize the mask during the optimization process and perturb only part of the image. In sum using MP-G, perturb only 400% and ~39% (Table ST) of the ImageNet-S and Places365-S images respectively.
ImageNet BB
MP (ImageNet)
MP-G (ImageNet)
MP (Places365)
MP-G (Places365)
20.28+12.01%
99.30:11.249
40.22-123.11 %
99.22 1 1.23%
39.22 26.666 %
Table S1: Average mean and deviation of the percentage of pixels perturbed by MP and MP-G for generating ImageNet-S and Places365-S heatmaps. For ImageNet-S, the ground truth bounding box (BB) covers mean area of 20.28% whereas, MP perturbs ~ 99% of the image and MP-G perturbs area closer to the actual ground truth distribution. See Fig S] for the qualitative represenation of these numbers.
(a) Original
(b) MP heatmap
(c) MP-perturbed
(d) MP-G heatmap
(e) MP-G perturbed
E00
Figure S1: MP almost perturbs the entire image generate its respective heatmap. Top row: An ImageNet-S image (a) and its respective MP (b) and MP-G (d) heatmaps. The pixels perturbed by MP (c) covers almost 99% of the pixels. In contrast, MP-G often perturbs ~40% of the image (e). Bottom row: An example from the Places365-S dataset. MP-G just perturbs the area around the house (e) whereas, MP perturbs almost the entire image including the trees (c). These figures qualitative evidence for the numbers in Table ST
13

---

Under review as a conference paper at ICLR 2020
B EVALUATION ON THE DELETION AND INSERTION METRICS
The deletion metric (Petsiuk et al. 2018) measures the target-class probability changes as we grad- ually zero the input pixels the highest attributions the descending order. The idea if the attribution values in heatmap correctly reflect the discriminative power of the input pixels, knock- ing out the fighest-attribute
pixels should quickly cause the probability approach However, this metric has two issues: (1) it makes an unrealistic assumption that input features are independent; (2) the probability changes may be due to the unrealistic, are masked out. For completeness, we evaluated all 6 methods via the code released by |Petsiuk al. (2018) on the deletion metric and also inverse version i.e. the insertion metric (Petsiuk al. 2018) both datasets of ImageNet-S and Places365-S. S. However, did not find significant differences between the G-methods and their counterparts (Fig
of-distribution inputs when some pixels
ImageNet-S
Places365-S
SP
SP-
LIME
LIME
UME-6
MP-G
MP-G
0.0
0.1
0.2
0.3
9.4
0.5
0.6
0.7
0.9
1.0
0.0
0.1
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Normallzzed Area Under Curve
Normalized Area Undier Curve
(a) Deletion (lower is better)
(b) Deletion (lower is better)
Baxeline
Daneline
SP-6
SP-6
UME-6
LIME-G
MP-6
MP.0
0.3
0.6 0.7
Normalized Area Under Curve
Normalized Area Under Curve
(c) Insertion (higher is better)
(d) Insertion (higher is better)
Figure S2: Error plots for the Deletion and Insertion metrics (Petsiuk et al. 2018) for all 6 attri- bution methods across two different datasets: ImageNet-S (a-c) and Places365-S (b-d). We used random noise heatmaps (i.e. having no information about the input image or the classifier) as the baseline method (cyan bars). We did not find any significant differences between G-methods and their respective counterparts across both two metrics and both datasets. Note that our Deletion and Insertion scores for the SP and LIME algorithms match closely to those in Petsiuk al, (0018) See Table S2 for the exact numbers.
14

---

Under review as a conference paper at ICLR 2020
Dataset and Metric
ImageNet-S
Places365-S
Deletion
Insertion
Deletion
Insertion
Baseline
0.197
0.433
0.126
0.311
SP
0.193
0.734
0.168
0.419
SP-G
0.166
0.685
0.138
0.414
LIME
0.109
0.800
0.112
0.462
LIME-G
0.127
0.813
0.137
0.459
MP
0.153
0.765
0.135
0.457
MP-G
0.141
0.701
0.141
0.421
Table S2: On the Deletion (lower is better) and Insertion (higher is better) metrics, did not find any significant differences between the G-methods and their counterparts. See Fig. S2 for an error plot of this table.
C LIME-G IS MORE ROBUST THAN LIME ON IMAGES OF SCENES, CLOSE-UP AND TINY OBJECTS
We have shown that LIME-G is more robust than LIME consistently all 3 different similarity metrics (see Sec. 4.3 in the main text). Here, we aim to understand the image distributions where LIME-G was more robust than LIME and vice versa.
For each of the three metrics, we computed a set of top-100 100 score differences between LIME-G vs. LIME. Interestingly, we found the intersection of the three contains images of mostly scenes, close-up or tiny objects (see Fig. (S3). In contrast, the common set of images where LIME is more robust than LIME-G contains mostly birds and medium-sized objects. These image distributions intuitively align with the domains where DeepFill-vl is capable of inpainting and suggest that the performance of G-methods can be improved further with cass-conditional inpainters.
15

---

Under review as a conference paper at ICLR 2020
Similarity Metrics Pearson correlation of HOGs
Method
SSIM
Spearman
SP
0.698=0.114
0.604:0.100
0.404
SP-G
0.781=0.095
0.691+00.093
0.317=0.206
LIME (50)
0.553=:0.00
0.848
0.573:0.077
LIME-G (50)
0.647
0.896110.022
0.667110.065
LIME (150)
0. 0..03
0.708:10.025
0.155:10.072
LIME-G (150)
0.371 0.0551
0.776
0.379
MP
0.673=0.111
0.676:10.099
0.800=1.1006
MP-G
0.356+00.09
0.572++0.071
0.644+00.12
Table S3: The results in this table the number forms of Fig G-methods of SP and LIME more robust to random parameters across different sensitivity metrics.
Similarity Metrics
Method
SSIM
Pearson correlation of HOGs
Spearman
SP
0.577=0.177
0.674
0.452:10.288
SP-G
0.720-101222
0.755-10.056
0.332:10.208
LIME (50)
0.392:10.074
0.802:10.036
0.594:10.078
LIME-G (50)
0.076
0.865:10.027
0.722:10.058
LIME (150)
0..46
0.701+0.026
0.201+00001
LIME-G (150)
0.012=0.061
0.780:10.022
0.511:0.051
MP
0.112
0.736:100097
0.846:10.124
MP-G
0.463=0.089
0.594+1.082
0.654
Table S4: The results in this table are the number forms of Fig S4 as the ImageNet-S dataset.
The results follow the same trend
16

---

Under review as a conference paper at ICLR 2020
Images where LIME-G outperformed LIME across all three sensitivity metries
Images where LIME-G underperformed LIME across all three sensitivity metrics
Figure S3: Common images across all three metrics where LIME-G is consistently more robust than LIME (top) and vice versa (bottom). Interestingly, found the intersection of the three sets contains images of mostly scenes, close-up or tiny objects (top). In contrast, the common set of images where LIME is more robust than LIME-G contains mostly birds and medium-sizze objects (bottom).
17

---

Under review as a conference paper at ICLR 2020
5P
SP
SP-6
LINE (50)
LIME
LIME (50)
LIMS (15.))
LIME (158)
LIME (150)
LIME-G (150]
LIME-G (150)
LIME-G (150)
MP
MP-C
HP-C
0.0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
(a) SSIM
(b) Pearson correlation of HOG features (c) Spearman rank correlation
Figure S4: Bar plots comparing the robustness (higher is better) of G-methods and their counter- parts when changing hyperparameters (described Sec. (4.3) under three different similarity met- rics: SSIM (a), Pearson correlation of HOG features (b), and Spearman rank correlation (c). Each bar shows the mean and standard deviation similarity score across 1000 pairs of heatmaps, each produced for one random Places365-S image. SP-G is more robust than SP under SSIM (a) and Pearson correlation (b). In contrast, MP-G is consistently more sensitive than MP under all metrics (red bars shorter than light-red bars). The exact numbers reported in Table S4
18

---

Under review as a conference paper at ICLR 2020
(a) ImageNet images where YOLO-v3 detects more than one objects and were not ImageNet-S.
YAmson
(b) Random images in ImageNet-S.
Figure S5: We filtered images where YOLO-v3 detected more than one objects (a). The maining images are used in ImageNet-S (b). We believe our observations on the ImageNet-S dataset carry over the full ImageNet.
19

---

Under review as a conference paper at ICLR 2020
(a) Images in Places365 images that YOLO-v3 detected more than one objects and were not used in Places365-S.
(b) Random Places365-S images.
Figure S6: Example images from Places365-S dataset (b). The images where YOLO-v3 detected more than objects (a) are more cluttered compared to the images used in Places365-S (b).
20

---

Under review as a conference paper at ICLR 2020
(a)
(b) Mask (c) Preserve
(d) Delete
(c) Real
(f) Mask
(g) Preserve
(h) Delete
Figure S7: Inpainting using the preservation objective generates unrealistic samples (Sec4.1) We randomly chose 50 validation-set images (a) from 52 ImageNet bird classes and compute their seg- mentation masks via pre-trained DeepLab model (Chen et al. 2017) (b). We found that using the DeepFill-vl inpainter to inpaint the foreground region (i.c. our "deletion" task) yields realistic samples where the object removed (d). In contrast, using the inpainter to fill in the background region (i.e. "preservation" task) yields unrealistic images whose backgrounds contain features (e.g. bird feathers beaks) unnaturally pasted from the object (c).
21

---

Under review as a conference paper at ICLR 2020
Top- 10 cases where LIME-G outperformed LIME
Top- cases where LIME-G underperformed LIME
Real
LIME
LIME-G
LIME
LIME-G
640
Figure S8: Top-10 cases where the LIME-G outperformed (left) and underperformed (right) LIME on the object localization task (IoU scores). From left right, on each row: we show a real image with its ground-truth bounding box, LIME heatmap & its derived bounding box, LIME-G heatmap & its derived bounding box. See https::tt.... /driev/e//rri/ie//e///ee// google. com/ (ariveuu/2 2/ folders/ 10JeP 9dpuoa 0M1 2ccpuuo0016x222218B8B... j07NN5X for examples of the LIME and LIME-G IoU results.
22

---

Under review as a conference paper at ICLR 2020
Top- 10 cases where MP-G outperformed MP
Top- 10 cases where MP-G underperformed MP
Real
MP
MP-G
MP-G
Figure S9: Top-10 cases where the MP-G outperformed (left) and underperformed (right) MP on the object localization task (loU scores). From left to right, on each row: we show a real im- age with its ground-truth bounding box, MP heatmap & its derived bounding box, MP-G heatmap & its derived bounding box. See httts://drive www.looooooo..ooooooooooooooooo. /ttmm/ddiee///////..errer BttpoosssdKKpPPP_____eee__
for more examples of the MP and MP-G IoU
sults.
23

---

Under review as a conference paper at ICLR 2020
Top- 10 cases where SP-G outperformed SP
Top- 10 cases where SP-G underperformed SP
Real
SP
SP-G
Real
SP
SP-G
3.81
Figure $10: Top-10 cases where the SP-G outperformed (left) and underperformed (right) SP on the object localization task (loU scores). From left to right, on each row: we show a real image with its ground-truth bounding box, SP heatmap & its derived bounding box, SP-G heatmap bounding box. In the the heatmap localizes some unique features of the object as compared to the images in the top cases where the heatmap covers the entire image. See https://dd google......oooo cwm/ ptrie//iiie/ee ve/u/ 3
derived where SP-G has lower IoU score than SP (right panel), we observed
2/folders http:///XXGMMMM2X22220 2/Fooddarr/XXX5MMAMMM
for more examples of the SP and
SP-G IoU results.
24

---

Under review as a conference paper at ICLR 2020
nail
0.917
0.730
0.615
0.187
0.166
nail
0.984
0.990
0,985
0.974
0.511
nail
0.497
0.150
0.077
0.640
0.180
SP-G
nail
0.956
0.960
0.394
0,984
0.977
Figure S11: Random intermediate perturbation samples by SP and SP-G on the same image from the nail class in ImageNet. SP-G drops the target-class probability only when the patch cover major area the nail (e.g. the center 0.394-probability sample in the bottom panel). This figure is zoom-in version of the samples in Fig.
25

---

Under review as a conference paper at ICLR 2020
LIME
steam locomozive
0.001
0.000
0.026
0.040
0.011
LIME-G
steam locomozive
0.722
0.727
0,500
0.990
0.969
LIME
steam locomocive
0.328
0.001
0.004
0.303
0.025
LIME-G
0.575
0.944
0.981
LIME
steam lacomazive
SSIM: 0.342
UME-G
Steam locomozive
SSIM: 0.750
Figure $12: Qualitative evidence supporting the LIME-G vs. LIME sensitivity experiment in
Sec. 4.3 For both LIME and LIME-G, per image, compute average SSIM score across all 10 pairs of 5 heatmaps. We then take the difference between LIME-G and LIME and sort them in the descending order. This steam locomotive image is random image from the top- 100 ImageNet-S cases where LIME-G outperformed LIME. Top four rows: Here, we compare pairs LIME vs. LIME-G perturbation samples that were ereated from the same random superpixel masks. LIME- G samples cause large probability drops only when some discriminative feature is removed from the image and thus results in more localized heatmaps. Bottom two rows: heatmaps by LIME and LIME-G, each from a random seed. While LIME-G heatmaps are more consistent, LIME heatmaps noisy and varies. See Fig. ST3 and Figs. S14 for similar observations in ImageNet- S and Places365-S dataset respectively. See https: /rive. google.oo...oooooooooooooooo. com/ htmm//rivee///// for more examples.
26

---

Under review as a conference paper at ICLR 2020
LIME
0.000
0.301
LIME-G
KUVAS2
0.52B
0.944
0.793
0.704
0.071
LIME
kuvasz
0.000
0.000
0.022
0.000
0.013
LIME-G
LIME
kuvasz
SSIM: 0.465
LIME-G
Kuvasz
SSIM: D.8U5
Figure S13: Here, we show the same figure as Fig. S12 (see its caption) but for another random image among the top-100 ImageNet-S cases where LIME-G outperformed LIME on the SSIM similarity metric. See poattps://driin google. www.coo//c//ooooo htmm//r.vve////fooeers SecKiigiXXSSSSSSooooooooSSSSSAB
for more examples.
27

---

Under review as a conference paper at ICLR 2020
LIME
lighthouse
0.000
0.020
0.641
0.337
0.001
LIME-G
lighthouse
0.005
0.968
0.868
0.961
0.052
LIME
lighthouse
0.006
0,000
0.193
0.30C
0.000
LIME-G
lighthouse
LINE
lighthouse
SSIM: 0.483
UME-G
lighthouse
SSIM: 0.931
Figure S14: Here, we show the same figure as Fig. $12 (see its caption) but for a ran- dom image among the top-100 Places365-S where LIME-G outperformed LIME the SSIM similarity metric. See
LaxxpFacceLeeooooopppvsaezzzz22d for more examples.
28

---

Under review as a conference paper at ICLR 2020
LIME
gas_station
0.005
0.009
0.006
0.001
0.008
LIME-G
gts_station
0.519
0.990
0.714
0.930
0.914
LIME
gas sastitaionn
0.023
0.002
0.008
0.30C
0.002
LINE-G
UIME
yes_3t aas_station
SSIM: 0.350
UME-G
gas_station
SSIM: 0.772
Figure S15: Here, we show the same figure as Fig. $12 (see its caption) but for a ran- dom image among the top-100 Places365-S where LIME-G outperformed LIME the SSIM similarity metric. See
comm/diive/n/2//ooderrs
LaxprE3cdieeeDoooopppsssezzEEEEd for more examples.
29

---

Under review as a conference paper at ICLR 2020
LIME
brambling
0.984
0.998
0.979
0.018
0.214
UMEE-6
brambiing
0.999
0.999
0.998
0.998
0.376
LIME
brambling
LIME-G
brambiing
0.995
0.988
0.997
0.982
0.999
LIME
brambiing
SSIM: 0.750
UME-G
brambling
SSIM: 0.445
Figure S16: Here, show the figure Fig. S12 (see its caption) but for random im- age among the top-100 ImageNet-S cases where LIME-G G underperformed LIME on the SSIM similarity metric. LIME-G samples remain high target-class probabilities and therefore pro- duced heatmaps that are more sensitive than those of LIME, Similar observations can be found in Fig. SS7 and Figs. S18 S19 1sKWig 4 KWW.SS TBhJRART for more examples.
See https: drive gwogle.oooooooooooooooooooo... wwtam//dive///////aa//a
30

---

Under review as a conference paper at ICLR 2020
LIME
tricycle
0.997
0,881
0,998
0.981
0.928
LIME-G
ticycle
1,000
0.997
1.000
0,999
0.992
LIME
tricycle
0.003
0.996
0.001
0.996
0.996
LIME-G
briyccle
LIME
tricycle
SSIM: 0.967
UME-G
ticycle
SSIM: 0.572
Figure S17: Here, we show the same figure as Fig. S12 (see its caption) but for a ran- dom image among the top-100 ImageNet-S where LIME-G underperformed LIME the SSIM similarity metric. See /tttps://drii: google hwtpoolle..o/driveeeeeooeo wwwm/driwee/////oo..errrrr Lss.looeecSSSSPOOOOooOAASSSTTTTE for more examples.
31

---

Under review as a conference paper at ICLR 2020
LIME
0.062
0.027
0.001
0.338
0.016
LIME-G
aquarium
0.947
0.962
0.859
0.382
0.840
LIME
aquarium
0.009
0.013
0.102
0.342
0.003
LIME-G
LIME
aquarium
SSIM: 0,755
UME-G
SSIM: 0.377
Figure S18: Here, we show the same figure as Fig. S12 (see its caption) but for a ran- dom image among the top-100 Places365-S cases where LIME-G underperformed LIME the SSIM similarity metric. See https://drive http///rvvv...oglleeeee. wwwm//diwee/////oodeer.. Laxpre3coLeeeooDQPpSvSSSEEEEEESE for more examples.
32

---

Under review as a conference paper at ICLR 2020
LIME
rice_pase jeedp
0.066
0.059
0.055
0.062
0.030
LIME-G
rico ree_paday
0.883
0.972
0.958
0.973
0.896
LIME
rice paddy
0.894
0.161
0.010
0.457
0.346
LIME-G
sce_paday
0.961
LIME
Ausn Spppt 2211
SSIM: 0.749
UME-6
niceppaday
SSIM: 0.385
Figure S19: Here, we show the same figure as Fig. S12 (see its caption) but for a ran- dom image among the top-100 Places365-S cases where LIME-G underperformed LIME the SSIM similarity metric. See https::/drive: wwweriive.ooolleee wowm/driwe//////ooderre LaxxFFaceeLeeDoDQQppsSSSEEEaSLEL for more examples.
33

---

Under review as a conference paper at ICLR 2020
Top-100 100 cases where LIME-G outperformed LIME
Top- 100 cases where LIME-G underperformed LIME
LIME-G LIME
0.8
0.8
0.6-
SSIM
SSIM
0.4
LIME-G LIME
0.0
0.0
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3 0.4
Std. of probability across 500 perturbed samples
Std. of probability across 500 perturbed samples
(a) Pearson correlation 0.856
(b) Pearson correlation 0.758
Figure S20: Scatter plots for the top-100 images where LIME-G outperformed (a) and
underperformed (b) LIME on the SSIM similarity scores. Each data point the seatter plot we compute two statistics from an image. y-axis: We compute the average SSIM score over all 10 possible pairs out of 5 heatmaps computed by running each method (LIME & LIME-G) using random seeds. x-axis: We compute the standard deviation (Std.) of the target-class probabilities for all N 500 perturbation samples. We observe high Pearson correlation between the two measures both cases when LIME-G outperformed (a) and underperformed LIME (b). That is, robustness of attribution maps strongly correlates with the variance in the probability distribution of the perturbed samples.
34

---

Under review as a conference paper at ICLR 2020
10000
9000
8000
7000
5000
4000
3000
2000
Vgsaw puzzle
maze
woods.
paper towel
acom pot
hair slide
hatchee
gyromitra
conch
great grey owl
noneycomb. Indri
bulletprool vest
shower curtain
wire haired fox terrier
hummiiggine
braimbling
Gorman short short-halres halred
cleaver
solar dich-
barn spider
banded geckk
picket Tence
lifebeat
pinwhhol
echidna
comic book
quilletine
switch
zebra-
space shuttic
electrrcc fan
street sige sign-
earthstal
kite.
Sealyham berrier
organ-
candle
lbex-
goose
panpipo
African chameleon
envelope
langur
wall clock
carpenter's
bolo
of the
(a) LIME histogram distribution is skewed
10000
7000
6000
5000
4000
3000
2000
1000
gyst sorrmii
bramin urambiing spocec
barn guerro
Surto
grey whabe
hussard
Ibox
water makke
whi whiptall
hen-afftthe: wooddds
suiphur butterffy
electrnc fan
cabbage buttorfly
eft tarantula
boa constrictar
lesser panda
exolot!
firebaa-
hamster
fiddlor crab
space shutle
wheal
safe
stean locomative
indigo bunting
dhole
green mambo
mimbemill:
coral reef
meerkat-
moviing van
organ
balo ti
yellow lady s siipper disc
Sealyham terrier dugong
tick
hummingghing
peat get
black
(b) LIME-G histogram distribution is almost uniform
Figure S21: We ran LIME and LIME-G on 200 images, each run has 500 intermediate perturbation samples. Here, for LIME (a) and LIME-G samples (b), we show histogram of the top-1 predicted class labels for all 200 runs x500 samples = 100,000 images. The set of 200 images comprises of cases where LIME-G outperformed (100 images) and underperformed (100 images) LIME on the SSIM sensitivity metric (Sec. 4.3) LIME perturbed samples are highly biased towards few jigsaw puzzle, maze classes (top panel), which somewhat intuitive given the gray-masked images (see Figs. ST77) In contrast, the histogram of LIME-G samples are almost uniform. x-axis: For visualization purposes, we sorted the top-1 labels and showed only first 50 labels.
35

---

Under review as a conference paper at ICLR 2020
LIME
LIME
LIKE
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.68
0.65
0.8
3.0
0.0
0.2
0.8
0.6
0.8
1.0
(a) SSIM
(b) Pearson correlation of HOG features (c) Spearman rank correlation
Figure S22: Bar plots comparing the LIME LIME-G robustness (higher is better) across two different numbers of superpixels E {50, 150} under three different similarity metrics: SSIM (a), Pearson correlation of HOG features (b), and Spearman rank correlation (c). For each image in 1000 random ImageNet-S images, we produced a pair of heatmaps by running LIME (light-blue) or LIME-G (dark-blue) with different numbers of superpixels S (50, 150). Each bar shows the mean similarity across all 1000 heatmap pairs. LIME-G is consistently more robust than LIME, specifically by ~200 under the SSIM (a) and Spearman rank correlation (c).
36


Reviews:
[{"review_id": "SkxCsZgccr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is focused on perturbation-based local explanation methods; methods that only need black-box(ish) access to the model and generally seek to find a region, pixel, etc's importance score through by removing that region, pixel, etc.. The intuition is that an important region if removed, will result in a large drop in the predicted class confidence. One main issue with such methods is that the removal itself can throw the image out of the data distribution and therefore result in a drop in confidence, not because of the region's importance, but because of the network's unpredictable behavior in such unseen parts of the input space. The work is focused on giving a solution to this problem: instead of removal through blurring, graying, etc, use inpainting; i.e. replace the removed region with using given the rest of the image. The idea has already been discussed out in the literature and the novelty of the work seems to be twofold: They introduce the same method in a way that is not curated for a specific perturbation-based method and could be concatenated with \"ANY\" given (or future) perturbation-based local explanation method (which authors notate by calling it ${existing_method}-G, they study robustness to hyper-parameter choice. \n\nThe paper is quite well written and the experiments are comprehensive. I have two major comments/issues with the work:\n\n1- The contribution of this work given existing work (more specifically the famous Chang et al work seems not to be enough for a venue like ICLR. If I want to list the contributions, it would be as follows (I would appreciate if the authors could correct me as the score is subject to change given more clarification on the matter):\n    - This work utilizes an inpainting step in combination with several methods while previous work is focused on meaningful perturbations method. This, although useful, does not introduce a novel technical contribution. The main technical contribution has been the use of inpainting (to be more exact, using generative models to approximate P(c|x_r)) which has been done before on a few previous works.\n    - The work argues that the use of inpainting in Chang et al (focused on keeping the salient object and removing background) was invalid as the inpainter model is not trained to do such a task. It could be argued that one could train another inpainting model that \"is\" capable of such a thing and therefore the general argument would not hold. One drawback of this approach, however, would be that training such an inpainting model might be difficult.\n    - Hyperparameter robustness. Studying this question is valuable. However, given that the assumption of this work and previous works is that generative approaches are generally better (even not considering the hyperparameter robustness), I am not sure how this knowledge could be used.\n\n2- Both this work and the previous works run on the assumption mentioned at the beginning of this review which basically says that non-generative perturbation-based methods throw the image out of data distribution and this is bad for such and such reasons. Although intuitively clear, I could not find any evidence in this work suggesting any meaningful difference using objective measures. One would assume that such phenomena would manifest itself clearly using insertion-deletion explanation metrics while as the authors report there was no significant difference. (Section 4.1 results clearly show a difference but this is not related to how the downstream explanation task is affected) For all it is know, generative methods have the drawback of being computationally more expensive than a simple blurring or replacing with random noise. (And a major elephant in the room is whether using an inpainter is actually taking the data back to the true data distribution which seems to be on an unproven assumption that these generative models are capable of learning the data manifold)\n\nMinor comments:\n    -Section 4.2 is really interesting. Thanks!\n    - Fig 3 results: MP is more robust than MP-G and I couldn't find any explanation of why this method behaves specifically different than the other two in the experiments section. It might be better to move the explanation in the discussions to the experiments.\n    - The task of most generative perturbation-based methods is to find a way to approximate P(c|x_\\r) which is the conditional probability given the non-removed part of the image. Usually, they do the approximation by sampling several images from the conditional P(x_r|x_\\r) (conditional inpainting)  using the generative model and averaging the prediction probability. This work seems not to be concerned with these specifics and directly feeds one of such samples. Could you explain this choice\n    - For studying the robustness of LIME, apart from the random seed, couldn't one change the hyperparameters of the superpixel method? Tha one seems more of a practical problem.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1e1xRnRFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a deep visualization technique for black-box image classifiers that feeds modified versions of the original input by means of an off-the-shelf (black box too) image inpainting approach (DeepFill-v1), in order to capture changes in the classification performance. In particular, the substitution of the input image follows three published paradigms: Sliding Patch (SP), Local Interpretable Model-Agnostic Explanations (LIME), Meaningful Perturbation (MP). Whereas the states of the art use gray images (SP, LIME)/blurred versions (MP) as substitution on different spatial supports (regular patch SP, random-shaped superpixel regions LIME, learned continuous region MP), the proposed approach inserts there the output of the inpainting.   \nThere are problems in the paper, major and minor.\nMajor: \n1)\tThe technical contribute is minimal, the author combine two already existent techniques.\n2)\tThe results are not convincing: a) the comparison are not fair, since on two out of three techniques the authors consistently change the competitors, letting them work with zero-value occluders or random noise. Only the third competitor has been employed in is original form (blurring the images). b) results are better (higher classification drop, plus other metrics) with the proposed approach wrt the first two competitors, which looks strange. Can the authors try to apply the comparative approaches in their original versions?\n3)\tThe fact that we are using an inpainting tool which may work in some cases (in providing well-distributed patches) but in other may fail, corrupting consequently the overall following analysis is a price I don’t want to pay, so I prefer some synthetical but controlled artifact. Actually, in the case of inpainting failure will generate structured noise, hard to be managed.   \nMinor/improvements\n--In the introduction the authors should spend a little more two or three words in explaining on which basis Adebayo et al. 2018 questions the correctness of the heatmap, since this is something on which the authors are building their hypothesis.\n--The title is misleading, the authors are talking about generic feature removal but in reality we are considering the image domain only.(check)\n--Figure 1’s caption should report the references for the three SP LIME and MP\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeyNBBAqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to improve perturbation-based explanation techniques by complementing the perturbation step with an inpainting step that removes artefacts caused by it.\n\nThe approach is sound and intuitive.\n\nThe authors show the flexibility of their approach by applying it to a variety of perturbation-based attribution techniques.\n\nIt is unclear what is the computational cost of the inpainter. Perturbation-based explanations are generally quite slow due to having to evaluate the function many times, and therefore a further slowdown could harm practical use. I'm curious whether the inpaiting approach could be in some way also extended to faster explanation techniques (e.g. gradient-based, or propagation-based).\n\nEvaluation experiments are not fully conclusive. Bounding box experiments are rather indirect and the deletion/insertion metrics do not systematically show the performance improvement of using inpainting. Perhaps the deletion metric should have been equiped with inpainting as well in order to avoid deletion artefacts. (See e.g. Samek'17 MoRF / LeRF experiments where various perturbation schemes are tested for deletion).\n\nThe evaluation benchmark is restricted to perturbation-based approaches. It could have been useful to broaden the comparison to non-perturbation approaches.\n\nAn experiment I found particularly interesting is the robustness to perturbation hyperparameters. Given the difficulty of designing evaluation metrics that can support hyperparameter selection, hyperparameter insensitivity is indeed strongly desirable.\n\nI'm wondering whether it is really necessary to use a strong deep neural network inpainter since the goal is just to remove artefacts. Some inpainters provided as part of standard computer vision libraries work quite well, and do not need to be trained and adapted to a certain shape of missing data.\n\nOverall, the paper presents an interesting and sound approach to improve perturbation-based explanations. Experiments are extensive, although some of them remain so far not fully conclusive.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Perturbation-based methods often produce artefacts that make the perturbed samples less realistic. This paper proposes to corrects this through use of an inpainter.  Authors claim that this results in more plausible perturbed samples and produces methods more robust to hyperparameter settings. 
Reviewers found the work intuitive and well-motivated, well-written, and the experiments comprehensive.
However they also had concerns about minimal novelty and unfair experimental comparisons, as well as inconclusive results. Authors response have not sufficiently addressed these concerns.
Therefore, we recommend rejection.
Decision: Reject