Title: Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Authors: Luisa M Zintgraf
Taco S Cohen
Tameem Adel
Max Welling
Venue: ICLR 2017
Keywords: Deep learning
Applications

Abstract: This article presents the prediction difference analysis method for visualizing the response of a deep neural network to a specific input. When classifying images, the method highlights areas in a given input image that provide evidence for or against a certain class. It overcomes several shortcoming of previous methods and provides great additional insight into the decision making process of classifiers. Making neural network decisions interpretable through visualization is important both to improve models and to accelerate the adoption of black-box classifiers in application areas such as medicine. We illustrate the method in experiments on natural images (ImageNet data), as well as medical images (MRI brain scans).

TL;DR: Method for visualizing evidence for and against deep convolutional neural network classification decisions in a given input image.

Review discussion:
Did you use some strategy to choose the example images you are presenting (e.g. for Figure 3, you could have picked the images where the difference between marginal sampling and conditional sampling was largest), did you choose randomly or did you handpick the examples?
We chose the images from among a small set of images in order to show a range of behavior of the algorithm. The shown images are quite representative of the performance of the method in general.
E.g. for Figure 3, we picked the images that are both interesting and representative; there exist examples where the difference it not as obvious, but we never observed marginal sampling to give better results than conditional sampling.
Thank you for your answer! Your samples look very impressive.
Of course one might object that this is just due to the hand picking. As a suggestion, probably you could easily prove that your method works well in general by showing a figure with something like 8x4 random stimuli from ImageNet alongside the visualization of the correct output unit (and pointing out that the stimuli are random). I think this would make a very convincing point.
The authors of this work propose an interesting approach to visualizing the predictions made by a deep neural network. The manuscript is well written is provides good insight into the problem. I also appreciate the application to medical images, as simply illustrating the point on ImageNet isn't interesting enough. I do have some questions and comments.
1.  As the authors correctly point out in 3.1, approximating the conditional probability of a feature x_i by the marginal distribution p(x_i) is not realistic. They advocate for translation invariance, i.e. the position of the pixel in the image shouldn't affect the probability, and suggest that the pixels appearance depends on the small neighborhood around it.
However, it is well known that global context makes an big impact on the semantics of pixels. In "Objects in Contexts", authors show that a given neighborhood of pixels can take different semantic meanings based on the global context in the image. In the context of deep neural networks, works such as "ParseNet" also illustrate the importance of global context on the spatial label distribution. This does not necessarily invalidate this approach, but is a significant limitation. It would be great if the authors provided a modification to (4) and empirically verified the change.
2. Figure 7 shows the distribution over top 3 predictions before and after softmax. It is expected that even fairly uniform distributions will transform toward delta functions after softmax normalization. Is there an additional insight here?
4.
Finally, in 4.1, the authors state that it takes 30 minutes to analyze a single image with GooLeNet on a GPU? Why is this so computationally expensive? Such complexity seems to make the algorithm impractical and analyzing datasets of statistical relevance seems prohibitive.
Thank you very much for your review and input. We are going to address the comments/questions below.
1. Ideally, we would condition on the full image to sample individual pixels, i.e., p(x_i | x_\i). Since it is computationally infeasible to model such a probability distribution, we resort to an approximation and in this case decided to make the simplification of assuming translation invariance (equation 4) and conditioning only on a small neighborhood around the pixel.
However, as the reviewer points out, the pixel probability could also depend on the context of the larger image (an illustrative example in "Objects in Contexts" is that a yellow blob can be a lemon or a tennis ball, depending on the scene in the image). We still think that given a small enough patch (k in alg. 1) that is marginalized, the pixel values (which are somewhat but not entirely coupled to their semantic meaning) are mostly dependent on their neighborhood.
Having said that, we can try to think of ways to modify (4) to get an even better approximation by using a conditional distribution that takes more information about the whole image into account. This does not necessarily have to be all the raw pixels, but could be (as pointed at by the reviewer) more semantic information, like scene labels.
In fact, for the MRI scans we also performed experiments where we split up the 3D image into a 20x20x20 grid and also provide the Gaussian distribution with the index in that grid, i.e., instead of (4) we use p(x_i | x^_\i, grid_index), since the distribution of pixel values in the special case of MRI scans does depend on spacial location as well. We found that this slightly improves the interpretability of the results.
It would be an interesting next step to analyze in more detail how a modification of (4), or a stronger probabilistic model, influences the results.
2. What we observed in our experiments is that usually the explanations for the first and second highest classes after softmax are pretty much complementary (i.e., what speaks for one class does not speak for the other and vice-versa) and that the prediction difference for the lower classes has a much lower magnitude and is also less visually expressive. As the reviewer points out, the softmax tends to enlarge differences in its inputs, so we expect that for a given pixel, the intensities of the output across classes is close to a 1-hot-vector. We think this explains why when having many classes (1000 for the ImageNet dataset) for the majority of low-scoring output classes, the visualizations are not very meaningful.
Still, for the top 2-3 scoring classes the outputs of the softmax are sensitive enough to small changes in input space, and Figure 7 nicely illustrates how the classifier uses the softmax to weigh the top scoring classes against each other, and ultimately decides for one of the classes (even though they are very similar, like different dog breeds).
4. Given our default settings, for each ImageNet image, we have to roughly make 227x227x10 (around 17,000) evaluations - i.e., for each patch (~227x227 patches) take samples (10) from the multivariate Gaussian and forward pass the image through the network (it's a bit less than this, depending on the actual input size of the network and the k in algorithm 1). Analyzing one image therefore depends strongly on how fast we can sample pixel values, and even more on how fast we can evaluate the classifier.
We agree with the reviewer that for some datasets, this makes this approach less practical than other methods. E.g., the sensitivity map from Simonyan et al. (2013) requires only a single backward pass through the network and is therefore very fast compared to our method.
However we also believe that there are many cases where a longer waiting time is worth having a more expressive explanation (given that we can provide signed information w.r.t. the support for/against single classes compared to the sensitivity map).
For example in a medical setting we believe it would be acceptable to even wait 1-2 days for a very insightful analysis and explanation of the individual patient's data (e.g., an MRI scan). For datasets like ImageNet, other methods are generally more practical for a quick analysis (or live analysis of videos like in Yosinski's deepvis toolbox).
Still, compared to the training time these DCNNs usually take we think that it might be feasible to use our method to get additional insight into how the DCNN makes decisions (e.g., let it run for a week and have around 300 images analyzed).
The authors propose a way to visualize which areas of an image provide mostly influence a certain DNN response mostly. They apply some very elegant and convincing improvements to the basic method by Robnik-Sikonja and Konononko from 2008 to DNNs, thus improving it's analysis and making it usable for images and DNNs.
The authors provide a very thorough analysis of their methods and show very convincing examples (which they however handpicked. It would be very nice to have maybe at least one figure showing the analysis on e.g. 24 random picks from ImageNet).
One thing I would like to see is how their method compares to some other methods they mention in the introduction (like gradient-based ones or deconvolution based ones).
They paper is very clearly written, all necessary details are given and the paper is very nice to read.
Alltogether: The problem of understanding how DNNs function and how they draw their conclusions is discussed a lot. The author's method provides a clear contribution that can lead to further progress in this field (E.g. I like figure 8 showing how AlexNet, GoogLeNet and VGG differ in where they collect evidence from). I can think of several potential applications of the method and therefore consider it of high significance.
Update: The authors did a great job of adopting all of my suggestions.
Therefore I improve the rating from 8 to 9.
Thank you very much for your review. We understand that for the reader it would make a convincing point to see results on random picks, and therefore ran additional experiments on 34 randomly selected ImageNet images. We also added the results of the method from Simonyan et al. (2013) for direct comparison. Please see https://www.dropbox.com/s/0eoe1krqg4m6gv8/results_random_legend.png for the results (we will add them to the appendix of the paper in a revised version).
Further, we made our code publicly available, see https://github.com/lmzintgraf/DeepVis-PredDiff .
Thank you, the new plot looks very nice and helpful! I have one additional suggestion: For your own visualizations, "white" means "no relevance", while for the comparision plots from Simonyan "black" has the meaning of "no relevance". I would recomment to invert the color scale for the Simonyan visualizations to make this more comparable.
Additionally I would expect the Simonyan images to be more readable with the inverted color scale because it is easier to see small structure in black on white instead of white on black.
I noticed that for some images the ImageNet class was not obvious to me -- mainly because I don't know all 1000 ImageNet classes. It might be worth adding the correct (or predicted) class name e.g. in vertical to the left of each row.
Kudos for releasing the code!
If you give changing the Simonyan color scale a try, I will happily increase my rating.
Thank you for your comment.
The reason we went with white on black was that Simonyan et al. use this (https://arxiv.org/pdf/1312.6034.pdf, page 5).
But we agree, for comparison to our method it's better if white parts also correspond to pixels that have no influence on the result.
We tried black on white (https://www.dropbox.com/s/6bkg27i4uz84iew/results_random_wb.png?dl=0) and red on white (https://www.dropbox.com/s/0oa2lx2d06hn5yc/results_random_rw.png?dl=0) now, and think that the latter is the most interpretable (and still fits well to the color scheme we use for our method).
Thankyou for giving it a try! I agree, the red color scheme is even better. The added captions additionally make the results more interpretable because now one can see what the network has been looking for. As promised, I increased the grade of my review and I look forward to trying the code myself!
Thanks a lot for the increased rating, and valuable input for improving the paper - we have updated it accordingly.
The paper presents a theoretically well motivated for visualizing what parts of the input feature map are responsible for the output decision. The key insight is that features that maximally change the output and are simultaneously more unpredictable from other features are the most important ones. Most previous work has focused on finding features that maximally change the output without accounting for their predictability from other features. Authors build upon ideas presented in the work of Robnik-Šikonja & Kononenko (2008).
The results indicate that the proposed visualization mechanism based on modeling conditional distribution identifies more salient regions as compared to a mechanism based on modeling marginal distribution. I like that authors have presented visualization results for a single image across multiple networks and multiple classes. There results show that the proposed method indeed picks up on class-discriminative features. Authors have provided a link to visualizations for a random sample of images in a comment – I encourage the authors to include this in the appendix of the paper.
My one concern with the paper is – Zeiler et al., proposed a visualization method by greying small square regions in the image. This is similar to computing the visualization using the marginal distribution. Authors compute the marginal visualization using 10 samples, however in the limit of infinite samples the image region would be gray. The conditional distribution is computed using a normal distribution that provides some regularization and therefore estimating the conditional and marginal distributions using 10 samples each is not justified.
I would like to see the comparison when grey image patches (akin to Zeiler et al.) are used for visualization against the approach based on the conditional distribution.
Thank you very much for your review.
The method of Zeiler et al. is indeed similar to our method in that they also remove information in of the image (by occluding parts of it with a grey patch) to evaluate how important image regions are (by looking at the correct class probability as a function of the grey patch, whereas we look at the difference in output probabilities).
However we think that when using this grey patch, there is not only information removed/occluded from the image, but also new information introduced - a grey patch looks like *something* to the classifier (e.g., the sky on a rainy day, the carrosserie of a grey car) and we therefore think that by just replacing pixels with their mean value (which is approx grey for ImageNet) could bias the results depending on what the classifier learned about which images usually contain grey areas. The output of the classifier would therefore always shift towards the classes that frequently contain a lot of grey. By instead using patches from all classes (like in the marginal distribution) and averaging the class scores, this bias can be removed (and even further by using the conditional distribution).
We ran some more experiments, and you can see the results when using a grey patch instead of conditional sampling here:
https://www.dropbox.com/s/af5n8frrn0g0mh6/comparison_graySquare.png
For many examples the difference is not very large, but for example in the "dam" or "scuba diver" image we can see how the method assigns too much relevance to rather uniform (and uninformative) regions like the sky. Given this and our theoretical reasoning, we believe that using conditional sampling is in general more sensible.
However, an interesting subject of future research is to use more sophisticated models for conditional sampling, which might also include more information from the whole image (as another reviewer suggested). We believe that unimportant regions that are easily predictable by other (neighbouring) pixels could then be downweighed even more which would lead to even more interpretable results.
The reason we used 10 samples for both marginal and conditional sampling were that on the one hand we didn't see a significant difference when using more samples, and also we wanted to have comparable computation times (conditional sampling takes just slightly longer due to the sampling procedure).
Also, marginal sampling in the limit does not correspond to  just using one sample of the mean pixel value (i.e., approx grey), since the expectation cannot just be pushed into the computation of the class probabilities.
We have included the visualisations for random images (comparing to the method of Simonyan et al.) in the appendix of a revised version of the paper.
Thanks for including the comparison with Zeiler et al.
While for categories such as "dam" and "scuba diver" the proposed method works better, for other categories such as "dish rag" - the grey area based method seems to work better. It seems like if the content of interest such as sky or rag has uniform texture then the proposed method will not capture the entire area. What do the authors think about it?
Secondly, it would be great if quantitative evaluation between the proposed and past approaches is made. One way would be to use the PASCAL dataset that has segmentation masks for objects. The segmentation masks obtained by thresholding the heatmaps can be compared against these segmentation masks. This comparison won't be ideal because for many objects only a subset of pixels might constitute discriminative regions and thus if the visualization technique correctly identifies these pixels it will be penalized in the intersection over union metric used for evaluation segmentation predictions. This issue can be partially mitigated by using the following metric - count of pixels within the object mask / count of pixels outside the object mask. This metric is going to penalize false positives (i.e. regions outside the object masks that are thought to be important but are actually not). Even this metric is not perfect, because the CNN might be using context for classification.
However, the standard segmentation and the metric proposed above are good starting points for quantitative evaluation.
Thank you again for your comment.
Here are some thoughts on why we think the grey patch method seems to work better in the “dish rag” example:
- Since the entire image is so uniform, we think that using a grey patch always introduces the same amount of different information; i. e, the original image + grey patch all look very similar (independent of where the grey patch is) but at the same time very different from the original image (without the grey patch). This way, the prediction difference is always very high (the classifier is drawn to prediction some other class that is closer to things with grey patches) across the whole region. Our method on the other side puts samples into the image that try to fill out the missing region.
The differences are much more subtle, and the prediction difference more sensitive to the quality of the conditional sampling. This problem seems to occur, as you point out, if the content of interest has uniform texture - if there are uniform regions in the image plus some other object of interest (e. g. , the parachute) we do not get this behaviour. We believe that using a better model for the conditional distribution in this case could lead to better results. We used a normal joint distribution over pixel patches, which might not be sufficient to replicate texture well enough.
- Another thing to investigate would be to find out how the CNN usually makes a "dish rag" prediction (in this example, it misclassifies it and ranks "dish rag" as the 4th probable class); is it the texture, color, or shape? That way we could get a better sense of what the CNN is looking for, and make a better evaluation of the results.
Quantitative evaluation of methods that explain classifier decisions we believe are difficult, since it is not clear what the desired output is, and thus how we should evaluate an explanation. Segmentation masks are very close to how human decision-making is done, but can be very different from how a classifier makes decisions. The main issue is that the classifier might indeed (as you pointed out) not focus on the object itself, but more on contextual information. In our experiments we observed that the method of Simonyan et al. (sensitivity maps using partial derivatives of the class score wrt the input pixels) more often tends to highlight the object itself, compared to our method.
Simonyan et al. have demonstrated in their paper that their method can be used for segmentation tasks, but we do not think that our method is necessarily suited for that since it serves a different purpose. Simonyan et al. highlight for which pixels the CNN output is most sensitive to small changes in value, whereas our proposed method evaluates how much and in what way image regions influence the decision.
Consequently, our method might 'split up' objects: part of a cat can look like a tabby cat, whereas a different part can look more like a siamese cat (and there are about 5 cat species in the ImageNet classes). That means we would have to use the thresholded heat maps of the absolute values, which in turn could lead to issues for other classes.
Therefore, we believe that finding appropriate evaluation metrics is an interesting direction for future work.

Meta-review:
role: PC
data: comment: Reviewers felt the paper was clearly written with necessary details given, leading to a paper that was pleasant to read. The differences with prior art raised by one of the reviewers were adequately addressed by the authors in a revision. The paper presents results on both ImageNet and medical imagery, an aspect of the paper that was appreciated by reviewers.

Decision: Accept (Poster)