Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a theoretically well motivated for visualizing what parts of the input feature map are responsible for the output decision. The key insight is that features that maximally change the output and are simultaneously more unpredictable from other features are the most important ones. Most previous work has focused on finding features that maximally change the output without accounting for their predictability from other features. Authors build upon ideas presented in the work of Robnik-Šikonja & Kononenko (2008).

The results indicate that the proposed visualization mechanism based on modeling conditional distribution identifies more salient regions as compared to a mechanism based on modeling marginal distribution. I like that authors have presented visualization results for a single image across multiple networks and multiple classes. There results show that the proposed method indeed picks up on class-discriminative features. Authors have provided a link to visualizations for a random sample of images in a comment – I encourage the authors to include this in the appendix of the paper. 

My one concern with the paper is – Zeiler et al., proposed a visualization method by greying small square regions in the image. This is similar to computing the visualization using the marginal distribution. Authors compute the marginal visualization using 10 samples, however in the limit of infinite samples the image region would be gray. The conditional distribution is computed using a normal distribution that provides some regularization and therefore estimating the conditional and marginal distributions using 10 samples each is not justified. 
I would like to see the comparison when grey image patches (akin to Zeiler et al.) are used for visualization against the approach based on the conditional distribution. 


review_title: interesting approach to visualization

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors of this work propose an interesting approach to visualizing the predictions made by a deep neural network. The manuscript is well written is provides good insight into the problem. I also appreciate the application to medical images, as simply illustrating the point on ImageNet isn't interesting enough. I do have some questions and comments.
1.  As the authors correctly point out in 3.1, approximating the conditional probability of a feature x_i by the marginal distribution p(x_i) is not realistic. They advocate for translation invariance, i.e. the position of the pixel in the image shouldn't affect the probability, and suggest that the pixels appearance depends on the small neighborhood around it. However, it is well known that global context makes an big impact on the semantics of pixels. In "Objects in Contexts", authors show that a given neighborhood of pixels can take different semantic meanings based on the global context in the image. In the context of deep neural networks, works such as "ParseNet" also illustrate the importance of global context on the spatial label distribution. This does not necessarily invalidate this approach, but is a significant limitation. It would be great if the authors provided a modification to (4) and empirically verified the change.

2. Figure 7 shows the distribution over top 3 predictions before and after softmax. It is expected that even fairly uniform distributions will transform toward delta functions after softmax normalization. Is there an additional insight here?

4. Finally, in 4.1, the authors state that it takes 30 minutes to analyze a single image with GooLeNet on a GPU? Why is this so computationally expensive? Such complexity seems to make the algorithm impractical and analyzing datasets of statistical relevance seems prohibitive. 
review_title: Why is it so slow?

final_score: rating: null
review_content: Did you use some strategy to choose the example images you are presenting (e.g. for Figure 3, you could have picked the images where the difference between marginal sampling and conditional sampling was largest), did you choose randomly or did you handpick the examples?
review_title: How are the example images choosen?

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors propose a way to visualize which areas of an image provide mostly influence a certain DNN response mostly. They apply some very elegant and convincing improvements to the basic method by Robnik-Sikonja and Konononko from 2008 to DNNs, thus improving it's analysis and making it usable for images and DNNs.

The authors provide a very thorough analysis of their methods and show very convincing examples (which they however handpicked. It would be very nice to have maybe at least one figure showing the analysis on e.g. 24 random picks from ImageNet).
One thing I would like to see is how their method compares to some other methods they mention in the introduction (like gradient-based ones or deconvolution based ones). 

They paper is very clearly written, all necessary details are given and the paper is very nice to read.

Alltogether: The problem of understanding how DNNs function and how they draw their conclusions is discussed a lot. The author's method provides a clear contribution that can lead to further progress in this field (E.g. I like figure 8 showing how AlexNet, GoogLeNet and VGG differ in where they collect evidence from). I can think of several potential applications of the method and therefore consider it of high significance.

Update: The authors did a great job of adopting all of my suggestions. Therefore I improve the rating from 8 to 9.
review_title: Very interesting paper

Meta-review: Reviewers felt the paper was clearly written with necessary details given, leading to a paper that was pleasant to read. The differences with prior art raised by one of the reviewers were adequately addressed by the authors in a revision. The paper presents results on both ImageNet and medical imagery, an aspect of the paper that was appreciated by reviewers.
Decision: Accept (Poster)