Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors analyze show theoretical shortcomings in previous methods of explaining neural networks and propose an elegant way to remove these shortcomings in their methods PatternNet and PatternAttribution.

The quest of visualizing neural network decision is now a very active field with many contributions. The contribution made by the authors stands out due to its elegant combination of theoretical insights and improved performance in application. The work is very detailed and reads very well.

I am missing at least one figure with comparison with more state-of-the-art methods (e.g. I would love to see results from the method by Zintgraf et al. 2017 which unlike all included prior methods seems to produce much crisper visualizations and also is very related because it learns from the data, too).

Minor questions and comments:
* Fig 3: Why is the random method so good at removing correlation from fc6? And the S_w even better? Something seems special about fc6.
* Fig 4: Why is the identical estimator better than the weights estimator and that one better than S_a?
* It would be nice to compare the image degradation experiment with using the ranking provided by the work from Zintgraf which should by definition function as a kind of gold standard
* Figure 5, 4th row (mailbox): It looks like the umbrella significantly contributes to the network decision to classify the image as "mailbox" which doesn't make too much sense. Is is a problem of the visualization  (maybe there is next to no weight on the umbrella), of PatternAttribution or a strange but interesting a artifact of the analyzed network?
* page 8 "... closed form solutions (Eq (4) and Eq. (7))" The first reference seems to be wrong. I guess Eq 4. should instead reference the unnumbered equation after Eq. 3.

Update 2018-01-12: Upgraded Rating from 7 to 8 (see comment below)
review_title: Very interesting and clear work

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary of article: 
This paper organizes existing methods for understanding and explaining deep neural networks into three categories based on what they reveal about a network: functions, signals, or attribution. “The function extracts the signal from the data by removing the distractor. The attribution of output values to input dimensions shows how much an individual component of the signal contributes to the output…” (p. 5). The authors propose a novel quality criterion for signal estimators, inspired by the analysis of linear models. They also propose two new explanatory methods, PatternNet (for signal estimation) and PatternAttribution (for relevance attribution), based on optimizing their new quality criterion. They present quantitative and qualitative analyses comparing PatternNet and PatternAttribution to several existing explanation methods on VGG-19.

* Quality: The claims of the paper are well supported by quantitative results and qualitative visualizations. 
* Clarity: Overall the paper is clear and well organized. There are a few points that could benefit from clarification.
* Originality: The paper puts forth an original framing of the problem of explaining deep neural networks. Related work is appropriately cited and compared. The authors's quality criterion for signal estimators allows them to do a quantitative analysis for a problem that is often hard to quantify.
* Significance: This paper justifies PatternNet and PatternAttribution as good methods to explain predictions made by neural networks. These methods may now serve as an important tool for future work which may lead to new insights about how neural networks work. 

Pros:
* Helps to organize existing methods for understanding neural networks in terms of the types of descriptions they provide: functions, signals or attribution.
* Creative quantitative analyses that evaluate their signal estimator at the level of single units and entire networks.

Cons:
* Experiments consider only the pre-trained VGG-19 model trained on ImageNet. Results may not generalize to other architectures/datasets.
* Limited visualizations are provided. 

Comments:
* Most of the paper is dedicated to explaining these signal estimators and quality criterion in case of a linear model. Only one paragraph is given to explain how they are used to estimate the signal at each layer in VGG-19. On first reading, there are some ambiguities about how the estimators scale up to deep networks. It would help to clarify if you included the expression for the two-component estimator and maybe your quality criterion for an arbitrary hidden unit. 
* The concept of signal is somewhat unclear. Is the signal 
    * (a) the part of the input image that led to a particular classification, as described in the introduction and suggested by the visualizations, in which case there is one signal per image for a given trained network?
    *  (b) the part of the input that led to activation of a particular unit, as your unit wise signal estimators are applied, in which case there is one signal for every unit of a trained network? You might benefit from two terms to separate the unit-level signal (what caused the activation of a particular unit?) from the total signal (what caused all activations in this network?).
* Assuming definition (b) I think the visualizations would be more convincing if you showed the signal for several output units. One would like to see that the signal estimation is doing more than separating foreground from background but is actually semantically specific. For instance, for the mailbox image, what does the signal look like if you propagate back from only the output unit for umbrella compared to the output unit for mailbox? 
* Do you have any intuition about why your two-component estimator doesn’t seem to be working as well in the convolutional layers? Do you think it is related to the fact that you are averaging within feature maps? Is it strictly necessary to do this averaging? Can you imagine a signal estimator more specifically designed for convolutional layers?

Minor issues: 
* The label "Figure 4" is missing. Only subcaptions (a) and (b) are present.
* Color scheme of figures: Why two oranges? It’s hard to see the difference.
review_title: Important framework, tools, and criterion for understanding deep neural networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: I found this paper an interesting read for two reasons: First, interpretability is an increasingly important problem as machine learning models grow more and more complicated. Second, the paper aims at generalization of previous work on confounded linear model interpretation in neuroimaging (the so-called filter versus patterns problem). The problem is relevant for discriminative problems: If the objective is really to visualize the generative process,  the "filters" learned by the discriminative process need to be transformed to correct for spatial correlated noise. 

Given the focus on extracting visualization of the generative process, it would have been meaningful to place the discussion in a greater frame of generative model deep learning (VAEs, GANs etc etc). At present the "state of the art" discussion appears quite narrow, being confined to recent methods for visualization of discriminative deep models.

The authors convincingly demonstrate for the linear case, that their "PatternNet" mechanism can produce the generative process (i.e. discard spatially correlated "distractors"). The PatternNet is generalized to multi-layer ReLu networks by construction of node-specific pattern vectors and back-propagating these through the network. The "proof" (eqs. 4-6) is sketchy and involves uncontrolled approximations. The back-propagation mechanism is very briefly introduced and depicted in figure 1.

Yet, the results are rather convincing. Both the anecdotal/qualitative examples and the more quantitative patch elimination experiment figure 4a (?number missing) 

I do not understand the remark: "However, our method has the advantage that it is not only applicable to image models but is a generalization of the theory commonly used in neuroimaging Haufe et al. (2014)."  what ??

Overall, I appreciate the general idea. However, the contribution could have been much stronger based on a detailed derivation with testable assumptions/approximations, and if based on a clear declaration of the aim.


review_title: Interesting, but premature contribution on interpretability 

Meta-review: The paper shows that many of the current state-of-the-art interpretability methods are inaccurate even for linear models. Then based on their analysis of linear models they propose a technique that is thus accurate for them and also empirically provides good performance for non-linear models such as DNNs.
Decision: Accept (Poster)