Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a method to capture patterns of “off” neurons using a newly proposed metric. While the authors have considered only linear networks, the setup is still relevant because how often these networks can give meaningful results, and can possibly pave the way for future research into more general networks. 

Pros: The idea itself is interesting, the related works are discussed well, and MNIST experiments are very interesting. 

Cons/comments : The writing needs a lot of improvement if to be considered for a top venue like iclr.  Other than the MNIST experiments, which show and indicate the importance of relative contrast and boundary, I am not sure how other experiments are meaningful. CIFAR and smallnorb experiments are merely presented, without discussions on what the interpretation shows or helps over the existing methods. Infact, the other methods seem to capture a lot more information than the proposed method. I would suggest adding more discussions and more experiments that show interpretation that this method/metric helps with to make this work stronger.

Have the authors considered  the metric to consider “on” neurons instead of “off”neurons ? Is it possible to have a general metric that combines the two in some way ? Intuitively, its unclear to me why only the off patterns can help (except in specific cases as shown in MNIST experiments). 

 “and thus is responsible for the activity vi” – This is unclear to me. I understand the projection part though, but I cant make sense of this statement.

“interpretation and interpretability ” in the introduction – the writing is too informal. Making use of italicized phrases like “switched linear projection” does not help with the understanding at all, especially because “switched” is defined after using the term atleast thrice. 

Confirmation bias <-> “information we want to get”. 

The same issue right after eq 6. “Reference subtracted from ….” where the first word is italicized to probably imply some intuitive explanation, but for someone not familiar with what reference is just tends to confuse the reader.  
Please fix missing references.

Eq 7 seems written incorrectly, with the where “v= …”. Please fix. 

What is the variance of saliency checks ? In other words, if the experiment of 100 random samples is repeated (say) 20 times, how different are the corresponding coefficients across these repeated runs ?

Figure 3 is waste of space (move to appendix?)

I might be splitting the hairs but Theorem 1 does not warrant a theorem. The result/proof is too straightforward to be a theorem and is already known in some form in the folklore. 


review_title: Official Blind Review #7

final_score: rating: 3: Weak Reject
review_content: This manuscript introduces a novel method to explain activities of ReLU-based deep networks by constructing a linear subnetwork which only contains neurons activated by the input. The status of each neuron can be obtained given any input sample. Moreover, the author applies the notion of “neuron’s center”, which is a neutral data point that is similar to actual input x, but with differences in particular objects to cause f(x) be positive. The activity of each neuron can be decomposed into the attribution of each input pixel, and this decomposition can also be used to measure the contribution of each pixel to the network stability. Overall, the proposed methodology is intuitive and distinctive to the state-of-the-art interpretability methods.

However, the application constraint on the ReLU-based deep neural network prevents this method from being a model agnostic approach: the problem formulation would be much different if other non-linear activation functions are used. Although the experiment part visualizes the superiority of switched linear projections over other prevalent approaches, the evaluations contain mostly subjective assessment and the arguments are monotonous. I would suggest adding more experiments with quantitative analysis, or mathematically demonstrate why the proposed method is better than, say purely gradient-based method, in the linear case. In addition, additional experiments on a broader set of input data (e.g., tabular, text) could avoid the evaluations look cherry-pick. 

Minor issues: 

1. In figure 2, I think it would be better to write down explicitly the connections between v, \hat{b} and \hat{w} for each neuron given any input. Just seeing v and \hat{b} on top of each subfigure is a bit confusing. 
2. I spent a long time to understand the "neuron’s center" concept, it might be better to add some background or mathematical formulation.
3. In figure 4, when the digits get misclassified, the Insens explanation should highlight the patterns of wrongly predicted digits, but the patterns of neurons' inactive state sensitivity still look like the correct digits.
4. It would also be interesting to show how the Insens explanation would change when the input is under various kinds of adversarial attacks rather than adding simple Gaussian noise.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The work basically introduced a new way of looking at interpretability; instead of focusing on the source of activations in the network for a given input image, focus on the source of stability (non-active) neurons (in a ReLU network). The work starts by proving (although it is trivial) that in a ReLU (more generally any piece-wise linear) network, for a given input image, there is a locally linear relationship between a given neuron's activation and the image: v= w^T x + b. As the authors correctly mention, focusing on 'w' as the sensitivity analysis is basically the vanilla gradient method. The contribution, however, is focusing on the projection of bias and the introduced notion of 'centre'. With this provided notion, one can focus on the deactivated neurons in the network and how each input pixel is responsible for it. In other words, unlike previous work that focuses on the activation map, the authors correctly refer to the deactivated neurons as another source of the network's prediction.

I'm have reasons for both accepting and rejecting this work.  The work provides a new perspective and asks a very interesting question. The introduced method, although quite simple and trivial, is useful and the authors do a very good job of making valid and reasonable claims about their work's contribution and how it connects to the existing literature. The main drawback of the paper, however, is whether the contributions are enough for this venue. The paper does not convince me that the introduced method would result in better interpretability of deep networks compared to what is already there. Another minor (or for some people in the field major) issue is the experimental setup.  All of the experiments are focused on subjective examples and no objective measure of the introduced method is provided (and the field has many of those objective measures). Providing a few examples of the method in comparison with other methods is not sufficient. Anyhow, the experiment where they prove the usefulness of the method by adding background noise is interesting. I would personally suggest the authors to expand this experiment to testing the method's sanity using the sanity measures provided in previous work: https://arxiv.org/abs/1810.03292 The claims made about the results on smallNORB can be controversial as the authors interpret their method's flipping of importance to be the reality of what's happening in the network and the other method's focus on the edges as false; this is not clear to be true. My score would be subject to change if better experimental results are provided (and the other way round).

A few suggestions and questions:
* One very important issue with the method is that it considers all of the inactive neurons. We know that a substantial percentage of inactive neurons are just dead neurons the stability of which does not matter. How would the method address the issue?
* There definitely needs to be an objective measure of the introduced method's performance compared to previous work.
* The work seems very related to DeepLIFT while there is no mentioning of it.
* I'm not a fan, but adding results on a SOTA ImageNET paper always helps with making the experiments section crisper.
* The authors claim that even small perturbations will change the activation pattern. This is not a small claim and is definitely in need of more evidence.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Notes: 

  -Goal is to study the "active subnetwork" of Relu based networks for interpretability.  

  -The question of interpretation seems rather thorny.  

  -In Figure 4, the result for Insens seem alright, although it's weird that the data is just mnist digit / noise.  I feel like something with multiple objects would make it much clearer if there is an actual improvement?  For example on Figure 5 I'm not really sure if Insens is better.  The results often look worse to me than "DeepTaylor", especially on CIFAR10.  

Review: This paper proposes to improve the interpretation of relu based networks by considering the "inactive network" which could potentially become activated by local perturbations instead of just considering the active part of the network (which is locally linear).  I think this is a step in the right direction for the interpretation of relu based networks, although the results are somewhat borderline.  

Additionally the tasks could be much better, to show situations where an object is present but which is not related to the labels.  This would provide a much clearer test of the model's capabilities.  


review_title: Official Blind Review #1

Meta-review:  This paper proposes a method to capture patterns of the so called “off” neurons using a newly proposed metric. The idea is interesting and worth pursuing. However, the paper needs another round of modification to improve both writing and experiments. 
Decision: Reject