Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new method for DNN interpretability based on LIME, itself based on ensembling of large numbers of low-complexity models called "local explanation models". This method allows to better capture the relative importance of each feature, and is also able to recover the class-specific signals that DNNs use to distinguish between a number of classes. 

Score: Weak Accept
While this method seems like a good step forward, I am uncertain about its significance, mostly because of the choice of dataset and because it appears to be a minor change to a well known algorithm.

I think some high-level aspects could be better motivated:
- Why is interpreting black-box models a better avenue than building interpretable models?
- Why should we expect this method to work beyond MNIST? Will it work on large images where translational invariance is a vital aspect? Will it work on non-image data?
- Why does this relatively subtle change from LIME create such a dramatic visual difference? Do the authors have a stronger intuition than what is hinted at in the paper?

Other comments:
- The human evaluation is very interesting, and suggests that this method correlates much better with human judgement than previous ones. One key aspect that is missing, but hinted at, is where this difference comes from. It could be from the proposed change or from the class-specific information. (It should be easy to run KAR to test this?)
- It's not always clear how this should be reproduced. For example, the authors specify the number of samples for SmoothGrad and VarGrad but not LIME nor NormLIME.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Months ago, I read this article on arxiv (https://arxiv.org/pdf/1909.04200.pdf). It is an interesting work that tries to propose a simple yet effective interpretable model. I am not familiar with this research direction, and I try to make an educated guess.

Pros:
-- As the author suggested, the method is simple and effective.
-- The authors conducted user studies to demonstrate that the results generated by their proposed method is strongly favored over previous methods.

Cons:
-- Subscripts in equations need improvement to make them consistent. For example, in Equation (7), we have E_{y}, but in Equation (8), we have E_{Y_j} and E_{Y_k}.
-- Section 4, Figure 3, top, it seems obvious to choose the fourth one to distinguish the number 9? I feel this example is too easy and not convincing enough.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
This paper proposes a novel scheme to aggregate local explanations
from LIME to generate global and/or class-level interpretations in the
form of feature importances. The technical novelty lies in the
normalization of the feature weights across local explanations where
the proposed scheme utilizes a well-motivated L0 normalization in
place of the existing L2 normalization in Ribeiro et al., 2016. The
sparsity induced by the L0 normalization leads to more "easy-to-read"
global (or class level) "explanations" since the number of features
with non-zero importances is reduced.


The simple idea of generating sparse aggregations of local
explanations to obtain global (or class-level) interpretations is well
motivated and seems fairly intuitive (and I consider simplicity a
strength) and the extremely favourable empirical results, I am leaning
towards a reject for the current version of the paper. The main
reasons for this decision are as follows: 

- The proposed scheme is presented as closely tied to LIME, which
  severely limits the scope of the proposed solution. However, I can
  imagine that the high level idea of sparse aggregation of local
  explanations can easily generalize to many local explanation
  schemes. 
- The empirical evaluation needs to be improved (or better clarified)
  in my opinion. There are many potential baselines that are also not
  considered (see details below). Moreover, the evaluation is
  completely limited to a single data set (which is good as a strawman
  but not sufficient to make significant claims).


Clarifications re: baselines and empirical evaluations:

- No comparison to Ibrahim, et al., 2019 -- you can cluster in
  unsupervised setting but you can also aggregate within class in
  supervised setting. Generating class-level interpretations is an
  interesting idea but its novelty is somewhat unclear.
- In terms of explanations, there are contrastive explanations [1]
  which are not really discussed in this paper. This scheme could be
  more useful for generating class level interpretations.
- It is not clear why the negative correlations were removed in the
  user study. In certain situations, such as when distinguishing
  between digits 3 & 8, the absence of weights (potentially
  corresponding to negative weights) can be crucial explanations.
- It is not clear why in the SmoothGrad and VarGrad baselines, the
  class level interpretations are generated with just 10 images per
  class. Are the class level interpretations of LIME and NormLIME also
  generated from just 10 images per class. If that is the case, 10
  seems like a very small number and can potentially create noisy
  explanations. If LIME and NormLIME used more than 10 images to
  generate class level interpretations, the comparison does not seem
  fair. Please explain this discrepancy.
- For the evaluation in Sec. 5, it is not clear how the features are
  removed. Are they removed based on the global explanations (feature
  importances) of each baseline? If this is the case, are the global
  explanations of each baseline generated using the same amount of
  local explanations? Please clarify.
- The KAR analysis results in Fig. 4(b) need to be further
  investigated. First, it is not clear if this difference between the
  different baselines something that happens in multiple datasets or
  just this one. Moreover, it seems natural that something like
  SmoothGrad and VarGrad would be able to capture the feature
  redundancies if the redundancies were the actual cause for this
  marked difference between the relative performances in Fig. 4(a) and
  4(b). This could definitely use a better discussion.


[1] Dhurandhar, Amit, et al. "Explanations based on the missing:
Towards contrastive explanations with pertinent negatives." Advances
in Neural Information Processing Systems. 2018. 
review_title: Official Blind Review #1

Meta-review: The paper aims to extract the set of features explaining a class, from a trained DNN classifier.

The proposed approach relies on LIME (Ribeiro et al. 2016), modified as follows: i) around a point x, a linearized sparse approximation of the classifier is found (as in LIME); ii) for a given class, the importance of a feature aggregates the relative absolute weight of this feature in the linearized sparse approximations above; iii) the explanation is made of the top features in terms of importance.

This simple modification yields visual explanations that significantly better match the human perception than the SOTA competitors. 

The experimental setting based on the human evaluation via a Mechanical Turk setting is the second contribution of the approach. The feature importance measure is also assessed along a Keep and Retrain mechanism, showing that the approach selects actually relevant features in terms of prediction. 
Incidentally, it would be good to see the sensitivity of the method to parameter $k$ (in Eq. 1).

As noted by Rev#1, NormLIME is simple (and simplicity is a strength) and it demonstrates its effectiveness on the MNIST data. However, as noted by Rev#4, it is hard to assess the significance of the approach from this only dataset. 

It is understood that the Mechanical Turk-based assessment can only be used with a sufficiently simple problem.  However, complementary experiments on ImageNet for instance, e.g., showing which pixels are retained to classify an image as a husky dog, would be much appreciated to confirm the merits and investigate the limitations of the approach.

Decision: Reject