Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper, inspired by the established technique of model ensembling, proposes two methods (AGG-Mean and AGG-Var) for aggregating different model explanations into a single unified explanation. The authors mathematically prove that the derived explanation is guaranteed to be more truthful than the average performance of the constituent explanations. In practice, the aggregation consistently outperforms *all* individual explanations, not just their aggregated performance. Additionally, the paper introduces a new quantitative evaluation metric for explanations, free of human intervention: IROF (Incremental Removal of Features) incrementally grays out the segments deemed as relevant by an explanation method and observes how quickly the end-task performance is degraded (good explanations will cause fast degradation). Solid validation confirms that the IROF metric is sound.

I support paper acceptance. The experimental section is particularly strong, and makes a convincing argument for both the aggregation methods and the IROF metric. Even though I am not very familiar with the explainability literature and I would not be able to point out an omitted baseline for instance, the wide range of model architectures and aggregated explanation techniques makes a solid case. I appreciate the experiments on low-dimensional input, where the authors are deliberately showing a scenario in which their method does not score huge gains; this brings even more credibility to the paper. The presentation itself is clear, and there are no language or formatting issues.

The only obvious downside of AGG-Mean and AGG-Var is that one would have to implement and run all constituent evaluation methods, which is expensive. Just as an idea for future work: given N explanation methods, one could ablate away one method at a time, thus getting an idea of whether any of the N explanations are redundant in the presence of others. Recommending a minimal set of useful explanation methods to the NLP community would then decrease the overall complexity of replicating the end-to-end explanation system.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper has two main messages: 1- Averaging over the explanation (saliency map in the case of image data) of different methods results in a smaller error than an expected error of a single explanation method. 2- Introducing a new saliency map evaluation method by seeking to mitigate the effect of high spatial correlation in image data through grouping pixels into coherent segments. The paper then reports experimental results of the methods introduced in the first message being superior to existing saliency map methods using the second message (and an additional saliency map evaluation method in the literature). They also seek to magnify the capability of the 2nd message's evaluation method by showing its better capability at distinguishing between a random explanation and an explanation method with a signal in it.


I vote for rejecting this paper for two main reasons: the contributions are not enough for this veneue, and the paper's introduced methods are not backed by convincing motivations. The first message of the paper is trivial and cannot be considered as a novel contribution: the ''proof'' is basically the error of the mean is smaller than the mean of the errors. Additionally, this could have been useful if the case was that there was a need for playing the safe-card: that is, all of the existing methods have equal saliency map error and averaging will decrease the risk. Not only authors do not provide any evidence but also both the experimental results of the paper itself (results in Table 2 and Fig 4 are disproving this assumption) and the existing literature disprove it. Even considering this assumption to be correct, the contribution is minimal to the field and benefits of averaging saliency maps have been known since the SmoothGrad paper. The second contribution is an extension of existing evaluation methods (e.g. SDC) where instead of removing (replacing by mean) individual pixels, the first segment the image and remove the segments. The method, apart from being very similar to what is already there in the literature, is not introduced in a well-motivated manner. The authors claim that their evaluation method is able to circumvent the problem with removing individual pixels (which is the removed information of one pixel is mitigated by the spatial correlations in the image and therefore will not result in a proportional loss of prediction power) by removing ''features'' instead. Their definition of a feature, though, are segments generated by simple segmentation methods. There is a long line of literature showing the incorrectness of this assumption; i.e. a group of coherent nearby pixels does not necessarily constitute a feature seen by the network and does not necessarily remove the mentioned problem of the high correlation of pixels. This method does not remove "the interdependency of inputs" for the saliency evalatuion metric. Even assuming the correctness of this assumption, the contribution over what already exists in the literature is not enough for this venue.

A few suggestions:

* The authors talk about a ''true explanation''. This concept needs to be discussed more clearly and extensively. What does it mean to be a true evaluation? It is also important to prove that the introduced evaluation metric of IROF would assign perfect score for a given true explanation.

* The mentioned problem of pixel correlations that IROF seeks to mitigate is also existing in other modalities of data and the authors do not talk about how IROF could potentially be extended.

* The qualitative results in the text and the appendix do not show an advantage. It would be more crips if the authors could run simple tests on human subjects following the methods in the previous literature.

* There are many many grammatical and spelling errors in the paper. The font size for Figures is very small and unreadable unless by zooming in.

* Many of the introduced heuristics are not backed by evidence or arguments. One example is normalizing individual saliency maps between 0-1 which can naturally be harmful; e.g. aggregating a noisy low-variance method with almost equal importance everywhere (plus additive noise) and a high-variance one which does a good job at distinguishing important pixels - AGG-VAR will not mitigate this issue. 

One question:

The authors introduced aggregation as a method for a ''better explanation''. It has been known that another problem with saliency maps is robustness: one can generate adversarial examples against saliency maps. It would be an interesting question to see whether aggregation would improve robustness rather than how good the map itself is.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper presents a study on explanation methods, proposing an interesting way to aggregate their results and providing empirical evidence that aggregation can improve the quality of the explanations.

The paper considered only methods using CNN for classifying images, leaving other applications for future investigation.

The results show aggregation of different explanation methods leads to better explanations, and is therefore exploitable, for high dimensional images while degrades with low-dimensional ones. In the latter case, it happens that the aggregated explanation explains a bit worse than the best non-aggregated explanation (from the graph it seems a very small difference though). This is odd because I would have assumed to see an improvement (at least a small one) using information from more than one system.

The paper also presents a score for evaluating explanation methods, which shows good results.

The paper is interesting and well written, the experimental campaign extensive enough and the methods are presented clearly.

There are some minor problems to be solved:
- In the abstract, there are two periods before the last sentence.
- When defining the size of matrices I suggest using \times instead of x to improve readability
- The caption in Figure 1 says "The decrease of the class score over the number of removed segments is reported.", but I cannot find where this decrease is reported.
- On page 5, in the parentheses it is reported that a given explanation method is no better than random choice. I would say that it is better (or not worse) than random choice, otherwise the methods would not provide any useful information.
- On page 6, forty images are considered in the text while fifty images are considered in the caption of Table 1. Please align the number with the correct one.
review_title: Official Blind Review #1

Meta-review: This paper describes a new method for explaining the predictions of a CNN on a particular image. The method is based on aggregating the explanations of several methods.   They also describe a new method of evaluating explanation methods which avoids manual evaluation of the explanations.

However, the most critical reviewer questions the contribution of the proposed method, which is simple.  Simple isn't always a bad thing, but I think here the reviewer has a point.  The new method for evaluating explanation methods is interesting, but the sample images given are also very simple -- how does the method work when the image is cluttered?   How about when the prediction is uncertain or wrong?
Decision: Reject