Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: --------- AFTER rebuttal

1) "We identify issues with current masking procedures as proposed in other papers"

One of the major issues with the current masking procedure is that the resulting image is out of the data distribution. Even though your method achieved high accuracy in Table 2 for correctness, the generates images is still out of the data distribution. 

2) "We propose a cost-effective masking technique that doesn’t require retraining of the underlying classifier"

The authors compared against zero and gray masking for correctness. None of those masking methods require retraining of the underlying classifier. It is not clear, which previous masking technique required retraining of the underlying classifier?

3) We also further show that when performing a comprehensive evaluation, there is no one clearly better explainer and thus practitioners need to be careful about which explainer they choose.

This is an observation made upon through exploratory analysis and is not a technical novelty.

4) " Confidence on the other hand, also informs us about the per-instance behavior."

The confidence measures the change in probability assigned to the ground truth class. Table 3 should also show the variance in the confidence to understand the instance-level behavior.

The experiments given in the paper, it looks like confidence and correctness are positively correlated. An example of the model where they are not positively correlated will help the reader understand the importance of each of these terms.

5) " Effect of Thresholding on results"
Thank you for the explanation and new experimental results.




------------------------- BEFORE rebuttal
The paper proposed different metrics for comparing explainers based on their correctness (ability to find most relevant features in an input, used in prediction), consistency (ability to capture the relevant components while input is transformed), and the confidence of the generated explanations. To evaluate correctness, the authors proposed to study the change in the classification accuracy of the target model, under a perturbed dataset where the most relevant regions (as given by explainer) of the image is preserved and the remaining content is replaced with non-informative backgrounds for the target class. For consistency evaluation, the authors proposed to apply transformations like rotation, translation and flip that doesn’t semantically change the input image. For confidence evaluation, they compared the prediction performance on the original image, masked image (only salient regions) and inverted masked image (only non-salient regions). 

Major
•	The paper lack technical novelty.
•	The confidence component looks redundant and can be incorporated in the correctness component.
•	The inverse saliency map idea is already proposed in “Evaluating the visualization of what a deep neural network has learned” for evaluating saliency maps. There the authors gradually replace the most salient regions with random noise and observe a decrease in prediction accuracy.
•	Most of the saliency maps producing methods, generate continuous maps. For making, we need to convert the continuous map to binary by using a threshold. An analysis of choosing different values as threshold is missing. By choosing an appropriate threshold, the size of the most salient region can be controlled. Thus, although Grad-Cam spread saliency over large area, we can use a higher threshold to define the binary mask.
•	Grad-CAM, integrated grad and smooth grad are all gradient-based saliency maps. There are perturbation-based saliency maps, which aims to find most salient regions such that removing those regions produce a maximum drop in prediction accuracy. Example “Interpretable Explanations of Black Boxes by Meaningful Perturbation.”, “Object detectors emerge in deep scene cnns” .  An evaluation of such methods is missing.

Minor:
•	The text in the figures has very small font size and is not readable.
•	

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: See post-rebuttal updates below!

Summary
---

(motivation)
There are lots of heat map/saliency/visual explanation approaches that try to deep image classifiers more interpretable.
It's hard to tell which ones are good, so we need better ways of evaluating explanations.
This paper proposes 3 such explanation evaluation metrics, correctness, consistency, and confidence.

(approach - correctness)
An explanation is correct if it highlights enough of an image for a classifier to tell the correct class with only the highlight parts of the image.
The default way to evaluate on only highlighted portions is to set the non-highlighted bckground to black/grey.
Instead, this method finds images with the same ground truth class which the classifier scored the lowest of all such images, forming a low-confidence baseline.
It copies the background from one of these images instead of using a black/grey background
to try and put the masked image back into the distribution of images from the ground truth class.
This style of masking is used to compute correctness.

(approach - consistency)
An explanation is consistent if it is invariance w.r.t. a number of mostly semantically invariant transformations.
These include small affine transformations, horizontal flips, vertical flips, and adding noise.

(approach - confidence)
An explanation is confident if the masked images it produces still have high condidence under the classifier.
Masked images are produced as for correctness, by copying a distractor from the same class into the background.

(experiments)
The experiments compare existing explanations (LIME, Grad-CAM, Integrated Gradients, SmoothGrad) using the proposed metrics.
1. Correctness: Classifiers have higher accuracy on explanation-masked images than on images they were least confident on (the ones used to fill in the background).
2. Grad-CAM is most correct, followed by SmoothGrad, Integrated Gradients, and LIME.
3. Consistency: Grad-CAM explanations are most resilient to the proposed transformations with Integrated Gradients, SmoothGrad, and LIME being successively less invariant.
4. Confidence: Explanation-masked images have higher scores for their ground truth class than the low-confidence baseline images.
5. Hyperparameter variations in the correcness/confidence metrics mostly preserve the ranking of methods, though the absolute values of performance do change substantially.

(conclusion)
The paper concludes that Grad-CAM is usually the best of the methods tested according to the new metrics and that LIME is the worst.


Strengths
---

I really like the related work section. It could be a valuable resource going forward.

I like the research direction of this paper very much. I think that enumerating a suite of complementary benchmarks is a good way to measure explanation quality because we can only come up with benchmarks that capture a small part of what we want so far.


Weaknesses
---


I see some major conceptual flaws with these metrics:

* In section 3.1 it seems like the first reasons that normal masking failed is not solved by the proposed approach. The generated images are still out of distribution because the "foreground" and the "background" don't match.

* I'm concerned about the low-confidence distractor images used in the background. They are from the same ground truth class as the high confidence images they are pasted into the background of, correct? The correctness metric is supposed to capture whether or not an explanation highlights all the class-relevant content in an image and no more. However, information that the explanation did not highlight (the background) can inform the classifier of the ground truth class because the background came from an image of that class (even if a low confidence one). This is especially true because the relevant objects might be in differrent positions in the two images. Thus it could be that the explanation did not highlight informative content but the classifier still gets the corresponding masked image correct because of the background. How often does this happen?

* Consistency is supposed to measure "the ability of the explainer to capture the relevant components" under semantically invariant transformations.
The reported metric is mimized when the explanation is the same before and after a variety of transformations.
If this were the case then at least one of them must be wrong in the sense that it would not have captured some relevant components
(unless perhaps it just highlighted everything and was thus useless).
Because of the transformation (e.g. 15 degree rotation) the relevant components would have been at a different position, but the best explanation according
to this metric would have been at the same position. Thus this metric seems to reward explanations for not capturing relevant components.


Parts I Didn't Understand:

* In section 3.1, I don't understand the second reason that masking failed. In what sense is masking made meaningless? How is that sense different from the out of distribution concern from the first point?


Missing Details / Presentation Weaknesses:

* Missing reference to [1] which provides more metrics.

* The meaning of confidence is different than it normally is and this may be confusing.
Neural networks should be well calibrated, not necessarily confident (in the commonly used sense of [3]).


Minor flaws:

* Masking by replacing the background with grey (i.e., the bias of the first conv layer) rather than black is more common (e.g., [2] and Grad-CAM). A grey background negates the bias. It's not clear that the background should cancel the bias, but it would be nice to compare to both grey and black masking in Table 7.


[1]: Adebayo, Julius et al. “Sanity Checks for Saliency Maps.” NeurIPS (2018).
[2]: Zeiler, Matthew D. and Rob Fergus. “Visualizing and Understanding Convolutional Networks.” ECCV (2013).
[3]: Guo, Chuan et al. “On Calibration of Modern Neural Networks.” ICML (2017).


Final Evaluation
---

This paper relies solely on theoretical arguments to show its metrics capture meaningful information. Empirically, it only shows that the proposed metrics can differentiate between some popular explanations. It does not empirically show that the differentiation is meaningful (e.g., by measuring agreement with human judgement). This by itself isn't a problem. However, above I detailed significant flaws in the theoretical justification for the metrics, so I can't recommend these metrics (this paper) on either a theoretical or an empirical basis.

Quality: Per above, I do not think the arguments/evidence in the paper support its conclusions.
Clarity: The paper could be clearer, but can be understood without too much effort.
Originality: These metrics are new enough, being novel variations on prior approaches.
Significance: If I was convinced the metrics made sense then I would guess this paper would be very impactful. As is, I don't think it will have much impact.

The quality of the paper is my reason for the low rating. I'm interested to see whether what others think to make sure I've understood the paper correctly and analyzed it accurately. If my understanding is incorrect I could definitely raise my rating.

Post-Rebuttal Evaluation
---
After reading the other reviews and the author responses and taking a brief look at the updated paper I still think this paper should be rejected.

The authors' response to my comments clarified my understanding of the consistency metric. Now I understand it and think it is a useful metric.

However, I did not find clarification about the confidence or correctness metrics, though I agree they are not redundant. They still don't really quite make sense to me. This puts me in about the same position as R3, who also doubts those metrics. In the end, this leaves my initial evaluation essentially unchanged. I still recommend rejection because the paper relies on a theoretical understanding of what makes confidence and correctness metrics useful and that understanding is not provided.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper studies the interesting question of comparing the deep network visualization algorithms quantitatively. Several metrics are proposed, including correctness, consistency and confidence. 

I like the notion of consistency, where an explainer should produce the same explanation under transformations of the image that does not change its “semantic content”. 

However, I am confused or unconvinced by several arguments made in the paper, and if the authors can clarify them I am willing to increase my review. I think the major issue is that most metrics are justified with flimsy arguments, not compared with prior work, and do not lead to consistent ranking of the models. 

Correctness: I am not convinced by the correctness evaluation for several reasons

1. The combined image is still out of distribution, and it is unclear why this is better compared to e.g. using a white background. 
2. Does it favor visualization methods with a blob-shaped saliency map vs. scattered dots shaped saliency map? Does it favor methods with a larger salient region? For example, just from visual appeal, I do not think smoothgrad is worse than gradCAM, but the number says otherwise. I think the arbitrariness of this metric makes the numbers hard to believe. 
3. If the original image is already incorrectly classified (since they are the ones where the classifier assigns the lowest probability) it is hard to imagine that adding random background can make the performance worse.

Therefore, it is also unclear what to make of the numbers in e.g. Table 2. There are so many metrics, precision, recall, F1, and none of them seem particularly well justified. They also do not rank the model in the same way. Which result should a practitioner believe? 

Confidence: I am not sure the confidence vs. number of pixels comparison are useful. Across all methods, it seems to be more pixels -> increased confidence, which is unsurprising. I think the results are only useful if one method pareto dominates another, which is not what is observed in the experiments. 

I do not understand the difference between confidence and correctness. It seems like both measure how well a model can predict the correct class given only the salient region. For example, if method A has higher confidence and lower correctness compared to method B, what does that mean? Under which situation should one choose method A over method B? 
review_title: Official Blind Review #3

Meta-review: The paper proposes metrics for comparing explainability metrics.

Both reviewers and authors have engaged in a thorough discussion of the paper and feedback. The reviewers, although appreciating aspects of the paper, all see major issues with the paper. 

All reviewers recommend reject.  
Decision: Reject