Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper presents a new saliency map interpretability method for the task of image classification. It considers the saliency map as a random variable and computes the posterior distribution over it. The likelihood measures the predictions of the classifier for an image and its perturbed counterpart. The prior encodes positive correlation among adjacent pixels. Variational approximation is used to approximate the posterior.  

————————————————————————————————————————————————————————————————————————————————


Strengths:

S1) The paper is very well written and easy to understand.

S2) The paper does a good job of combining the ideas of perturbation based saliency map methods and total variation regularization with variational approximation in proposing their interpretability approach.

S3) The proposed approach is able to generate real-time saliency maps, and, therefore, is computationally cheap.

S4) The paper shows that their proposed interpretability method passes the sanity check from [Adebayo et al. (2018)].

————————————————————————————————————————————————————————————————————————————————


Weaknesses:

W1) My biggest concern with the paper is that the proposed approach entails training another (non-interpretable) network to explain a given pretrained classifier. One of the goals of interpretability is to ensure that the pretrained classifier doesn’t encode any undesired biases learnt from the data. How can we ensure that this newly trained network doesn’t encode any biases of its own, especially when this network is also trained on the same data?


W2) Another key concern is the evaluation of the proposed approach. Details below:

W2a) I am not sure if I understand the importance of qualitative examples correctly. The paper claims that VarSal is able to produce high quality object borderlines as compared to other methods. Is that a desirable property we should expect from interpretability methods? Ideally, we want the saliency maps to highlight regions which the classifier considered the most important. It is not clear to me how we can evaluate that from qualitative examples. 

W2b) Why is the case where pixels with the largest k% saliency values are erased more prone to creating unnecessary artifacts than the case where pixels with the smallest k% saliency values are erased? Especially, when compared to other methods, is there a reason to believe that these artifacts might be more common in some methods than others?

W2c) Why is Real-Time [Dabkowski & Gal (2017)] omitted from the pixel perturbation benchmark (Figure 5a)? Given that it is computationally equally cheaper to the proposed approach, comparison with Real-Time seems to be the most important one. 

W2d) The proposed approach has many similarities to [Chen et al. (2018)]. How does it compare to the proposed approach / what are the advantages of the proposed approach over [Chen et al. (2018)]?

W2e) The paper in its current form doesn’t provide an explicit take-away message. Given a number of saliency map based interpretability methods available at this point, why should a researcher choose the proposed approach over other works? I think it would be a good idea for the authors to discuss the advantages as well as the disadvantages explicitly, highlighting applications/tasks/conditions where their approach might be better suited than others as well as cases where it might be better to avoid the proposed approach.


W3) A very clear difference between the proposed approach and previous works is that the proposed approach explains the predicted probability distribution while the previous works focus on the ground-truth target. The difference is clearly an advantage of the proposed approach. However, it is important to disassociate the contribution of this difference from the rest of the approach. Is it possible for the authors to create an ablation of their approach keeping this aspect same as the previous works and observe how that version compares (to previous works as well as the proposed approach) qualitatively and quantitatively? 

W4) Saliency maps highlight the parts of the input image that the classifier finds most important while making a prediction. In that sense, isn’t it more reasonable that for a given classifier and an image, the saliency map is deterministic? What does it mean intuitively for this saliency map to have randomness? It would be great if the authors can discuss why it makes more sense to treat it as a random variable instead of a deterministic quantity.

W5) Can the authors please describe the construction of the graphical model in Figure 1? Shouldn’t there be a solid line from ‘M’ to ’s’?

W6) The paper says that [Fong & Vedaldi (2017)], [Fong et al. (2019)] and [Dabkowski & Gal (2017)] -- “all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce a binary mask”. What is the limitation here? And, how is the proposed method overcoming that limitation?

W7) The take-away message from the uncertainity over explanation subsection in unclear. What is the significance of the uncertainity saliency maps? And, what should one learn by generating these maps for the corrupted datasets? 

W8) How is the hyperparameter alpha in the soft-TV Gaussian prior selected? 


——————————————————————————————————————————————————————————————
——————————————————————————————————————————————————————————————

Update after rebuttal: I thank the authors for their responses to all my questions. They satisfactorily answer some of my concerns. However, I still have two major concerns: 1) the faithfulness of the proposed approach, and 2) I see the potential contribution of uncertainty saliency maps but without an application/evaluation, their significance is unclear. I disagree that uncertainty/confidence generated from the same mechanism that generated the explanation is more trustworthy than the explanation itself. Hence, I cannot recommend the paper for acceptance.
review_title: Interesting idea but concerns with methodology and evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: Overview:
This paper proposes a new interpretability method for image classification networks. It considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map.  The likelihood function and the prior distribution are then designed to make the posterior distribution over the saliency map explain the behavior of the classifier’s prediction. Quantitative evaluation on the perturbation benchmark as well as qualitative result show the effectiveness of the proposed method over baselines. 

Pro:
+ The proposed method is very efficient at the inference stage after the training with extra a encoder.
+ From results on the perturbation benchmark, the proposed method achieves better or comparable performance compared with baseline methods.

Concerns:
- The evaluation is a kind of weak. Since it is hard to evaluate the interpretability method on one single benchmark, why not following existing explanation works e.g. Schulz et al. (2020) to conduct evaluations on multiple tasks with different metrics and network backbone. These tasks could be object localization, image degradation (Schulz et al. 2020), etc.
- For the current evaluation on perturbation benchmark, the proposed method is comparable and sometime worse than some recent perturbation-based methods such as SmoothGrad-squared, FIDO. Although the proposed method is quite efficient at the inference stage, it requires extra training steps and extra encoder with parameters. It is a kind of trade-off compared with other  perturbation-based methods. 
- The selected visualization examples have quite simple background. Is it possible to show example with complicated scenes such as samples from PASCAL VOC, MSCOCO, Visual Genome datasets.
- It is not clear whether the proposed method can be generalized to other kinds of data rather than images, such as text data, video data, vision-language tasks etc.

review_title: A interpretability method with variational approximation for image classification networks

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: This paper proposed a method for generating saliency maps for image classifiers that are stochastic (instead of deterministic). The probabilistic model assumes a saliency map random variable that generates the data with a classifier. The inference is done by variational methods. The paper presents several qualitative examples and a comparison to previous work using the pixel perturbation benchmark.

Strengths:
- Saliency map generation is an important problem of interest to the ICLR community.
- Stochastic saliency map is interesting and novel to my knowledge.
- The paper is generally easy to understand.

Concerns:
- Current empirical experiments compare to several saliency generation methods in the pixel perturbation benchmark. The authors should also compare to FullGrad and use the remove-and-retrain framework as in (Srinivas & Fleuret, 2019). This will help us fully understand the performance of this method.
- A positive point is the improved inference time. This is shown in Figure 5b but not discussed in the text, only briefly mentioned in the conclusion. I also think that improved inference time is at the cost of longer training time (the authors can correct me on this).
- A central claim is that stochastic saliency maps improve explanation. The authors have provided some intuition and examples, but I think more evidence is needed.
- The presentation of the paper can be improved: the abstract and the introduction should highlight the contributions and summarize the main results, which the experiment section should fully discuss; the sanity check and implementation details can be cut or moved to supplementary.
review_title: Interesting idea, not fully evaluated

Meta-review: Overall the reviewers had various positive things to say about the paper, including that it was well written and easy to understand, topical, that the method was sensible, novel and interesting and that the computational efficiency (i.e. real time) was appealing.  However, all the reviewers thought it wasn't quite ready for acceptance, mainly citing concerns with the empirical evaluation.  It seems they had trouble interpreting the empirical results and placing the work with respect to other relevant methods.  

It seems in the author response, the authors did much to add to the experiments, but ultimately the reviewers were not comfortable with acceptance.  Taking the reviewers' feedback into account and adding the desired empirical evaluation would make this a much stronger submission to a subsequent conference.
Decision: Reject