Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary\
This paper proposes a new 'baseline' for attribution methods tailored to deep neural networks. DNN attribution methods like integrated gradients, deep lift and others require a baseline to compare to as part of the computation. The choice of a baseline has been controversial in the literature, and a good method to select a baseline remains an open problem. This paper seeks to address that problem. Specifically, this paper seeks to develop a baseline for one-vs-one explanations as opposed to one-vs-all explanations. Consider an MNIST model, a one-vs-one attribution would attribute why an input is say a '2' and not a '4', i.e., it is contrastive against a particular target class and not all classes. This paper proposes to use a StarGAN for generating these baselines. The paper then evaluates explanations derived using the new baseline and shows that they explanations 'perform' better. 

Overall, I think the paper tackles an important problem, but I have several concerns with the motivation, the appropriateness of the baseline definition in this work, and the evaluation. I'll expand on these concerns in the later part of the review, so I am not recommending an accept in its current form. 

Significance/Quality\
The paper tackles an interesting and potentially challenging problem. However, motivation is still somewhat unclear, and there are critical problems with the evaluations used as justification here. I go into these at the end of this review. 

Clarity/Writing\
The paper is generally easy to follow. I problem I had reading it is that there are a few sentences that are stated as fact without any justification. For example, the paper notes, "The minimum distance training sample in section 2.2 is a true class-targeted baseline proposed in the past." What is a 'true' class-targeted baseline? Such statements should probably be reformulated. 

Minor changes
Lasted paragraph of section 2.3: "has posted a challenging", posted is probably not the desired word here. 

Questions and Concerns

- Motivation: it is still not clear to me why a one-vs-one attribution is desirable? More should probably be done here to motivate this. The biggest need though is motivation for the one-vs-one attribution baseline. In several statements, the paper alludes to properties of baselines used in expected gradients and other methods, stating the reasons why these baselines are undesirable. I agree, but why should a one-vs-one baseline be preferable to these? Ideally, the paper will set out a list of desirable properties; then show that the baseline derived from GANMEX satisfies these. It is still not clear to me why a notion of minimum distance in a different target class is the right one. Can the authors say more about why this should be the case? 

- Evaluation: I'll preface my concerns here with the fact that I think evaluating model attributions or explanations in general is a difficult and open problem. This said, I don't think any of the evaluations presented in this paper can be taken as showing that the GANMEX baseline is the desirable one. First, the perturbation-based evaluation does not provide consistent rankings (see: Tomsett et. al. (AAAI Sanity checks for saliency metrics)). I suspect the gini index will have the same problems as those discussed in the Tomsett et. al. paper. The sanity checks themselves, i.e. the cascading randomization, will tell you if a method should be ruled out and not whether a method is effective. Consequently, I don't think the sanity checks can say much in judging baselines. Having said all of this, I think the way to evaluate a baseline is to take a task where the truth ground-truth rankings are known a priori, train a model to respect and align to the true ground truth. Now one can compare attributions from such a model for a normal baseline and a baseline from GANMEX. Assuming the attribution method itself is a reliable one, then one can quantify improvements due to the GANMEX baseline. 

A paper that might be related to this work that also incorporated generative modeling: https://arxiv.org/abs/1807.08024.pdf

Overall, the concerns above make me hesitant about this current draft; however, I am happy to revise my assessment if the authors think I am wrong. 
review_title: Considers an important problem but conclusions not clear

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper looks to use GANs to generate baselines for attribution methods. The focus on one-vs-one feature importance explanations is novel, to the best of my knowledge. The paper attempts to make progress on the baseline selection problem that has plagued the feature importance community. 

Strengths
- As far as I know, the authors' contribution of one vs one attribution (compared to one vs any attribution) is novel. Whilst other works have alluded to this or ran heusristic experiments, this paper does a good job of formalizing the notion.
- The ability for GANMEX to live on top of any other attribution method makes it an attractive addition to existing attribution methods. Thank you for visualizing the baselines generated by GANMEX, quite helpful :)

Weaknesses
-  A computational complexity analysis is required to gauge the practical utility of generating baselines with GANMEX. Also, it would be nice to give complexity of GANMEX compared to FIDO, EG, and simple nearest neighbor baselines. 
- In addition to the visual comparisons provided, it would have been helpful to evaluate explanations using exisiting evaluation criteria in the attribution literature (i.e., faithfulness, sensitivity, monotonicity, etc.) This paper has the opportunity to broadly assess the effects of various baselines on attributions.

Questions
- While GANs seem like an attractive choice of deep generative model (DGM) for this problem, can you comment on or experiment with other DGMs (i.e., VAEs or specifically VAEACs [1])? However, any DGM that has latent class separation should suffice. You would be able to perform optimization in the latent space [2, 3, 4] and achieve similar class separation, as described in Figure 1.
- The attributions in Figure 3.E seem like noise, while zero baseline seems visually appealing -- can you provide some intuition for why this occurs? The GAN feels like overkill for MNIST, but might be suitable for other high dimensional problems wherein the baseline needs to pick up on small nuances in the data.

[1] https://openreview.net/forum?id=SyxtJh0qYm

[2] https://arxiv.org/abs/1806.08867

[3] https://arxiv.org/abs/2006.06848

[4] https://arxiv.org/abs/1807.08024
review_title: Using GANs to get baselines when finding feature attributions

final_score: rating: 5: Marginally below acceptance threshold
review_content: Paper Summary:

This paper considers the less-explored baseline selection issue in attribution methods for one-vs-one explanations of multi-class classifiers. The key insight is to construct the closest and realistic target class baseline. To this end, an existing image-to-image translation GAN model, namely StarGAN, is leveraged to transform an input example to another example in a target class yet is close to the input. This baseline can be integrated with a variety of attribution methods, including integrated gradient, DeepLIFT, Occlusion, and deepSHAP, and shows consistent improvements over zero baseline and minimum distance training sample for one-vs-one explanations. The experiments are conducted on three datasets – MNIST, SVHN, and apple2orange.

Paper Strengths:

This paper addresses an important yet overlooked baseline selection problem. The way the authors address this problem is interesting by leveraging GAN models. Empirical evaluations demonstrate the effectiveness and generalizability of the proposed approach.

Paper Weaknesses:

1) The main weakness of this paper to me is the evaluation section. The proposed approach is only validated on simple datasets like MNIST and SVHN. It would be more convincing to show the effectiveness of the proposed approach on natural images and a large number of classes, like CIFAR and ImageNet, as used in the previous work such as IG.

2) Following the comment in 1), the prior works, such as IG and DeepLIFT, have been used to analyze other types of models and been evaluated on other types of data, such as genomics and neural machine translation. In addition to images, would the proposed approach also apply to these domains?

3) As illustrated in Figure 1, the key assumption of the proposed approach is that a GAN model (StarGAN) is able to generate examples that are much closer to the input examples than the training examples (i.e., minimum distance training sample). Under what conditions would such an assumption hold?

4) Following the comment in 3), it would be interesting to show and analyze some failure cases.

5) In the proposed approach, the StarGAN directly uses the already trained model classifier as its discriminator. How if the StarGAN trains its own discriminator without using the model classifier?

6) It would be interesting to show the hyper-parameter (different trade-off lambdas) sensitivity.

7) I understood that the authors focused on one-vs-one explanations. But I am interested to hear the authors’ thoughts on how to extend the proposed approach to one-vs-all explanations.

After Rebuttal: 

I thank the authors for the rebuttal. I have also read the other reviewers’ comments. Unfortunately, the rebuttal is unconvincing and sometimes vague. I keep my original rating.
review_title: Official Blind Review #4

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper claims to present a novel GAN-based model explainability, for generating one-vs-one explanations, by incorporating to-be-explained classifier as part of the GAN. They use GANs to produce a baseline image which is a realistic instance from a target class that resembles the original instance.

Positive aspects:
- a novel approach for generating one-vs-one explanation baselines leveraging GANs
- the proposed approach improves the saliency maps for binary classifiers 

Negative aspects:
- the paper lacks clarity
- the approach is demonstrated on cherry-picking examples. Have doubts of its generalization capability

Please find below some of my concerns:
1. Your claim: "...we use GANs to produce a baseline image which is a realistic instance from a target class that resembles the original instance". Why do you need a GAN? Why don't you use a network to generate a confusion matrix to analyze the performance of the classifier? And based on this analysis you could explain why, for instance, the digit '0' is classified as a '6'. 
2. Related with the previous point, your analysis is very limited. You assume '0' is classified as '6'. Could '0' be classified as an '8' or '9'? It is not clear from your analysis. There are no comments on these cases. Looks like your examples to defend your approach are cherry-picked.
3. I am not sure how to interpret Figure 2. 
Some other comments:
1. The paper lacks novelty. The authors' contribution is not clear.
2. The experimental validation is limited and not convincing. The authors use just some simple datasets (MNIST, SVHN). What about more complex datasets, like CIFAR10, LSUN, etc.? Could your approach explain the mis-classification in these cases?

review_title: Recommendation to Reject

Meta-review: This work investigates the choice of a 'baseline' for attribution methods. Such a choice is important and can heavily influence the outcome of any analysis that involves attribution methods. The work proposes doing (1) one-vs-one attribution in a sort of contrastive fashion (2) generating baselines using StarGAN.

The reviewers have brought out a number of valid concerns about this work:

1. One-vs-one attribution appears to be novel, and distinctive enough from the more prevalent "one-vs-all" formulations. I am perhaps more optimistic than the reviewers that such a formulation is in fact useful, but I can see where the hesitancy can come from.
2. It's not clear that the evaluation shows that the proposed method is in fact superior to the others. All the reviewers touched upon this one way or another.
3. Somewhat simplistic datasets used for evaluation (noted that there are CIFAR10 results in the rebuttal).

This was more borderline than the scores would indicate. I thank the authors for the extensive replies and extra experiments. I encourage them to incorporate more of the feedback and resubmit to the next suitable conference. I do believe that doing experiments on ImagetNet (like previous work does, such as IG) would be quite worthwhile and convincing. I suspect the computational expense could be mitigated by re-using pretrained networks, of which there are many available for ImageNet specifically.
Decision: Reject