Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In line with further work on counterfactual explanations, this work proposes to provide multiple counterfactual explanations in terms of VAE style disentangled latent variables. The idea of using VAEs to obtain multiple counterfactuals is novel and quite useful for multiple practical applications. 
main_review: The idea of using VAEs to generate multiple counterfactuals is novel. Authors have evaluated their work on 3 image datasets and a number of useful evaluation metrics and baselines. The results are promising. 

Could you please comment on similarity/differences with this work: 
https://arxiv.org/pdf/1905.12698.pdf


summary_of_the_review: While there is work on diverse counterfactual explanations and different VAEs (DIP-VAE, Beta-VAE, etc), and work algorithmic recourse, this work seems to marry the two ideas of of providing multiple counterfactuals based on disentangles latent variables. This can be quite helpful in practise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a generation-based explainability method, DISSECT, that generates Concept Traversals to provide a counterfactual explanation. The Concept traversals (CTs) are sequences of generated examples with increasing degrees of concepts' influence on the classifier's decision. The CTs are able to provide disentangled explanations for different concepts. 
main_review: Originality: The task is interesting, the proposed DISSECT is based on Explanation by Progressive Exaggeration (EPE) while proposing disentanglement modules. It is good to provide the pipeline comparison between EPE-mod and DISSECT, because they may look similar, which shows limited novelty.

Clarity: Overall is clear, however, some sentences are too long to follow: e.g., 1st paragraph in the Method section. 

Significance and experiments: 

1 Baselines method and evaluation setting are thorough, while it would be better to show ablation study and show the necessity of each component, especially the difference between EPE-mod and DISSECT.

2 I am curious about the generalization to a novel dataset, how to define the meaning of each concept that influences the classification? 

3 For fidelity of the explanation, I wondered about the correctness of the concept. How to verify the correctness of the concepts? How to rank the importance of the concepts for an explanation?

4 For How to guarantee the provided CTs can be easily understood by humans? How to avoid confusing users when more concepts emerge?


summary_of_the_review: I am borderline for this paper in the current stage and I may increase my score if the author can solve the above concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce DISSECT: Disentangled Simultaneous Explanations via Concept Traversal. The method creates counterfactual explanations for which the concept increasingly influences the classifier's decision. Experimentally, the authors show that the proposed method generally performs better than existing methods on several aspects ("importance", "realism", "distinctness", "stability"). 
main_review: **Strengths**:

- novelty: the approach taken in this paper is very different from other counterfactual explanation methods. This is the first paper I've read that creates several explanations which increasingly rely on a concept. I think this aspect is very interesting and could be relevant in practice. For example, as highlighted in the paper, it can be used to depict how a mole turning from benign to malignant may progress. 
- experiments/evaluation: The authors demonstrate the efficacy of their method over three different datasets (each well motivated for the problem at hand) and extensively evaluate the performances. Specifically, they introduce metrics for each of the desiderata  ("importance", "realism", "distinctness", "stability").  The proposed method generally performs better than the benchmarks (VAEs designed for disentanglement, CSVAE, EPE, and modifications thereof). 

**Weaknesses**: 

- several of the evaluation metrics are first introduced in this paper. It would be great to see some more justification on whether the evaluation metrics actually measure what they are trying to target. E.g., for realism, one could simulate data for which we know the "ground truth" and then evaluate the metric.
- to a certain extent, the method is designed to target these metrics. As such, I would expect it to perform better. Having said that, I think this is only a small weakness. Evaluation protocols are not widely agreed upon in the field, and so there isn't a clear protocol to follow. 

**Clarity**: 

Generally, the paper is quite well-written. The architecture is well-motivated (particularly the loss in section 3.1) and the experimental setup (both evaluation metrics and choice of datasets) is well-argued. 

**Questions**:
- did you perform a sensitivity analysis for the loss function? While the motivation for the loss function is extensively described, I am curious whether all terms are necessary. 

**Minor comments** (did not affect the score in any way):
- use the following symbol twice for the left quotation marks in latex: `
- there are some minor typos/grammar/spacing errors. 
summary_of_the_review: Overall, I would recommend an acceptance. The field of interpretability is becoming increasingly more important. The authors introduce a novel method, called DISSECT, which brings a meaningful contribution to the aforementioned field.  The method is novel (bringing a fresh perspective) and performs well empirically. I did not give an 8 as I still had some reservations about some aspects of the paper (as highlighted in the section before).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a generative approach to model explanations, where explanations are composed of automatically generated counterfactual examples that differ by the observed magnitude of a given learned concept. Through a series of extensive experiments, the authors show that the proposed approach, DISSECT, can successfully disentangle useful and realistic concepts.

The main contributions of the paper is the novel approach that generates several counterfactual examples across a concept spectrum, and a new synthetic dermatology dataset that allows testing such methods where they matter most. 

main_review: This is an original paper that introduces a novel approach to model explanations. It builds on several advancements in the literature, and aims at producing a solution that is useful for the users that will actually run it. It is clearly written and easy to follow, and includes an extensive experimental analysis that compares the approach to many strong baselines and on several substantially different dataset.

While I do think that this is overall a good paper, I do have some minor issues that I would like to see clarified. First, I am worried about the dependency on a strong generative model that can successfully disentangle concepts. It could be that this approach would fail on data with structures that do not allow such an easy generation process, such as natural language. To strengthen the findings in the paper, I would like to see some exploration of the relationship between generation quality and explanation quality.

Second, I’m not sure that the fact that this method does not depend on predefined user concepts is a good thing. It can certainly be advantageous if we want to explore hidden biases, but it could also be problematic in cases where we want to test a specific hypothesis regarding a concept we already know and care about. How would an approach like DISSECT deal with cases where we care about some specific concepts, but would also like to discover unknown biases?

Thirdly, I think that the paper might benefit from a deeper discussion of the importance of generating realistic counterfactuals from a causal perspective. The main motivation for producing counterfactuals for explanations is the estimation of causal effects, and connecting DISSECT to this literature can improve the paper in my opinion.

Fourthly, this paper seems fairly based on a specific paper, “Explanation by Progressive Exaggeration”. While the authors are clear about their use of this work, the similarity of the approaches is significant. In fact, the overall objective is identical except for the extra term that enforces disentanglement. Given that, the technical contribution of this paper is a bit thin. 

Lastly, I have trouble understanding the qualitative result in Figure 5. From carefully observing it, I am not convinced that DISSECT does a substantially better job than EPE-mod in disentangling the confounding concepts. While I do find the quantitative evidence convincing, I would consider clarifying or replacing Figure 5.

summary_of_the_review: While I do think that this is overall a good paper, I do have some minor issues that I would like to see clarified. First, I am worried about the dependency on a strong generative model that can successfully disentangle concepts. Second, I’m not sure that the fact that this method does not depend on predefined user concepts is a good thing. Thirdly, I think that the paper might benefit from a deeper discussion of the importance of generating realistic counterfactuals from a causal perspective. Lastly, I think that the technical contribution might be a bit too thin, given prior work that this paper builds on.
review_title: null

Meta-review: This paper proposes a counterfactual explanation method, termed DISSECT, for image classification. While previous work is concerned with generating one single counterfactual, DISSECT aims to produce multiple counterfactuals, with each illustrating one possible way the class label could be altered. Intermediate images between the benign example and the counterfactuals are also generated to show how the decision boundary is crossed.
The reviewers find the idea novel, the presentation clear, and the empirical evaluation thorough.  However, there are concerns regarding whether the method will generalize to other domains because it relies on a strong generative model.  In addition, there is no human-subject study to show whether and how much the method really help an end-user.
Decision: Accept (Poster)