Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focus on generating Visual Counterfactual Explanations (VCE) for any arbitrary classifier using diffusion models. Prior work generates VCE reliably on adversarial robust models or generative models limited to smaller number of classes. This work leverages diffusion models to generate realistic and minimal semantically altered VCE examples. The authors propose to condition the generated image on the target class using the desired classifier and condition on the original image with L1 distance regularization. The parameterization of such formulation allowed to fix the hyperparameters across different images and models i.e. no need to finetune the hyperparameters of diffusion process for different images and models. Inspired by prior works that shown VCEs are reliably generated on adversarially robust models, authors leverage such robust models and align the gradients of non-robust models with robust models using cone projection. Such projection enabled to generate VCEs reliably for arbitrary classifier. Qualitative results show that the proposed method generates VCE better than the existing works.
strengths_and_weaknesses: Strengths:
1)	Paper is well written and motivation of the problem formulation is clearly explained.
2)	Formulation for the class conditioning and distance regularization is clear.
3)	Proposed cone projection of the gradients of non-robust classifier on to the gradients of robust classifier helped to generate semantically meaningful VCEs for non-robust classifiers.
4)	Generated VCEs are semantically meaningful and perceptually looks improved than the baselines.

Weakness:
1)	Generally lacking a quantitative measure to evaluate the generated VCEs. Evaluation is mainly performed with visual inspection.
2)	As the integration of cone projection shown to be helpful, however it is not clear why this particular projection is chosen. Are there other projections that are also helpful? Is there a theoretical proof that this cone projection resolves the noise of the gradients in non-robust classifiers?

Overall, I think the proposed technique yield better VCE and interesting for the community. I also think that the strengths outweigh the weakness. However, I would be open to hear other reviewers opinion here.

questions: Please refer Weakness above.
limitations: Authors addressed the limitations and negative impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper focuses on model-agnostic visual counterfactual explanation (VCE). The proposed method builds on the diffusion model and generates counterfactual samples for the target class with minimal changes. The authors first handle the robust classifier by using adaptive parameterization. And then they solve the non-robust classifier by projecting the gradient of a robust classifier onto the cone of the non-robust classifier. They experiment on ImageNet classifier and demonstrate qualitatively for dozens of examples. 
strengths_and_weaknesses: Strengths:
Although it is not new to use the diffusion model for generating counterfactual samples, the proposed method is new in the sense that it has no constraint on the classifier to be explained. In addition, the proposed method can handle a bigger class set. The qualitative results support their claim overall. 

Weaknesses:
It seems to depend on a robust classifier anyway. As stated in the paper, the computational expense might be high because of the time-related generative process. More importantly, there are no quantitative comparisons between models, which makes it hard to evaluate. Although it is hard to design quantitative metrics for VCE, it is still possible to measure validity and closeness. 
questions: 1. How do you select the project angle (30 degrees)? Is it empirically determined? 
2. Is $\Sigma_{\theta}$ diagonal? If so, does that mean each feature/pixel of the generated sample is independent?
3. With the projection of the gradient for the non-robust classifier, how hard is the training? 
limitations: See the above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a framework to generate visual counterfactual explanations using diffusion processes. In order to generate non-trivial and close to input CFs, the authors modify the diffusion process accordingly: by introducing a distance function in the diffusion process and the choice of the starting point, adaptive re-parameterization, and cone projection of the gradient (using a robust classifier) as regularization. 
strengths_and_weaknesses: Strengths:
- The authors focus on an important topic of generating realistic CF examples. 
- The paper is well-written and easy to follow.
- The authors have made novel contributions by adapting diffusion models into generating CF images.
- The experiments show the effectiveness of the proposed method against baselines.

Weaknesses:
- A key factor in generating realistic CFs using the proposed method is the existence of an adversarially robust model which is not often the case. 
- No quantitative comparison in terms of the realism/validity/closeness of the generated CFs. 


questions: - I suggest the authors discuss the fail cases of the method and show some examples. 
- I suggest the authors add a closely related work [1] in their discussions
- I suggest the authors add Supp B.4 to the main paper.

[1] Khorram and Fuxin. Cycle-Consistent Counterfactuals by Latent Transformations. CVPR 2022. 

limitations: The authors have discussed the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a method to generate visual counterfactual examples (VCEs) using a diffusion model. Given a classifier we would like to explain, the method generates counterfactuals that are realistic and minimal perturbations of a query image and change the query's predicted class. Realism is obtained using a diffusion model as an image prior. The method is similar to other VCEs except using a diffusion model to enforce realism, and the main contributions are the specific techniques used to guide a diffusion model toward generated VCEs.
strengths_and_weaknesses: I found this to be an interesting paper. The method is sensible and the results are decent. I found the evaluation partially convincing, but with some weaknesses. I will elaborate on these dimensions below:

**Originality**
The main contributions of this paper are in the details. The paper takes two fairly well-established things, VCEs and diffusion models, and puts them together. The exact way they are combined seems reasonable, but also not all that different than other work that uses classifiers or score functions of various kinds to guide generative models. At a glance, the results look quite similar to text/classifier-guided image manipulation. Nonetheless, the target application is slightly different than general image manipulation (rather it is VCE generation) and the specific details (adaptive parameterization, cone-projection) are, to my knowledge, novel.

**Quality** 
The method appears to be sound, and the quality of the results is high, at least along the three criteria the paper targets -- validity, realism, minimality. The qualitative comparisons do seem to show the advantages of DVCEs over baselines and ablations, but I'm wary of reading too much into qualitative results. The user study was not terribly convincing -- DVCEs only win on one out of three metrics.

**Clarity**
There are two issues with the writing that could be improved:
1. The math was hard for me to follow. This is in part because the derivations refer to a lot of results from previous works. I would appreciate if the math could be made more self-contained. It also feels like the method is actually pretty simple and could be stated more plainly, e.g., Eqn 10 could be explained as: "take a step in the direction that increases the classifier score while not moving too far away from $\hat{x}$" (this is just an example, my point is just that some of the mathiness could be translated into plainer English, alongside the formal derivations). Equation 6 was especially hard for me to parse and I'm not sure I understand what is intended (although I think I get how $f_{\text{dn}}$ works).
2. In my opinion, the paper somewhat overclaims the contributions. There are three primary ways. First, it is emphasized a few times (e.g., Line 22) that DVCEs are model-agnostic and can be applied to any classifier, and that this distinguishes them from past work. However, results are only shown on ImageNet classifiers, and, as the paper mentions, a prereq is having a diffusion model and robust classifier for the domain. This was not quite what I expected from a method described as "model-agnostic" and that (line 56) "overcomes" the need for retraining for each new classifier (retraining is still needed if you deviate from ImageNet). Second, the success of the user study is overclaimed. Line 68 says DVCEs outperform baselines on the user study, but in fact out of three user study metrics DVCEs were best on just one. The paper argues that, taken holistically, the user study shows that DVCEs are best, but I wouldn't say that's home run evidence. Third, the paper claims (e.g., line 156) that DVCEs "can be applied to any image classifier regardless of whether it is adversarially robust or not." I find this slightly misleading, since a robust classifier is still required. It could be rephrased as something like "can be applied to any image classifier as long as you are given a robust image classifier for that same task."

**Significance**
I believe the paper studies an important topic, and contributes a tool that indeed seems useful. However, in some ways the paper is a step back from other recent work in this area. For example, whereas DVCEs only show one counterfactual, [21] shows multiple counterfactuals, along different visual dimensions. The DVCE results look very similar to text-guided image editing, and indeed this is apparent in the use of an image editing paper, [2], as one of the baselines. This raises the question: what exactly distinguishes the kinds of VCEs presented here from the kind of results in papers like [2] (which did not target counterfactual explanation). The contribution would be clearer if there were metrics or applications of DVCEs that were specific to the goal of _explaining_ classifiers. What do we learn about different classifiers from these visualizations? Can our gain in understanding be measured? These questions are barely addressed by the present paper. From a practical standpoint, the applicability of the method is limited by the requirement of a robust classifier for the task and a good diffusion model for the domain.


questions: My main questions and suggestions are the following:

1. I did not find the evaluation to be completely convincing. I would like to see stronger quantitative evidence that DVCEs are better explanations than the baselines/ablations. This could be quantitative measures of validity/realism/minimality, more convincing user studies, or other metrics that get more at the heart of the "usefulness" of the explanations.

2. I would be interested in seeing Section 4.2 greatly expanded. I think the big thing missing from the paper is a demonstration that we can use DVCEs to actually get insight into different classifiers -- see their biases, understand their weaknesses, pick which of two is better, etc. Section 4.2 barely scratches the surface of these questions, but it is a good start.

limitations: The limitations are adequately discussed.
review_title: null

Meta-review: metareview: On the whole after the reviews and discussion, the reviewers are in agreement that this is an interesting method with some novel contributions and high-quality results, which should be of interest to the community.
confidence: Less certain
Decision: Accept