Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a deep inversion approach that uses a pre-trained classifier to generate the counterfactual explanations of a given image. In other words, the method introduces small, discernible perturbations in an image that changes its class while keeping it realistic. The method makes use of certain inductive biases such as strong image priors, manifold consistency objective, and progressive optimization strategy to generate those counterfactual explanations.
main_review: Contribution: The method differs from other counterfactual generating methods which use generative models. This method relies on inverting a discriminative model (a pre-trained classifier) to generate counterfactuals which I believe is novel and pretty interesting. Controlling this inversion (manipulating the synthesis part) is difficult and the proposed method provides ways to control it via certain inductive biases.

Main comments/ questions:
- The paper provides ablation for the design choices which is great. However, I am missing the comparisons with the known counterfactual methods such as generative models based methods.

- The results are verified on simpler datasets where the difference in classes requires very small manipulations/ perturbations. Also, the number of classes in these datasets is very limited for e.g. there are only three classes in CelebA. How would the method deal with more complicated datasets such as image net? How do the perturbations look there as the data manifold is more complicated? I would be interested in seeing results on at least a subset of image-net or similar datasets.

Clarity:  The paper is fairly well-written and easy to follow. However, the motivation of the approach is somewhat lacking. For e.g. the authors say that one cannot use previously proposed counterfactual generative models because of privacy constraints. It would be great if they can provide some references. Moreover, I also found the notations to be a bit confusing. I will expand on this below.

- In eq 1, where does $\mathcal{C}$ come from? Is it a subset of $\mathcal{M}$?

- In line 141, what’s z?

- I didn’t understand the role of contrastive objective in preserving the ordering of the samples in DEP (sec. 3.3(a)). And where is this contrastive objective used?

- In the experimental setup, for designing a classifier, why is the gradient penalty and length scale parameter set to 0.5?

I generally liked the idea, but I believe that the paper can be improved further with more results. For now, I am inclined towards borderline rejection. 

limitations_and_societal_impact: None that I know of.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary: The paper proposed a novel approach to achieve deep model inversion for a pre-trained deep classifier using the classifier alone, without having access to the training dataset or generative models. Given a query image, the proposed deep model inversion generates a counterfactual explanation that is realistic looking and produces the desired classification outcome.   
Earlier work on image synthesis by inverting deep classifiers either creates unrealistic images or creates perturbation of query image that successfully changes the classification decision but resulting pixel-level manipulations are not interpretable/meaningful (similar to an adversarial attack).  
The proposed approach has four essential components:
1.	Counterfactual image is synthesized using an image prior. The authors experimented with an un-trained UNET-based model as a deep image prior (DIP) and Fourier mapping with SIREN activation as implicit neural representation (INR) for mapping noise to the counterfactual image.
2.	To ensure high similarity between counterfactual and query images, the authors used ISO to minimize discrepancies over the entire image and LSO to minimize differences between embeddings extracted from different layers of the deep classifier.
3.	To ensure that the generated counterfactual image is realistic and lies on the data manifold, the proposed model uses a manifold consistency (MC) loss. The MC-loss is quantified using an auxiliary loss predictor function in DEP or the radial basis kernel similarity in DUQ. Both methods provide ways to capture epistemic uncertainty due to out-of-distribution (OOD) samples.
4.	The model uses a functional consistency loss to ensure the counterfactual image produces a desired outcome from the deep classifier.
Finally, the model uses progressive optimization to learn the image prior layer-by-layer. 

main_review: # After rebuttal

the authors have performed extended experiments to compare against existing work, that has substantially improved their manuscript. So I would like to change my rating from 6 to 7

----------------------------
Major comments
•	In section 3.2, "Choice of Image Priors: Total variation + l2": From equation 4, the initialization of the counterfactual image is not clear.  
•	 In section 3.2, "Choice of Image Priors: DIP and INR," the definition of z is not clear. The authors should consider moving text from supplementary material to the main manuscript to clarify how the counterfactual image was synthesized using DIP and INR.   
•	 In Eq. 5, the mc-loss term is dependent on the counterfactual image and the deep classifier. In DUQ, the mc-term considers the embedding of the centroid for the target and source class and hence depends on y. Eq. 5 should be updated to show this dependence. 
•	In Eq. 8, the definition of s* is not explicit. 
•	 In section "Findings: LSO vs. ISO,": only qualitative results in Fig.3 are provided. To compare the quality of the images generated across different models, the authors should consider reporting results on metrics such as FID score. 
•	In section "Findings: Manifold consistency is essential,": only qualitative results in Fig.4 are provided.  
•	In Table 1: the best scores are not highlighted to show the best performing model. 
•	Fig.4 and Table 1 emphasize identifying the model with the most localized and meaningful counterfactual changes. But no results are reported on whether the changes are important/relevant for classification. The authors should consider comparing against post-hoc feature importance-based methods such as Grad-CAM and quantify the intersection between salient regions obtained from different methods. The authors should also consider reporting log odds metric, which is a measure of change in prediction when k% of the most relevant features in the input data are masked. 
•	In Fig. 6, the authors assumed the reader know about ABCD signatures for understanding skin-lesion images. An example to describe how perturbations are consistent with this signature will help readers understand the results. 
•	In Fig.7, usually, the uncertainty of the classifier increases with shifts in data distribution. It would be interesting to see how certain the classifier is in its prediction for the counterfactual images. Does the proposed method successfully create a counterfactual image with a certain prediction for a query image with high uncertainty? 
•	To improve the readability of the figures in the result section, authors should consider reporting f(x) before and after counterfactual changes. 
•	Sometimes, a classifier may not use visual clues meaningful to humans to make prediction decisions, i.e., a classifier may rely on spurious features or shortcuts to make prediction decisions. An application of the current method on discovering such shortcuts will improve the quality of the manuscript. 
•	Update the references to arxiv versions to point to actual publications.   



limitations_and_societal_impact: no significant social concern
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper authors propose to generate counterfactuals from a query image and a given trained deep classifier. One is interested to find out a semantically reasonable alternative explanation, where image would be classified from class y to y'. This is all performed without the help of an explicit generator (such as GAN). 
main_review: If we think about adversarial examples, then yes changing the input with a minimal adversarial noise, we can obtain classification result to any target class. But such an "counterfactual" would not be of real use for practitioners as the noise is hand-crafted to mess  up the classification. Semantic and manifold constraints are needed to generate an image that has meaning. 

Even though authors say that they do not use generator models, but in deep image priors, decoder model is included and I am thinking that constraining the search using the generator model is a pretty crucial feature. Similar work was done in activation maximization paper (that authors cite), where latent code of autoencoder is manipulated and the decoder generates the "counterfactual". 

I would like to see  a bit more discussion in this paper about the meaning and usefulness of these kind of generated examples. Is the goal to generate good smiling images from non smiling ones, or is it to debug the deep model, where darkened eyes can lead to classifier to decide that image is old vs actually being young. Defining the final goal exactly can help then to conduct experiments that will ultimately find out whether the goal was reached or not. For example if the goal is to debug then, then it is quite important to know how the classifier was fooled by the new image. Some numerical measurement is then essential to show. 

In the example CFs it would be interesting to see the strength of activation of the target class. Such as does semantically meaningful generation result in an image that actually is not strongly classified as target. Even better would be to quantify the uncertainty of classification, such as using BNN type schemes. 

Ultimately, I see that authors have dug into deep learning toolbox to build the present system. What I liked about it is that it gives the idea that all of these options were tried out and we have thus arrived at an conclusion. But at the same time each particular block was known in advance, so technical novelty is in the combination. 

Other comments: 
M(x) in Eq. (1) is just one point? Maybe input to M(.) needs to be the whole training set? Or maybe the meaning is a local data manifold, centered on query point x? 

What is R(x) in  Eq. (2) ? Obviously it needs to be a scalar valued function.  A more mathematical definition at this point is needed. 

3.2.b should be explained with precision. Now it is hard to grasp how DIP would work, especially compared to a and b. 

In 3.2.c.i what is b_1 ? In the next equation b is obviously scalar, but in this one it has to be 2-d vector. 

What is s^* in (8)

In Section 3.5, classifier discrepancy measure should possibly also take into account the stochastic nature of training. This  would mean that multiple instances of  F^c are estimated from the same training set. Then the final CD measure is an interval instead of a point estimate. 

In all Figures, the meaning of bluish sub-Figures is not explained. I am guessing that it is a pixel level difference image. 

Figure 3, I personally do not see any of the images as smiling. To me this tells that this kind of empirical study should be reinforced with perceptual experiments. 

Why L_mc is termed manifold consistency when you work on the score level only? I would imagine manifold consistency meaning that we take latent space of the images as make sure that we do not drift far from that latent image manifold. How do you reconcile this concept with the one used in your work?


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: **(motivation)** Many approaches exist to generate counterfactual images, but what should be done when no training data is available? Current approaches lead to adversarial-looking approaches, which are not satisfying enough as an explanation.

**(approach)** The authors use Deep Inversion and a novel objective that ensures the data stays in distribution and propose a method to evaluate the quality of counterfactuals.

**(experiments)** The authors run experiments on Celebi and ISIC.


main_review: **Strengths**

- The work is well-motivated and thoroughly describes the problems that counterfactual generation faces.
- The method is evaluated on two image datasets.
- The approach is novel.
- The work proposes a new metric for evaluating the counterfactuals, which might be less susceptible to bias.

## **Weaknesses**

- Since the authors propose a new metric, it makes sense to compare/discuss why the proposed approach is more appropriate than currently existing methods. However, a related work section about metrics for counterfactual evaluation is missing. It would strengthen the paper to present a comparison between the proposed methods and that of Dabkwoski & Gal (https://arxiv.org/abs/1705.07857), which is a close metric to what the authors propose. 
- The authors do not evaluate against a baseline. While it is fine to exclude many methods for their reliance on training data, there do exist some methods without this limitation, e.g., Occlusion or Goyal et al. (https://arxiv.org/abs/1904.07451 ). This would make the evaluation more sound and strengthen it, as the existing methods probably perform worse.
- I can see only minor changes for some of the images, especially for the baldness attribute, if the person in the image has a lot of hair. The method does not seem to add wrinkles or similar structures that come with age. Generally, minor changes of attributes need more discussion in the paper. Why does this happen?
- It would be good to know if any images are showing only adversarial examples and, if so, how many. 
- The checklist is not filled out correctly. Code is missing.

## **Missing Details / Points of Confusion**

- Line 141: explicitly introduce z as noise

## **Presentation Weaknesses**

- Figure descriptions (e.g., 3) are not detailed enough: How are the heatmaps computed?
- There is no bold in tables for best values, and an explanation of what optimal values of MSE and concentration would be is missing in the description of Table 1

## **Suggestions**

- Experiment with baldness and age for both bald <-> hairy and young <-> old
- Explain why the proposed measure for evaluation is better and compare it with another metric.

## **Preliminary Evaluation**



**Clarity:** the paper is clear in its motivation and explanation of the method and experiments.

**Originality:** the problem and approach of creating counterfactuals without access to the training data is novel.

**Significance:** the problem is significant, especially if training data is not available. 

**Quality:** the paper is well-written but the formatting of the tables and figures could be improved. Furthermore, the missing comparison with other methods does affect the quality of the paper.



**tl;dr:** This paper proposes an interesting approach worth publishing, but the experimental part should be improved by adding comparisons with another metric and a baseline method. It also could benefit from a more in-depth discussion of drawbacks (what happens when bias is involved, the danger of still finding adversarial changes only).

# After Rebuttal

I thank the reviewer for their extensive answer. They address my points well and I raised my score to 7.
limitations_and_societal_impact: NA
review_title: null

Meta-review: The reviews highlight that this a technically sound, well-written contribution to the problem of counterfactual explanations in the novel and relevant setting where only the classifier is accessible. The manuscript is therefore accepted. The impact of the paper could be further strengthened if it was put in context of other ongoing work of counterfactual explanations, eg [1].

[1] Karimi et al. A survey of algorithmic recourse: contrastive explanations and consequential recommendations
Decision: Accept (Poster)