Designing Counterfactual Generators using Deep Model Inversion
Authors: Jayaraman J. Thiagarajan
Vivek Narayanaswamy
Deepta Rajan
Jia Liang
Akshay Chaudhari
Andreas Spanias
Venue: NeurIPS 2021 Poster (2021)
Keywords: explainable AI
deep inversion
counterfactuals
image priors
implicit neural representations

Abstract: Explanation techniques that synthesize small, interpretable changes to a given image while producing desired changes in the model prediction have become popular for introspecting black-box models. Commonly referred to as counterfactuals, the synthesized explanations are required to contain discernible changes (for easy interpretability) while also being realistic (consistency to the data manifold). In this paper, we focus on the case where we have access only to the trained deep classifier and not the actual training data. While the problem of inverting deep models to synthesize images from the training distribution has been explored, our goal is to develop a deep inversion approach to generate counterfactual explanations for a given query image. Despite their effectiveness in conditional image synthesis, we show that existing deep inversion methods are insufficient for producing meaningful counterfactuals. We propose DISC (Deep Inversion for Synthesizing Counterfactuals) that improves upon deep inversion by utilizing (a) stronger image priors, (b) incorporating a novel manifold consistency objective and (c) adopting a progressive optimization strategy. We find that, in addition to producing visually meaningful explanations, the counterfactuals from DISC are effective at learning classifier decision boundaries and are robust to unknown test-time corruptions.

Designing Counterfactual Generators using Deep
Model Inversion
Jayaraman Thiagarajan Lawrence Livermore National Laboratory jjayaram@llnl. gov
Vivek Narayanaswamy Arizona State University vnaray290asu. edu
Deepta Rajan
IBM Research Al r. wwdeepttgmmil.comm com
Jason Liang
Akshay Chaudhari
Andreas Spanias Arizona State University spanias$asu edu
Stanford University j sialiiag@@taaffrd.cco
Stanford University
akshayacdstannfrrd.coo
Abstract
Explanation techniques that synthesize small, interpretable changes to given im- age while producing desired changes in the model prediction have become popular for introspecting black-box models. Commonly referred to as counterfactuals, the synthesized explanations required to contain discernible changes (for easy interpretability) while also being realistic (consistency to the data manifold), this paper, focus on the where we have access only to the trained deep classifier and not the actual training data. While the problem of inverting deep models to synthesize images from the training distribution has been explored, our goal is to develop a deep inversion approach to generate counterfactual explanations for given query image. Despite their effectiveness in conditional image synthe- sis, we show that existing deep inversion methods are insufficient for producing meaningful counterfactuals. We propose DISC (Deep Inversion for Synthesizing Counterfactuals) that improves upon deep inversion by utilizing (a) stronger image priors, (b) incorporating novel manifold consistency objective and (c) adopting a progressive optimization strategy. We find that, in addition to producing visually meaningful explanations, the counterfactuals from DISC are effective learning classifier decision boundaries and are robust to unknown test-time corruptions.
Introduction
With the growing need for deploying deep black-box models into critical decision-making, there an increased emphasis on explainability methods that can reveal intricate relationships between data signatures (e.g., image features) and predictions. In this context, the so-called counterfactual (CF) explanations [1] that synthesize small, interpretable changes to given image while producing desired changes in model predictions to support user-specified hypotheses (e.g., progressive change in predictions) have become popular. Though counterfactual explanations provide more flexibility over conventional techniques, such as feature importance estimation 2 4 5 6) by exploring the vicinity of a query image, an important requirement to produce meaningful counterfactuals is to produce discernible local perturbations (for easy interpretability) while being realistic (close to the underlying data manifold). Consequently, existing approaches rely extensively on pre- pre-trained generative models synthesize plausible counterfactuals to scenarios where one cannot access the training data or pre-trained generative models, for example, due to privacy requirements commonly encountered many practical applications.
[
7
10 By design, this ultimately restricts their utility
In this paper, we focus on the problem where we have access only to trained deep classifiers and not the actual training data or generative models. Synthesizing images from the underlying data distribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

by inverting deep model, while not requiring access to training data, is well investigated topic of research. For e.g, Deep Dream [T] synthesizes class-conditioned images by manipulating a noisy image directly in the space of pixels (or more formally Image Space Optimization (ISO)) constrained by image priors such as total variation [2] to regularize this ill-posed inversion. However, Deep Dream is known to produce images that look unrealistic, often very different from the training images, thus limiting their use in practice. Consequently, Yin et al., proposed DeepInversion [33 that performs image synthesis in the latent space of pre-trained classifier (Latent Space Optimization (LSO)) and leverages layer-specific statistics (from batchnorm consistent with the training data distribution. This was showed to produce higher-quality images, particularly in the context of performing knowledge distillation T5 using the synthesized images.
[+] to constrain the images to be
Proposed Work. In contrast, this work aims to develop a deep model inversion approach that generates counterfactual explanations by exploring the vicinity of given query image, instead of synthesizing an arbitrary realization from the entire image distribution. As illustrated in the example in Figure [ existing deep in- version methods are ineffective when natively adopted for counterfactual generation. Due to use of weak priors, and the severely ill-posed nature of the problem, it introduces irrelevant pixel manipulations that easily satisfy the de- sired change in prediction. Hence, we propose DISC (Deep Inversion for Synthesizing Counter- factuals) that improves upon conventional deep model inversion by utilizing: (i) stronger image priors through the use of deep image priors [6]
Manilald
Impur
not smiling smilling
We propose DISC, a deep model inver- sion approach for query-based CF generation. Us- ing a strong image prior (INR in this example) and our manifold consistency constraint, along with progressive optimization strategy, DISC introduces discernibee yet semantically meaningful changes (rightmost) to the query image.
Figure 1:
(DIP) and implicit neural representations I7
(INR); (ii) novel manifold consistency objective that ensures the counterfactual remains close the underlying manifold; and (iii) a progressive optimization strategy to effectively introduce disceriile,, yet meaningful, changes to the query image.
From Figure T. find that our approach produces meaningful image manipulations, in order to change the prediction to the smiling class, while other deep inversion strategies cannot. Using empirical studies, we show that DISC consistently produces visually meaningful explanations, and that the counterfactuals from DISC are effective at learning model decision boundaries and are robust unknown test-time corruptions.
Our Contributions.
1. A general framework to produce counterfactuals on-the-fly using deep model inversion;
2. Novel objectives to ensure consistency to the data manifold. We explore two different strate- gies based on direct error prediction [8 19 and deterministic uncertainty estimation (20];
3. A progressive optimization strategy to introduce discernible changes to given query image, while satisfying the manifold consistency requirement;
4. A classifier discrepancy metric to evaluate the quality of counterfactuals;
5. Empirical studies using natural image and medical image classifiers to demonstrate the effectiveness of DISC over variety of baselines and ablations.
2 Related Work
Image Synthesis from Classifiers. Inverting pre- pre-trained deep model is a popular strategy for image synthesis scenarios where there is no access to the underlying training data. While deep model inversion- based methods such as Deep Dream [I]] and Deeplnversion [13] have been successful in generating candicoondi images, there have been other extensions to such approaches. For example, Dosovitskiy el al. (2I) proposed to invert representations of pre-trained CNN to obtain insights about what a deep classifier has learned. On similar lines, Mahendran et al. [2 22 addressed the problem of pre-image recovery, which in essence attempts recover an arbitrarily

---

encoded representation (in the latent space of a classifier) to realization on the (unknown) image manifold and to enable model diagnosis. Ulyanov et al. [T6] improved upon this iol-posed inversion by utilizing strong image prior in the form of deep image priors (DIP). They also explored the related problem of ietivaaton-maximizaan activation of a given output neuron in pre-trained classifier, and demonstrated the effectiveness of DIP. Despite the effectiveness of the deep model inversion methods for image synthesis, we find that such methods cannot be natively adopted for CF generation and are insufficient for producing meaningful pixel manipulations to a given query image.
[16], where the goal is to generate an image that maximizes the
Counterfactual Generation. Existing methods extensively rely on generative models to provide counterfactuals that explain the decisions of black-box model. Examples including CounteR. GAN 133 Counterfactual Generative Networks (CGNs) (24] and the methods reported in 77] g] have clearly demonstrated the use of generative models in synthesizing high-quality CFs for any user-specified hypothesis on the predictions. However, this requirement of access to training data or generative models can be infeasible in practical scenarios, for e.g., restrictions arising due to privacy requirements. In contrast, our approach formulates the problem of counterfactual generation using deep model inversion, and can produce meaningful counterfactuals using only the trained classifier.
3 Proposed Approach
In this section, we describe our approach for counterfactual generation that improves upon deep model inversion, and introduce new classifier discrepancy metric for evaluating CFs. There four key components that are critical to designing classifier-based counterfactual generators: (i) choice of metric for semantics preservation; (ii) choice of image priors regularize image synthesis; (iii) manifold consistency to ensure that the synthesized counterfactual lies close to the true data manifold; and (iv) progressive optimization strategy introduce gradual meaningful changes to a query image. Figures provide overview of DISC implemented with deep image prior and INR- INR-based for regularizing the counterfactual optimization process.
Eaoke
Donoie
Veaiti
Aweihe
Condens
Cankires
(a) Deep Image Prior
(b) Implicit Neural Representations
Figure 2: Overview of the proposed approach. For given query, DISC trains image generator, which can be implemented using deep image prior (DIP) or coordinate- neural representations (INR), based on three key objectives: (i) semantics preservation; (ii) manifold consistency; and (iii) function consistency.
In its simplest form, for given query x, a counterfactual explanation can be obtained as follows: arg min d(x,, x) s.t. F(x) = y, M(x).
where x C(x) is counterfactual explanation for x, F is pre-trained classifier model, M denotes the data manifold and y = y + is the desired change in the prediction. The metric d(., measures the discrepancy between the query image and the counterfactual (i.e., semantics preservation).
3.1 Choice of Metric for Semantics Preservation
We measure the discrepancy between query and its CF explanation, d(x, x), in the pixel space or in the latent space of the classifier. We now describe the high-level formulation of deep model bnversion-based CF generation.
a. Image Space Optimization (ISO). ISO for counterfactual generation involves the ill-posed optimization of an input x directly in the pixel space to generate an image semantically similar to the query x, while being consistent with a user-hypothesis on the prediction, y. Strategies such as Deep Dream [II perform ISO to synthesize artistic variations of images. Mathematically.
arg min d(x, x) + R(x) s.t. F(x) y.
(2)
3

---

where R(x) is suitable image prior to regularize the optimization.
b. Latent Space Optimization (LSO). LSO refers to the ill-posed problem of inverting an ar- bitrarily encoded representation from the latent space of deep classifier to realization on the (unknown) image manifold. Let V,(.) denote the 1th differentiable layer of the deep classifier. Then, counterfactual generation using LSO can be mathematically formulated as
arg min Σ d( (x), Vx(x)) + R()) S.t. =
(3)
Approaches such DeepInversion 13 perform LSO for conditional image synthesis (though with distribution-level comparison instead of our sample-level comparison) and achieve visually superior images when compared to ISO approaches.
3.2 Choice of Image Priors
As observed from and (3. the choice of the regularizer or image prior R(.) is central towards regularizing and tractably solve this challenging inverse problem. A variety of image priors have been proposed in the literature, and we investigate the following in this work.
Total Variation E2. Total variation [22] (TV) is popular regularizer that encourages images to contain piece-wise constant patches while the &g norm regularizes the range and energy of the image to remain within given interval. The TV norm and the &g norm are given by:
Rx = Σ EV (8igt1- xi,)² -xiii
(Ri+1. Ri,) (x)
(4)
b. Deep Image Priors (DIP). A Deep Image Prior (DIP) [6] leverages the structure of an untrained, carefully tailored convolutional neural network (e.g.. U-Net (25) to generate images and solve variety of ill-posed restoration tasks in computer vision. DIP has been found to produce high-quality reconstructions, based on the key insight that the structure of the network itself can act regularizer. Consequently, the synthesized image is re- re-parameterized terms of the weights @ of the prior model fo i.e x fz(z).
c. Implicit Neural Representations (INR). We also considered an alternative approach based on INR, which provide cheap and convenient way to learn continuous mapping from the image coordinates to the pixel values (RGB). While they have been found to be effective for image/volume rendering and designing generative models (26), we explore their use in deep model inversion (details in appendix). We build upon two key results to design our ANR-based counterfactual generators:
(i) Fourier mapping:
Based on NTK (neural tangent kernel) theory, [27] showed that using Fourier mapping can recover high-frequency features in oiw-dimesiional coordinate-basee image recon- struction. Hence, we use Fourier feature mapping to featurize 2-D input coordinates v [0, 1] before passing them through a coordinate-bail based MLP:
=(v) = (a1 cos(2rb] v), an (2xb2 v), v)....).
Using set of randomly chosen sinusoids, this maps the input points to the surface of high- dimensional hypersphere. Training the MLP network on these embedded points corresponds to kernel regression with the stationary composed NTK /NNK h2. where liNk denotes the neural tangent kernel corresponding to the MLP:
(ii) SIREN activation: In [17]. Sitzmann et al. showed that periodic activation functions are better suited for recovering natural images and their derivatives, when compared to standard activation functions. More specifically, SIREN uses sinusoid activation (xx) sin( Wx + b). We find that using both Fourier mapping coupled with SIREN activation leads to a very strong image prior.
3.3 Manifold Consistency
A key constraint in ①1) that is not included in the formulations in 2 is the manifold consistency, and interestingly, this is not inherently satisfied in deep model inversion. Consequently, even with strong image prior, it can synthesize images that do not belong to the original data distribution. This

---

can be particularly challenging when producing CF's that represent change in class labels, wherein one expects patterns specific target class be emphasized. To address this challenge, we extend the formulation in ( (and equivalently (2) to include a manifold consistency constraint, that is defined directly based on the classifier, without assuming access to class-specific statisties in the latent space. More specifically.:
arg migmin À
Σ [d(Vi()), Vx(x)) (x; (F
(5)
The first term for semantics preservation is same that of (3) and is used ensure that the inherent semantics of the query image is retained in the generated counterfactual (implemented as the & error). The second term Li (manifold consistency) penalizes solutions that do not lie close the data manifold and is designed by assuming access only to the classifier FF. The final loss term f (functional consistency) ensures that the prediction for the counterfactual matches the desired target e.g., categorical cross entropy. As illustrated in Figure 3 a central role in deep inversion-basee CF generation. Even with a strong image prior (DIP this example) and LSO, the generator produces di-triibbiitnno (OOD) CFs (with missing pixels) as guided by the semantic preservation term. Though the synthesized CF produces the desired class label with the classifier F. the explanation is not interpretable. In contrast, including the mm term (using DEP explained next) leads to a meaningful counterfactual that automatically fills in the missing pixels. Note that, this example is different from prior- image inpainting [66,, where a known mask is used to alter the loss function to recover the missing pixels. In this work, explore two different strategies to implement Lie:
the manifold consistency objective plays
without manifoldd
with manifold consistency
Input
consistency
Direct Error Prediction (DEP). Recently, in [19] was found that loss predictor trained jointly with the classifier can be used to effectively detect dis- tribution shifts and obtain accurate uncertainty esti- mates for given sample. Assuming that, a classi- fier model F is trained to optimize the primary loss construct auxiliary loss predictor G to estimate the loss s = (xx) = ppii Similar to [88 19,
Figure 3:
Need for manifold consistency. Without explicitly constraining the CF's to lie close to the true manifold, deep inversion- based generators can produce OOD images (missing pixels) that satisfy functional con- sistency. contrast, our approach able to create more faithful explanation by automat- ically filling in missing pixels.
Epri LcE (F yx)).y),
utilize auxiliary loss func- tion Laur (s, to train the parameters of G. In par- ticular, adopt the contrastive loss which aims preserve the ordering of samples based on their cor- responding losses from F. Let s; and Sy denote the losses of samples x, and X₁, while the corresponding estimates from & are 8; and 8, respectively. Now,
Louz Σ max (0, (..-((s,,))).66. - B,) +7). (4,5)
(6)
1,
if 85z
where I1(8;,8;)
otherwise.
Here is optional margin parameter (7 = 1 for our implementation). The overall objective for the joint optimization of the classifier and loss predictor is given by
Liotal 31 Cori B2Laux-
After the models are trained, we implement the manifold consistency term Lme directly based on the loss predictor, since the losses indicate regimes where the model fails make an accurate prediction:
S.ce
[[](()) s" ||
(8)
where s* denotes the target loss to be achieved. In experiments, set s" the median error estimate from DEP on a held-out validation set.
b. Deterministic Uncertainty Quantification (DUQ). The recently proposed DUQ 20 method is based on Radial Basis Function (RBF) 28 networks and has been showed to be highly effective at OOD detection. In its formulation, model is comprised of deep feature extractor F, exponential
5

---

kernel function along with a set of prototypical feature vectors (or centroids) for each class, DUQ is trained by optimizing the kernel distance between the features from F and the class-specific centroids and using moving average process to update the centroids. Once DUQ is trained, the uncertainty can be measured as the distance between the model output and the closest centroid. Details regarding DUQ training can be found in the appendix. In this work, we implement mm using margin-basel loss that maximizes the kernel similarity of the synthesized CF with the centroid of the target class, relative to the source class. Denoting the kernel similarity for CF with the centroid for class y as K KFF(x, (())) where o(y) corresponds the pre-computed centroid for class y, we define:
Sm max K K(F())) - (F(x), (()) + +7,0)
(9)
which indicates that kernel similarity w.r.t. the target class y should be greater than that with the source class at least by the margin (set to 0.5 in our experiments).
3.4 Progressive Optimization
CF generation is a highly under-constrained problem, that even with a strong image prior and the proposed manifold consistency constraint, it can easily converge to trivial solutions, i.e., irrelevant image manipulations. For example, one might expect to introduce large discernible changes by reducing the penalty Y for semantics preservation. However, given the large solution space (defined by the number of parameters in the DIP/INR generator fe), this often leads to unrealistic images. To circumvent this, we propose to adopt a progressive optimization strategy that gradually increases the number of layers in fo to be optimized and steadily relaxing the penalty A1 (by factor k) to allow for larger, yet interpretable, changes. More specifically, denoting the number of layers in fo by L, in each iteration we train the parameters of the first layers (i is incremented by 1 in the subsequent iteration) while keeping the parameters of the remaining L. - i layers at their initial state (details on how the layers are chosen for DIP and INR based generators can be found in the appendix). similar strategy has been shown to be effective for ill-posed restoration tasks using large-scale generative models such as Style-GAN [29] 30]. An outline of this progressive optimization process is provided in the appendix. We find that such progressive optimization leads to significantly better quality solutions allowing meaningful traversal from one class to another.
3.5 Evaluating Quality of CF Explanations using Classifier Discrepancy
A desired property in query-based explainers is that the synthesized changes are both interpretable and representative of the target class (e.g., smiling). To systematically evaluate the latter property, we propose the following synthetic experiment: Given a binary classifier F and training images, i.e., Xo- belonging to Class 0, we use our CF generator to synthesize examples for Class i.e., X₂ (using Class 0 images as input). and finally build secondary classifier F using [Xo, X1). The quality of the counterfactuals can thus be measured using the gap between the performance of and J on a common test set. We refer to this score as classifier discrepancy (CD).
4 Experiment Setup
Datasets. (i) CelebA Faces B1:
This dataset contains 202,599 images along with a wide-range of attributes. For our experiments, we consider 3 different attributes, namely smiling, bald and young. Note, we train a classifier for predicting each of the attributes independently. We report the results for bald and young attributes in the appendix; (ii) ISIC 2018 Skin Lesion Dataset :3]: : This
lesion diagnosis challenge dataset contains a total of 10, 015 dermoscopic lesion images from the HAM10000 database [33]. Each image is associated with one out of 7 discase states: Melanoma (MEL), Melanocytic nevus (MN), Basal cell carcinoma (BCC), Actinic keratosis (AK), Benign keratosis (BK), Dermatofibroma (DF) and Vascular lesion (VASC), Note, in all cases, we used stratified 90- 10 data split to train the classifiers.
Classifier Design. For all experiments, we resized the images to size 96 x 96 and used the standard ResNet-18 18 architecture 3d to train the classifier model with the Adam optimizer (35 batch size 1 128,
learning rate le 4 and momentum 0.9. For the DEP implementation (Section 3.3 we performed
average pooling on feature maps from each of the residual blocks in ResNet-18, and applied linear
layer of 128 units with ReLU activation. The haper-parameters in (7) were set at 81 1.0 and

---

Input
+
ISO INR
LSO L2
LSO DIP
LSO INR
Figure 4: ISO LSO with different choices of priors. Though none of the image priors inherently lead to discernible changes that reflect the properties of the target smiling class, we find that LSS with strong priors produces higher quality images compared to ISO.
82 = 0.5. For the case of DUQ, we set both the length scale parameter and the gradient penalty to 0.5.
Image Generator Design. For generator design, the deep image prior used the standard U-Net architecture and input noise images drawn from the uniform distribution 41|-1, 1]. For INR, chose 256 random sinusoids with frequencies b, drawn from a Gaussian distribution with mean and variance 100 to compute the Fourier mapping for the input coordinates.
Findings
LSO Offers Better Feature Manipulation Control over ISO with Different Priors. An important
design component in DISC is how we compute the semantic discrepancy between query x and CF either in the pixel space using ISO or in the latent space of the classifier using LSO. We perform comparative analysis of their behavior in CF generation using CelebA faces, particular when manipulating an image from the non-smiling class to be classified as smiling, by varying the choice of image priors. As observed in Figure 4 though LSO produces higher quality images compared to ISO when using a weak image prior (TV + E2), that quality gap vanishes with the use of a stronger prior, e.g., DIP. However, in terms of producing discernible changes that reflect the properties of the smiling class, neither approach is sufficient. In particular, ISO shows higher risk of making minimal, irrelevant modifications (refer to difference images Xx in Figure 4 that drive the prediction to desired label and hence, similar to 13 we recommend the use of TSSO but with stronger image priors to design CF generators.
Manifold Consistency is Essential to Produce Meaningful Explanations. As showed in the previ- experiment, deep model inversion does not produce discernible (and interpretable) image changes when applied for CF generation. In this context, we explore the impact of enforcing manifold consis- tency in DISC. In particular, we compare the following LSO-based CF generation implementations (with INR prior): (i) manifold consistency: (ii) DUQ-based Lme from (9); and (ii) DEP- DEP-based Lyne from (8. From the results Figure we find that the proposed DEP objective significantly improves over the standard LSO (with no Zmc) by introducing appropriate pixel manipulations near the mouth and cheeks in all examples and clearly represents the underlying semanties of the smiling class. In contrast, the RBF network-based DUQ performs very similar to the LSO + INR bascline and
7

---

Input
LSO + INR
LSO INR DUQ
LSD INR DEP
Input
LSO INR
LSO INR DUQ
LSO INR DEP
Figure 5: Importance of manifold consistency. The proposed DEP objective significantly improves over the standard LSO (with no m by introducing appropriate pixel manipulations near the mouth and cheeks in all examples. In contrast, we find that DUQ-based consistency is insufficient to emphasize the semantics the smiling class as seen in the difference images ([x -0xxx) X).
Table 1: Evaluating the quality of the synthesized CFs on CelebA faces and ISIC2018 skin lesion datasets. The MSE and concentration metrics for the CelebA dataset were obtained using CFs synthesized for 5000 images from the non-smiling class. On the other hand, for ISIC2018, we used 800 images from the MEL class and generated CF' for changing the prediction to NEV.
Method
Dataset
Metric
+SO
LSO DIP
LSO INR
LSO DIP DEP
ISO INR DEP
MSE
0.36 + 0.18
0.09 = 0.05
0.11 +0.07
0.19 10.11
0.22 +0.07
CelebA
Cone.
0.43 +0.22
0.29 10.16
0.28 + 0.19
0.26 +0.15
0.18 + 0.11
CD
0.31 + 0.05
0.23 10.03
0.26 10006
0.15 1 0.04
0.08 1 0.03
MSE
0.43 + 0.17
0.14 0.08
0.19 +0.09
0.23 #0.14
0.24 +00.33
Cone.
0.36 10.16
0.32 10.14
0.29 +: 0.13
0.25 10.09
0.22 +: 0.07
ISIC
CD
0.32 10.13
0.28 10.15
0.25 10.12
0.17 + 0.05
0.11 = 0.01
this emphasizes the inability of the kernel similarity metric to detect mild distribution shifts in data, though it has been proven successful for detecting severely OOD samples.
INR based Priors Produce Highly Concentrated Changes over DIP. Although both DIP and INR effective, a more rigorous comparison of those two image priors is required. For this purpose, we utilize two important evaluation metrics, namely: (i) ability to produce large discernible changes, measured using error between and in the pixel-space; and (ii) ability produce concentrated image manipulations [36], measured by thresholding (< 0.05) the difference image |x x and determining the area of the largest bounding box that contains all the non-zero values (between and 1). Ideally, explanations that have a larger MSE and consistently lower concentration are more likely to produce easily interpretable, localized changes. The results Table are obtained by randomly choosing 5000 images from the non-smiling class and synthesizing the corresponding counterfactuals for smiling. Similarly, for the ISIC2018 dataset, used 800 randomly chosen images from the MEL class and generated CFs for NEV. The naïve LSO TV + Eg baseline produces counterfactuals with significantly large MSE as well as concentration score, indicating that the CFs are uninterpretable and contain irrelevant perturbations all over the image. As expected, incorporating a stronger prior improves the concentration significantly, while also being highly conservative terms of MSE, i.e., non-discernible changes. Finally, enforcing manifold consistency using DEP achieves an optimal trade-off between the two metrics and produces meaningful CFs (Figures and 7). In particular, INR based generators produce highly concentrated image manipulations.

---

Input
I50 INR DEP
USO DIP DEP
Ippu
USO + INR
150 DIP DEP
Input
USO + -R-
USO DIP DIP-DEP
Figure 6: Comparison between DIP and INR with DEP manifold consistency. Although both DIP and INR are effective for LSO-based model inversion, we find that INR based generators produce highly concentrated and more apparent image manipulations.
GT: BKL
BCC
GT:NV
CP: VASC
GT:
OF: BKL
GGT MASC
GT BCG
Figure 7: Observations CF synthesis for examples from ISIC 2018 dataset. We find that, in a multi-class problem, our approach able to synthesize concentrated image changes, thus enabling to introspect deep models with arbitrarily complex decision boundaries. Moreover, such perturbations are consistent with the ABCD (asymmetry, border, color and diameter) signatures adopted by clinicians for diagnosing lesions.
LSO + INR + DEP Produces CFs with Low Classifier Diserepancy Score. We now evaluate the quality the counterfactuals using the proposed classifier discrepancy (CD) score. For this purpose. we consider random subset of 5000 images each from non-smiling (Xa) and smiling (X₂) classes respectively. Following the strategy described in Section 3.5 we train the classifiers F and F and measure the CD score as the difference in test accuracies, Acc.(F, (F., Xteest AF Xtear). Similarly, we repeat a simple evaluation for ISIC data by using 800 images from class MEL as Xu and images from class NEV as Xi. From Table we find that, using manifold consistency along with strong image prior produces significantly lower CD scores (0.08 with LSO + INR DEP on CelebA), when compared with LSO without manifold consistency (0.26 with ISO INR). In particular, using INR + DEP fairs the best and consistently produces highly meaningful counterfactuals.
DISC Explanations are Robust Under Test-Time Distribution Shifts. In several practical ap- plications, we often encounter shifts between the train and test distributions which makes model deployment challenging. Hence, we evaluate the behavior of the proposed approach under unknown distribution shifts test time. Note, we train the classifier on the clean CelebA faces dataset without introducing any corruptions. However, when we introduce corruptions at test-time, we find that (see Figure B). our approach (LSO INR + DEP) is able to robustly manipulate the appropriate regions in the query image. This can be attributed both to the ability of DEP to reflect challenging distribution shifts with higher error estimates [19] and our progressive optimization strategy to induce larger yet semantically meaningful changes (i.c., inherent noise clean up).

---

rpu
155 I&R DPP 5.50 OP DFP
150 INR + DPP 150 OP DFP
lepe
IS0 INC DEP
150 + TIP DEP
Brighnees
(Severity
Brightnees
(Soverity =
Mation Blur (Severity
Mation Blur (Severity
Grussian Noisa (Severlyy
(Severity
Figure DISC explanations are robust under test-time corruptions. We find that even under unknown test-time corruptions, our approach robustly manipulates the appropriate regions in the query image (e.g., mouth and cheeks for smiling). Such behaviour can be attributed both to the ability of DEP to reflect challenging distribution shifts [19 and our progressive optimization.
6 Conclusions
In this paper, presented DISC, general approach to design counterfactual generators for any deep classifier, without requiring access to the training data or generative models. We draw connections to the problem of deep model inversion and extend it to support counterfactual generation. DISC can learn a CF generator on-the-fly by leveraging different image priors and manifold consistency constraints, along with a progressive optimization strategy, synthesize highly-plausible explanations. Future extensions to this work include exploring the use of multiple target attributes simultancously in our optimization and applying this method timevaryying data.
7 Acknowledgements
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DEFAAC52-07NA273344.
References
[1] S. Verma, J. Dickerson, and K. Hines. Counterfactual explanations for machine learning: A review. arXiv preprint arXiv:2010.1052 2020.
[2] RR. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-basee localization. In Proceedings of the IEEE international conference computer vision, pages 61 618-226, 2017.
[3] H. Lakkaraju, N. Arsov, and Bastani. Robust black box explanations under distribution shift. International Conference Machine Learning 2020.
10

---

[4] A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3145-3153 PMLR, 2017.
[5] MT Ribeiro, S. Singh, and C. Guestrin. Why should trust you?" explaining the predictions of any classifier. In Proc. of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144, 2016.
[6] MT. Ribeiro, S. Singh, and C. Guestrin. Anchors: sesh-precssion model- -agnostc explanations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[7] AV. Looveren and J. Klaise. Interpretable counterfactual explanations guided by prototypes. In Joint European Conference on Machine Leurning and Knowledge Discovery in Dutabases, pages 650-665. Springer, 2021.
[8] A. Dhurandhar, P. Chen, R. Luss, C. Tu, P. Ting, K. Shanmugam, and P. Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, volume 31, 2018.
[9] S. Sumedha, P. Brian, C. Junxiang, and B. Kayhan. Explanation by progressive exaggeration. In International Conference on Learning Representations, 2020.
[10] Y. Goyal, Z. Wu, J. Ersst, D. Batra, D. Parikh, and S. Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pages
PMLR, 2019.
[11] A. Mordvintsev, C. Olah, and M. Tyka. Inceptionism: Going deeper into neural networks, 2015. [Last accessed 27-May-20211,
[12] A. Mahendran and A. Vedaldi. Visualizing deep convolutional neural networks using natural pre-images, International Journal of Computer Vision, 120(3):233-255, 2016.
[13] H. Yin, P. Molchanov, JM. Alvarez, Z. Li, A. Mallya, D. Hoiem, NK. Jha, and J. Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
[14) S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[15] H. Geoffrey, Oriol V., and Jeffrey D. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
[16] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9446-9454, 2018.
[17] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. Advances Neural Information Processing Systems, 33, 2020.
the [18] D. Yoo and L.S Kweon. Learning loss for active learning. Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 93-102, 2019.
[19] JJ. Thiagarajan, V. Narayanaswamy, R. Anirudh, PT. Bremer, and A. Spanias. Accurate and robust feature importance estimation under distribution shifts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7891-7898, 2021.
[20] J. Van Amersfoort, L. Smith, YW. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference Machine Learning, pages 9690-9700. PMLR, 2020.
[21] A. Dosovitskiy and T. Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEFE conference on computer vision and puttern recognition, pages 4829 4837, 2016.
11

---

[22] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
5 5188 188-,196, 2015.
[23] D. Nemirovsky, N. Thiebaut, Y. Xu, and A. Gupta. Countergan: Generating realistic counterfac- tuals with residual generative adversarial nets. arXiv preprint arXiv:2009.05/99 2020.
[24] A. Sauer and A. Geiger. Counterfactual generative networks. ICLR, 2021.
[25] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.
[26] Y. Chen, S. Liu, and X. Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8628 88888,66,,21 2021.
[27] M. Tancik, PP. Srinivasan, B. Mildenhall, S. Fridovich-Keil. N. Raghavan, U. Singhal, R. Ra- mamoorthi, JT. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
[28] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 36(11):2278-2324, 1998.
[29] G. Daras, J. Dean, A. Jalal, and AG. Dimakis. Intermediate layer optimization for inverse problems using deep generative models. arXiv preprint arXiv:2102.07364 2021.
[30] Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110-8119, 2020.
[31] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
[32] N. Codella, V. Rotemberg, P. Tschandl, M.E. Celebi, S. Dusza, D. Gutman, B. Helba, A. Kalloo, K. Liopyris, M. Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03368, 2019.
[33] P. Tschandl, C. Rosendahl, and H. Kittler. The ham 10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific data, 5:180161, 2018.
[34] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 2016.
[35] DP. Kingma and J. Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, San Diego California, 2015.
[36] C. Chang, E. Creager, A. Goldenberg, and D. Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations, 2018.
12


Reviews:
[{"review_id": "i-9p5_33iAK", "reviewer": "Reviewer_JzXC", "summary": "Summary: The paper proposed a novel approach to achieve deep model inversion for a pre-trained deep classifier using the classifier alone, without having access to the training dataset or generative models. Given a query image, the proposed deep model inversion generates a counterfactual explanation that is realistic looking and produces the desired classification outcome.   \nEarlier work on image synthesis by inverting deep classifiers either creates unrealistic images or creates perturbation of query image that successfully changes the classification decision but resulting pixel-level manipulations are not interpretable/meaningful (similar to an adversarial attack).  \nThe proposed approach has four essential components:\n1.\tCounterfactual image is synthesized using an image prior. The authors experimented with an un-trained UNET-based model as a deep image prior (DIP) and Fourier mapping with SIREN activation as implicit neural representation (INR) for mapping noise to the counterfactual image.\n2.\tTo ensure high similarity between counterfactual and query images, the authors used ISO to minimize discrepancies over the entire image and LSO to minimize differences between embeddings extracted from different layers of the deep classifier.\n3.\tTo ensure that the generated counterfactual image is realistic and lies on the data manifold, the proposed model uses a manifold consistency (MC) loss. The MC-loss is quantified using an auxiliary loss predictor function in DEP or the radial basis kernel similarity in DUQ. Both methods provide ways to capture epistemic uncertainty due to out-of-distribution (OOD) samples.\n4.\tThe model uses a functional consistency loss to ensure the counterfactual image produces a desired outcome from the deep classifier.\nFinally, the model uses progressive optimization to learn the image prior layer-by-layer. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# After rebuttal\n\nthe authors have performed extended experiments to compare against existing work, that has substantially improved their manuscript. So I would like to change my rating from 6 to 7\n\n----------------------------\nMajor comments\n•\tIn section 3.2, \"Choice of Image Priors: Total variation + l2\": From equation 4, the initialization of the counterfactual image is not clear.  \n•\t In section 3.2, \"Choice of Image Priors: DIP and INR,\" the definition of z is not clear. The authors should consider moving text from supplementary material to the main manuscript to clarify how the counterfactual image was synthesized using DIP and INR.   \n•\t In Eq. 5, the mc-loss term is dependent on the counterfactual image and the deep classifier. In DUQ, the mc-term considers the embedding of the centroid for the target and source class and hence depends on y. Eq. 5 should be updated to show this dependence. \n•\tIn Eq. 8, the definition of s* is not explicit. \n•\t In section \"Findings: LSO vs. ISO,\": only qualitative results in Fig.3 are provided. To compare the quality of the images generated across different models, the authors should consider reporting results on metrics such as FID score. \n•\tIn section \"Findings: Manifold consistency is essential,\": only qualitative results in Fig.4 are provided.  \n•\tIn Table 1: the best scores are not highlighted to show the best performing model. \n•\tFig.4 and Table 1 emphasize identifying the model with the most localized and meaningful counterfactual changes. But no results are reported on whether the changes are important/relevant for classification. The authors should consider comparing against post-hoc feature importance-based methods such as Grad-CAM and quantify the intersection between salient regions obtained from different methods. The authors should also consider reporting log odds metric, which is a measure of change in prediction when k% of the most relevant features in the input data are masked. \n•\tIn Fig. 6, the authors assumed the reader know about ABCD signatures for understanding skin-lesion images. An example to describe how perturbations are consistent with this signature will help readers understand the results. \n•\tIn Fig.7, usually, the uncertainty of the classifier increases with shifts in data distribution. It would be interesting to see how certain the classifier is in its prediction for the counterfactual images. Does the proposed method successfully create a counterfactual image with a certain prediction for a query image with high uncertainty? \n•\tTo improve the readability of the figures in the result section, authors should consider reporting f(x) before and after counterfactual changes. \n•\tSometimes, a classifier may not use visual clues meaningful to humans to make prediction decisions, i.e., a classifier may rely on spurious features or shortcuts to make prediction decisions. An application of the current method on discovering such shortcuts will improve the quality of the manuscript. \n•\tUpdate the references to arxiv versions to point to actual publications.   \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no significant social concern", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z0v-aPLUS7O", "reviewer": "Reviewer_Txfe", "summary": "The paper proposes a deep inversion approach that uses a pre-trained classifier to generate the counterfactual explanations of a given image. In other words, the method introduces small, discernible perturbations in an image that changes its class while keeping it realistic. The method makes use of certain inductive biases such as strong image priors, manifold consistency objective, and progressive optimization strategy to generate those counterfactual explanations.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Contribution: The method differs from other counterfactual generating methods which use generative models. This method relies on inverting a discriminative model (a pre-trained classifier) to generate counterfactuals which I believe is novel and pretty interesting. Controlling this inversion (manipulating the synthesis part) is difficult and the proposed method provides ways to control it via certain inductive biases.\n\nMain comments/ questions:\n- The paper provides ablation for the design choices which is great. However, I am missing the comparisons with the known counterfactual methods such as generative models based methods.\n\n- The results are verified on simpler datasets where the difference in classes requires very small manipulations/ perturbations. Also, the number of classes in these datasets is very limited for e.g. there are only three classes in CelebA. How would the method deal with more complicated datasets such as image net? How do the perturbations look there as the data manifold is more complicated? I would be interested in seeing results on at least a subset of image-net or similar datasets.\n\nClarity:  The paper is fairly well-written and easy to follow. However, the motivation of the approach is somewhat lacking. For e.g. the authors say that one cannot use previously proposed counterfactual generative models because of privacy constraints. It would be great if they can provide some references. Moreover, I also found the notations to be a bit confusing. I will expand on this below.\n\n- In eq 1, where does $\\mathcal{C}$ come from? Is it a subset of $\\mathcal{M}$?\n\n- In line 141, what’s z?\n\n- I didn’t understand the role of contrastive objective in preserving the ordering of the samples in DEP (sec. 3.3(a)). And where is this contrastive objective used?\n\n- In the experimental setup, for designing a classifier, why is the gradient penalty and length scale parameter set to 0.5?\n\nI generally liked the idea, but I believe that the paper can be improved further with more results. For now, I am inclined towards borderline rejection. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None that I know of.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "Ryzs1Acc1y7", "reviewer": "Reviewer_hrJB", "summary": "In this paper authors propose to generate counterfactuals from a query image and a given trained deep classifier. One is interested to find out a semantically reasonable alternative explanation, where image would be classified from class y to y'. This is all performed without the help of an explicit generator (such as GAN). ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "If we think about adversarial examples, then yes changing the input with a minimal adversarial noise, we can obtain classification result to any target class. But such an \"counterfactual\" would not be of real use for practitioners as the noise is hand-crafted to mess  up the classification. Semantic and manifold constraints are needed to generate an image that has meaning. \n\nEven though authors say that they do not use generator models, but in deep image priors, decoder model is included and I am thinking that constraining the search using the generator model is a pretty crucial feature. Similar work was done in activation maximization paper (that authors cite), where latent code of autoencoder is manipulated and the decoder generates the \"counterfactual\". \n\nI would like to see  a bit more discussion in this paper about the meaning and usefulness of these kind of generated examples. Is the goal to generate good smiling images from non smiling ones, or is it to debug the deep model, where darkened eyes can lead to classifier to decide that image is old vs actually being young. Defining the final goal exactly can help then to conduct experiments that will ultimately find out whether the goal was reached or not. For example if the goal is to debug then, then it is quite important to know how the classifier was fooled by the new image. Some numerical measurement is then essential to show. \n\nIn the example CFs it would be interesting to see the strength of activation of the target class. Such as does semantically meaningful generation result in an image that actually is not strongly classified as target. Even better would be to quantify the uncertainty of classification, such as using BNN type schemes. \n\nUltimately, I see that authors have dug into deep learning toolbox to build the present system. What I liked about it is that it gives the idea that all of these options were tried out and we have thus arrived at an conclusion. But at the same time each particular block was known in advance, so technical novelty is in the combination. \n\nOther comments: \nM(x) in Eq. (1) is just one point? Maybe input to M(.) needs to be the whole training set? Or maybe the meaning is a local data manifold, centered on query point x? \n\nWhat is R(x) in  Eq. (2) ? Obviously it needs to be a scalar valued function.  A more mathematical definition at this point is needed. \n\n3.2.b should be explained with precision. Now it is hard to grasp how DIP would work, especially compared to a and b. \n\nIn 3.2.c.i what is b_1 ? In the next equation b is obviously scalar, but in this one it has to be 2-d vector. \n\nWhat is s^* in (8)\n\nIn Section 3.5, classifier discrepancy measure should possibly also take into account the stochastic nature of training. This  would mean that multiple instances of  F^c are estimated from the same training set. Then the final CD measure is an interval instead of a point estimate. \n\nIn all Figures, the meaning of bluish sub-Figures is not explained. I am guessing that it is a pixel level difference image. \n\nFigure 3, I personally do not see any of the images as smiling. To me this tells that this kind of empirical study should be reinforced with perceptual experiments. \n\nWhy L_mc is termed manifold consistency when you work on the score level only? I would imagine manifold consistency meaning that we take latent space of the images as make sure that we do not drift far from that latent image manifold. How do you reconcile this concept with the one used in your work?\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "I60n8ax53G", "reviewer": "Reviewer_za8w", "summary": "**(motivation)** Many approaches exist to generate counterfactual images, but what should be done when no training data is available? Current approaches lead to adversarial-looking approaches, which are not satisfying enough as an explanation.\n\n**(approach)** The authors use Deep Inversion and a novel objective that ensures the data stays in distribution and propose a method to evaluate the quality of counterfactuals.\n\n**(experiments)** The authors run experiments on Celebi and ISIC.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The work is well-motivated and thoroughly describes the problems that counterfactual generation faces.\n- The method is evaluated on two image datasets.\n- The approach is novel.\n- The work proposes a new metric for evaluating the counterfactuals, which might be less susceptible to bias.\n\n## **Weaknesses**\n\n- Since the authors propose a new metric, it makes sense to compare/discuss why the proposed approach is more appropriate than currently existing methods. However, a related work section about metrics for counterfactual evaluation is missing. It would strengthen the paper to present a comparison between the proposed methods and that of Dabkwoski & Gal (https://arxiv.org/abs/1705.07857), which is a close metric to what the authors propose. \n- The authors do not evaluate against a baseline. While it is fine to exclude many methods for their reliance on training data, there do exist some methods without this limitation, e.g., Occlusion or Goyal et al. (https://arxiv.org/abs/1904.07451 ). This would make the evaluation more sound and strengthen it, as the existing methods probably perform worse.\n- I can see only minor changes for some of the images, especially for the baldness attribute, if the person in the image has a lot of hair. The method does not seem to add wrinkles or similar structures that come with age. Generally, minor changes of attributes need more discussion in the paper. Why does this happen?\n- It would be good to know if any images are showing only adversarial examples and, if so, how many. \n- The checklist is not filled out correctly. Code is missing.\n\n## **Missing Details / Points of Confusion**\n\n- Line 141: explicitly introduce z as noise\n\n## **Presentation Weaknesses**\n\n- Figure descriptions (e.g., 3) are not detailed enough: How are the heatmaps computed?\n- There is no bold in tables for best values, and an explanation of what optimal values of MSE and concentration would be is missing in the description of Table 1\n\n## **Suggestions**\n\n- Experiment with baldness and age for both bald <-> hairy and young <-> old\n- Explain why the proposed measure for evaluation is better and compare it with another metric.\n\n## **Preliminary Evaluation**\n\n\n\n**Clarity:** the paper is clear in its motivation and explanation of the method and experiments.\n\n**Originality:** the problem and approach of creating counterfactuals without access to the training data is novel.\n\n**Significance:** the problem is significant, especially if training data is not available. \n\n**Quality:** the paper is well-written but the formatting of the tables and figures could be improved. Furthermore, the missing comparison with other methods does affect the quality of the paper.\n\n\n\n**tl;dr:** This paper proposes an interesting approach worth publishing, but the experimental part should be improved by adding comparisons with another metric and a baseline method. It also could benefit from a more in-depth discussion of drawbacks (what happens when bias is involved, the danger of still finding adversarial changes only).\n\n# After Rebuttal\n\nI thank the reviewer for their extensive answer. They address my points well and I raised my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews highlight that this a technically sound, well-written contribution to the problem of counterfactual explanations in the novel and relevant setting where only the classifier is accessible. The manuscript is therefore accepted. The impact of the paper could be further strengthened if it was put in context of other ongoing work of counterfactual explanations, eg [1].

[1] Karimi et al. A survey of algorithmic recourse: contrastive explanations and consequential recommendations
Decision: Accept (Poster)