Beyond Trivial Counterfactual Generations with Diverse Valuable Explanations
Authors: Pau Rodriguez
Massimo Caccia
Alexandre Lacoste
Lee Zamparo
Issam H. Laradji
Laurent Charlin
David Vazquez
Venue:  (2021)
Keywords: Interpretability
Counterfactual
Explanations
Black-Box

Abstract: Explainability of machine learning models has gained considerable attention within our research community given the importance of deploying more reliable machine-learning systems. Explanability can also be helpful for model debugging. In computer vision applications, most methods explain models by displaying the regions in the input image that they focus on for their prediction, but it is difficult to improve models based on these explanations since they do not indicate why the model fail. Counterfactual methods, on the other hand, indicate how to perturb the input to change the model prediction, providing details about the model's decision-making. Unfortunately, current counterfactual methods make ambiguous interpretations as they combine multiple biases of the model and the data in a single counterfactual interpretation of the model's decision. Moreover, these methods tend to generate trivial counterfactuals about the model's decision, as they often suggest to exaggerate or remove the presence of the attribute being classified. Trivial counterfactuals are usually not valuable, since the information they provide is often already known to the system's designer. In this work, we propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss to uncover multiple valuable explanations about the model's prediction. Further, we introduce a mechanism to prevent the model from producing trivial explanations. Experiments on CelebA and Synbols demonstrate that our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods. We will make the code public.

Under review as a conference paper at ICLR 2021
BEYOND TRIVIAL COUNTERFACTUAL GENERATIONS WITH DIVERSE VALUABLE EXPLANATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Explainability of machine learning models has gained considerable attention within our research community given the importance of deploying more reliable machine-learning systems. Explanability can also be helpful for model debugging. In computer vision applications, most methods explain models by displaying the regions in the input image that they focus on for their prediction, but it is dif- ficult to improve models based on these explanations since they do not indicate why the model fail. Counterfactual methods, on the other hand, indicate how perturb the input to change the model prediction, providing details about the Unfortunately, current counterfactual methods make
model's recision-making ambiguous interpretations they combine multiple biases of the model and the data in a single counterfactual interpretation of the model's decision. Moreover, these methods tend to generate trivial counterfactuals about the model's decision, as they often suggest to exaggerate or remove the presence of the attribute be- ing classified. Trivial counterfactuals are usually not valuable, since the informa- tion they provide is often already known to the system's 's designer. In this work, we propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using diversity tiple valuable explanations about the model's s prediction. Further, we introduce mechanism to prevent the model from producing trivial explanations. Experi- ments on CelebA and Synbols demonstrate that our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-arr methods. We will make the code public.
loss to uncover mul-
INTRODUCTION
Consider a face authentication system for unlocking a device. In case of non-authentications (pos- sible false-negative predictions), this system could provide generic advices to its user such "face the camera" or "remove any face occlusions". However, these may not explain the reason for the possible malfunction. To provide more insights regarding its decisions, the system could instead provide information specific to the captured image (its input data). It might list the input features that most contributed to its decision (e.g., region of the input image), but this feature could be "face", which trivial and does not suggest an alternative action to its user. Further, it provides little useful information about the model. Instead, non-trivial explanations may be key for better understanding and diagnosing the system- including the data it was trained on- and improving its reliability. Such explanations might improve systems across wide variety of domains including in medical imaging [58), automated driving systems 48 and quality control in manufacturing [22].
The explainability literature aims understand the decisions made by a machine learning (ML) model such as the aformmentionned face authentication system. Counterfactual explanation meth- ods 11 13 can help discover the limitations of MI. model by uncovering data and model biases. The counterfactual explanation methods provide perturbed versions of the input data that emphasize features that contributed most to the ML model's output. For example, an authentication system is not recognizing user wearing sunglasses then the system could generate alternative image of the user's ' face without sunglasses that would be correctly recognized. This is different from other types of explainability methods such as feature importance methods [50] 51 4 and boundary ap- proximation methods (7]. The former highlight salient regions of the input but do not indicate how the MI. model could achieve different prediction. The second family of methods produce

---

Under review as a conference paper at ICLR 2021
explanations that are limited to linear approximations of the ML model. Unfortunately, these linear approximations are often inaccurate. In contrast, counterfactual methods suggest changes in the in- put that would lead to change in the corresponding output, providing information not only about where the change should be but also what the change should be.
Counterfactual explanations should be actionable, L.e., a user should be able act it. An action- able explanation would suggest feasible changes like removing sunglasses instead of unrealistic ones like adding more eyes to the user's face. Counterfactual explanations that are valid, proximal, and sparse are more likely to be actionable 44 38 That is, a counterfactual explanation that changes the outcome of the ML model (valid) by changing the minimal number of imput features (sparse), while remaining close to the input (proximal). Generating a set of diverse explanations increases the likelihood of finding an actionable explanation
A set of counterfactuals diverse if each one proposes to change a different set of attributes. Intuitively, each of these explanations shed light on a different action that user can take to change the MI. model's outcome.
49
38
Current counterfactual generation methods like xGEM [26] generates a single explanation that is far from the input. Thus, it fails to be proximal, sparse, and diverse. Progressive Exaggeration (PE) (53 provides higher-quality explanations more proximal than xGEM, but it still fails to provide diverse set of explanations. In addition, the image generator of PE is trained on the same data as the ML model in order to detect biases thereby limiting their applicability. Moreover, like the previous methods in the literature, these two methods tend to produce trivial explanations. For instance, an explanation that suggests increase the 'smile' attribute of a 'smile' classifier for an already-smiling face is trivial and it does not explain why a misclassification occurred. In this work, we focus on diverse valuable explanations (DiVE), that is, valid, proximal, sparse, and non-trivial.
We propose Diverse Valuable Explanations (DiVE), an explainability method that can interpret a ML model by identifying sets of valuable attributes that have the most effect on the ML model's output. DiVE produces multiple counterfactual explanations which are enforced to be valuable, and diverse resulting in more actionable explanations than the previous literature. Our method first learns generative model of the data using 3-TCVAE 5 to obtain a disentangled latent representation which leads to more proximal and sparse explanations. In addition, the VAE is not required to be trained on the same dataset as the ML model to be explained. DIVE then learns a latent perturbation using constraints to enforce diversity, sparsity, and proximity. In order to generate non- trivial explanations, DiVE leverages the Fisher information matrix of its latent space to focus its search on the less influential factors of variation of the ML model. This mechanism enables the discovery of spurious correlations learned by the ML model.
We provide experiments to assess whether our explanations are more valuable and diverse than current state-of-the-art, First, we assess their validity on the CelebA dataset [33 and provide quan- titative and qualitative results on a bias detection benchmark [53]] Second, we show that the gen- erated explanations are more proximal in terms of Fréchet Inception Distance (FID) 19] which is a measure of similarity between two datasets of images commonly used to evaluate the generation quality of GAN. In addition, we evaluate the latent space closeness and face verification accuracy, as reported by Singla et al. [53]. Third, we assess the sparsity of the generated counterfactuals by computing the average change facial attributes. Fourth, we show that DiVE more successful at finding more non-trivial explanations than previous methods and baselines. In the supplementary material we provide additional results on the outtt--distribution performance of DIVE.
We summarize the contributions of this work follows: 1) We propose DIVE, an explainability method that can interpret a ML model by identifying the attributes that have the most effect on its output. 2) DiVE achieves state of the art in terms of the validity, proximity, and sparsity of its explanations, detecting biases on the datasets, and producing multiple explanations for an image. 3) We identify the importance of finding non-trivial explanations and we propose a new benchmark to evaluate how valuable the explanations are. 4) We propose to leverage the Fisher information matrix of the latent space for finding spurious features that produce non-trivial explanations.
2 RELATED WORK
Explainable artificial intelligence (XAD) is a suite of techniques developed to make either the con- struction or interpretation of model decisions more accessible and meaningful. Broadly speaking, there are two branches of work in XAI, ad-hoe and post-hoe. Ad-hoc methods focus on making mod-
2

---

Under review as a conference paper at ICLR 2021
els interpretable, by imbuing model components parameters with interpretations that rooted in the data themselves [45] 39 25]. Unfortunately, most successful machine learning methods, in- cluding deep learning ones, uninterpretable [6 32 18 24]
Post-hoc methods aim to explain the decisions of non interpretable models. These methods can be categorized non-generative and generative. Non-generative methods use information from ML model to identify the features most responsible for an outcome for a given input. Approaches like
fit locally in- terpretable model. Others use the gradient of the ML model parameters to perform feature attri-
47
37
41
interpret ML model decisions by using derived information
I [T] sometimes by employing a reference distribution for the fea- tures [1 This has the advantage of identifying alternative feature values that when substituted for the observed values would result in a different mode outcome. These methods are limited to small contiguous regions of features with high influence on the target model outcome. In so doing, they can struggle to provide plausible changes of the input that are actionable by an user in order to propose plausible modifications of the input that change the model decision. However the generated perturbations usually found in pixel space and thus bound to masking small regions of the image without necessarily having a semantic meaning. Closest to our work are generative counterfactual expla- which synthesize perturbed versions of observed data that result in a corresponding change of the model prediction. While these methods provide valid and proximal explanations for model outcome, they fail to provide diverse set of non-trivial explanations. Mothilal et al. (38) addressed the diversity problem by introducing a diversity constraint between set of randomly initialized counterfactuals (DICE). However, DICE shares the same problems as 7 4 since perturbations are directly performed on the observed feature space, and does not take into account trivial explanations.
bution 159
60
52
54
50
correct a certain output or bias of the model. Generative methods such as 7 S
nation methods (26] 9 15 53
In this work we propose DIVE, a counterfactual explanation method that generates diverse set of valid, proximal, sparse, and non-trivial explanations. Appendix A provides more exhaustive review of the related work.
3 PROPOSED METHOD
We propose DiVE, an explainability method that can interpret ML model by identifying the latent attributes that have the most effect on its output. Summarized in Figure [ DiVE uses an encoder, decoder, and fixed weight MI model. The ML model could be any function for which we have access to its gradients. In this work, we focus on a binary image classifier in order to produce visual explanations. DiVE consists of two main steps. First, the encoder and the decoder are trained in an unsupervised manner to approximate the data distribution on which the ML model was trained. Unlike PE [53], our encoder-decoder model does not need to train on the same dataset that the ML model was trained on. Second, we optimize set of vectors to perturb the latent representation generated by the trained encoder. The details of the optimization procedure are provided in Algo- rithm in the Appendix. We use the following 3 main losses for this optimization: counterfactual loss LCF that attempts to fool the ML model, an proximity loss Cprox that constrains the cxpla- nations with respect to the number of changing attributes, and diversity loss Ldiv that enforces the explainer to generate diverse explanations with only one confounding factor for each of them. Finally, we propose several strategies to mask subsets of dimensions in the latent space to prevent the explainer from producing trivial explanations. Next we explain the methodology in more detail.
3.1 OBTAINING MEANINGFUL REPRESENTATIONS.
Given a data sample x E X, its corresponding target y E (0. 1), and a potentially biased MI. model fx) that approximates p(z))x,, method finds perturbed version of the same input X that produces a desired probabilistic outcome ÿ E [0, 1]. so that f(x) = ÿ. In order to produce semanti- performed on latent representation cally meaningful counterfactual explanations, perturbations z E Z C Rd of the input x. Ideally, each dimension in 2 represents a different semantic concept of the data, i.e., the different dimensions disentangled.
For training the encoder-decode architecture we use 3-TCVAE (5) since it has been shown to obtain competitive disentanglement performance (34] It follows the same encoder-decoder structure the
3

---

Under review as a conference paper at ICLR 2021
div + L prox
Trivial Counterfactuals
Z = Latent representation
e Perturbation
Encoder
Decoder
Not bald
bald
Non-Trivial Counterfactuals
Bald
ML model
CCF
Not Bald
Not bald
Not bald
Figure 1: DiVE encodes the input image (left) to explain into a latent representation Then perturbed by e and decoded as counterfactual examples. During training, Lcr finds the set of that change the ML model classifier outcome while Ldiv and Lprox enforce that the samples are diverse while staying proximal. These are four valid counterfactuals generated from the experiment in Section 4.4 However, only the bottom row contains counterfactuals where the man is still bald as indicated by the oracle or a human. These counterfactuals identify a weakness the ML model.
VAE (30] i.e., the input data is first encoded by neural network (o(2|x) parameterized by ø. Then, the input data is recovered by a decoder neural network po (x(22), parameterized by A. Using prior p(z) and a uniform distribution over the indexes the dataset p(1), the original VAE loss is:
AV (/tt: (o (x,, PKL (2x(x/x))lppzz))
(1)
where the first is the reconstruction loss and the second is the average divergence from the prior. The core difference of 3-TCVAE is the decomposition of this average divergence:
Ep(i) DKL (1o()|x)) (p(=)) + DKL. (go(z) II, (z)))
DEL (go x;)
+ EK (10(2)))1((3s)
where the arrow represents modification of the left terms and equality is obtained when 3 1. The third term on the right hand side is called total correlation and measures the shared information between all empirical marginals (2,) [o xx)). By using 1, this part is amplified and encourages further decorrelations between the latent variables and leads to better disentanglement.
In addition 3-TCVAE, we use the perceptual reconstruction loss from Hou et al. 20 This replaces the pixel-wise reconstruction loss in Equation | by perceptual reconstruction loss, using the hidden representation of a pre-trained neural network R. Specifically, we learn a decoder Dg generating image i.e., X Dg(z), and this image is re-encoded in hidden representation: = R()), and compared to the original image in the same space using a normal distribution. The reconstruction loss of Equation nnow now becomes:
E (log. R(Do(z)), 1)),
Once trained, the weights of the encoder-decoder are fixed for the rest of the steps of our algorithm.
3.2 INTERPRETING THE ML MODEL
In order to find weaknesses in the ML model, DiVE searches for a collection of n latent perturbation {e, == such that the decoded output &; Dz(z+e;) yields specific response from the ML model, i.e., f(x) for any chosen ÿ [0. 1]. We optimize e; by minimizing:
LDIVE(X, G, (6)))) = E; fcr(x,r(x) G. G)) + A Σ, ExpoxxXX, 6 + a. Laiv ((
(4)
where \, and a determine the relative importance of the losses. The minimization is performed with gradient descent and the complete algorithm can be found in Algorithm [ describe the different loss terms.
Appendix D We
Counterfactual loss. The goal of this loss function is to identify a change of latent attributes that will cause the MI. model to change it's prediction. For example, in face recognition, if the classifier

---

Under review as a conference paper at ICLR 2021
detects that there is smile present whenever the hair is brown, then this loss function is likely to change the hair color attribute. This is achieved by sampling from the decoder x = Dø + e), and optimizing the binary cross-entrope between the target and the prediction f(x):
LCF (x, V. E) .j log(s()) + (1 5) log(1 1 s(x)).
(5)
Proximity loss. The goal of this loss function is to constrain the reconstruction produced by the decoder to be similar in appearance and attributes as the input. It consists of the following two terms,
(6) where y is a scalar weighting the relative importance of the two terms. The first term ensures that the explanations can be related to the input by constraining the input and the output be similar. The second term aims to identify a sparse perturbation to the latent space Z that confounds the ML model. This constrains the explainer to identify the least amount of attributes that affect the classifier's 's decision in order to produce sparse explanations.
Sprox (x, c) = ||x x||1 + ? |||||2,
Diversity loss. This loss prevents the multiple explanations of the model from being identical. For instance, if gender and hair color spuriously correlated with smile, the model should provide images either with different gender or different hair color. To achieve this, we jointly optimize for a collection of perturbations (e, ) and minimize their pairwise similarity:
Laiv ((e
E,
|e:l12 ||ıllR
ifj
The method resulting of optimizing Eq. (DiVE) results in diverse counterfactuals that are more valid, proximal, and sparse. However, it may still produce trivial explanations, such as exaggerating smile to explain smile classifier without considering other valuable biases in the ML model such as hair color. While the diversity loss encourages the orthogonality of the explanations, there might still be several latent variables required to represent all variations of smile.
Beyond trivial counterfactual explanations. To find non-trivial explanations, we propose to pre- vent DIVE from perturbing the most influential latent factors of on the MI. model. We estimate the influence of each of the latent factors with the average Fisher information matrix:
F = Ep (zlx, ( V In V. In p(y/z))
where p(y= = 1|z) (D)( and p(y = 0/z) = f(D The diagonal values of F express the relative influence of each of the latent dimensions on the classifier output. Since the most influ- ential dimensions are likely to be related to the main attribute used by the classifier, we propose to prevent Eq. 4 from perturbing them in order to find more surprising explanations. Thus when pro- ducing n explanations, we sort Z by the magnitude of the diagonal, we partition it into n contiguous chunks that will be optimized for each of the explanations. We call this method DiVEpisher-
However, DiVErrsher does not guarantee that the different partitions of Z all the factors concerning trivial attribute are grouped together. Thus, we propose partition Z into subsets of latent factors that interact with each other when changing the predictions of the ML model. Such interaction can be estimated using F as an affinity measure. We use spectral clustering 55 to obtain partition of 2. This partition is represented as a collection of mask (m,))_ where m. E {0, } represents which dimensions of 2 are part of cluster i. Finally, these masks used in Equation 4 to bound each E to its subspace i.e., é E; o m,, where represents element wise multiplication. Since these masks are orthogonal, this effectively replaces Ldiv. In Section 4 we highlight the benefits of this clustering approach by comparing to other baselines. We call this method DiV DiVEpisherSpeettaal
4 EXPERIMENTAL RESULTS
In this section, we evaluate the described methods on 5 different aspects: (1) the validity of the generated explanations well as the ability to discover biases within the ML model and the data (Section 4.1 : (2) their proximity in terms of FID, latent space closeness, and face verification accu-
racy (Section
(3) the sparsity of the generated counterfactuals (Section 4.3) ; and (4) the ability
to identify diverse non non-trivial explanations for image misclassifications made by the ML model (Section 4.4 (5) the
performance of DiVE (Section 4.4.
5

---

Under review as a conference paper at ICLR 2021
Table I: Bias detection experiment. Ratio of generated counterfactuals classified as "Smiling" and "Non-Smiling" for a classifier biased on gen- der (frieed) and an unbiased classifier (f (unbisss)) Bold indicates Overall closest to the Ground truth.
Table 2: FID of DiVE compared to xGEM (26],
Progressive Exaggeration (PE) 53 xGEM
trained with our backhone (xGEM+), and DiVE
trained without the perceptual loss (DiVE-
Target label
Target Attribute XGEM
PE XGEM+ DIVE
DIVE
ML model
Smillng XGEM+
Non-Smiling xGEM+ DIVE
Smiling
model
PE
DIVE
PE
Present Absent
111.0 46.9
67.2
54.9
30,6
Male
0.52
0.94
0.89
0.18
0.24
0.16
fixsed
Female
0.48
0.06
0.11
0.82
0.77
0.84
112.9 56.3
77.8
62.3
33.6
Overall
0.12
0.29
0.22
0.35
0.33
0.36
Overall
106.3
35.8
66.9
55.9
29.4
Ground truch
0.75
0.67
Young
Mali
0,48
0.41
0.42
0.47
0.38
0.44
faeose
Female
0.58
Present
115.2
67.6
68.3
57.2
31.8
Overall
0.07
0.13
0.10
0.08
0.15
0.07
Absent
170.3
74.4
76.1
51.1
45.7
0004
0.00
Overall
117.9 53.4
59.5
47.7
33.8
we perform experiments on the CelebA database [33]. CelebA is a large-scale dataset containing more than 200K celebrity facial images. Each image is annotated with 40 binary attributes such as "Smiling".. "Male", and "Eyeglasses", These attributes allow us to evaluate counterfactual explanations by determining whether they could highlight spurious correlations between multiple attributes such as "lipstick" and "smile". In this setup, explainability methods are trained in the training set and ML models are explained on the validation set. The hyperparameters of the explainer are searched by -rossvvaliattion the training Explainers do not have access to the
Experimental Setup. As common procedure 26
9
53
set. We use the same train and validaton splits as PE 53 labeled attributes during training.
We test the out-if-distributtin (OOD) performance of DiVE with the Synbols dataset 31 Synbols an image generator with characters from the Unicode standard and the wide range of artistic fonts provided by the open font community. This provides us to better control on the features present in each set when compared CelebA. We generate 100K black and white of 32x32 images from 48 characters in the latin alphabet and more than 1K fonts. We use the character type to create disjoint sets for OOD training and we use the fonts to introduce biases in the data. We provide a sample of the dataset in Figure 8 in Appendix
We compare four versions of our method to three existing methods. DIVE, resulting of optimizing Eq. 4 DiVEFishers which extends DiVE by using the Fisher information matrix introduced in Eq. DiVepsherSpeearall which extends DiVEFisher with spectral clustering. We introduce two additional ablations of our method, DiVE- and DiVERandom DiVE is equivalent to DiVE but using pixel- based reconstruction loss instead of the perceptual loss. DiVERando uses random masks instead of using the Fisher information. Finally, we compare our baselines with xGEM as described in Joshi et al. (26). xGEM+, which is the same as xGem but uses the same auto-encoding architecture as DiVE, and PE as described by Singla et al. (53). For our methods, we provide implementation details, architecture description, and algorithm in Appendix D
4.1 VALIDITY AND BIAS DETECTION
We evaluate DiVE's ability to detect biases in the data. We follow the same procedure as PE [53 and train two binary classifiers for the attribute "Smiling". The first one is trained on a biased version of CelebA where all the male celebrities are smiling and all the female are not smiling (foiased). The second one is trained on the unbiased version of the data (funbiased). Both classifiers are evaluated on the CelebA validation set. Also following Singla et al. [53), we train an oracle classifier (foracee) based on VGGFace2 3 which obtains perfect accuracy on the gender attribute. The hypothesis that if "Smiling" and gender are confounded by the classifier, so should be the explanations.
Therefore, could identify biases when the generated examples not only change the target attribute but also the confounded one. To generate the counterfactuals, DiVE produces perturbations until changes the original prediction of the classifier (e.g. "Smiling" to "Non-Smiling").
We follow the procedure introduced in 22 53 and report a confounding metric for bias detection in In Table I The columns Smiling and Non-Smiling indicate the target class for counterfactual generation. The rows Male and Female contain the proportion of counterfactuals that are classified by the oracle Male and Female. We that the generated explanations for faisee are classified

---

Under review as a conference paper at ICLR 2021
Average number attributes changed per explanation and percentage of non-trivial explana- tions. This experiment evaluates the counterfactuals generated by different methods for ML model trained on the attribute 'Young' of the CelebA dataset. xGEM++ is xGEM+ using 3-TCVAE as generator.
Table 3:
PE 53
xGEM+ [26
xGEM++ DiVE DiVEraer
DiV DrVErrerSppeatalll
Attr. change
03.74
06.92
06.70
04.81
04.82
04.58
Non-trivial (%)
05.12
18.56
34.62
43.51
42.99
51.07
more often as Male when the target attribute is Smiling and Female when the target attribute is Non- Smiling. The confounding metric, denoted as overall, is the fraction of generated explanations for which the gender was changed with respect to the original image. It thus reflect the magnitude of the the bias as approximated by the explainers.
Singla et al. 153] consider that model is better than another if the confounding metric is the highest on foinsed and the lowest on funbiased- However, they assume that fosased always predicts the Gender based Smile. Instead, we propose to evaluate the confounding metric by comparing it to the empirical bias of the model, denoted as ground truth in the Table T
Details provided in Appendix [
We observe that DiVE is more successful than PE detecting biases although the generative model of DiVE was not trained with the biased data. While xGEM+ has a higher success rate at detecting biases in some cases, produces lower-quality images that are far from the input. In Figure 5 in Appendix B we provide samples generated by our method with the two classifiers and compare them to PE and xGEM+. We found that gender changes with the "Smiling" attribute with fulased while for funbiased it stayed the same. In addition, we also observed that for folased the correlation between "Smile" and "Gender" is higher than for PF. can also be observed that xGEM+ fails to retain the identity of the person in x when compared to PE and our method.
4.2 COUNTERFACTUAL EXPLANATION PROXIMITY
We evaluate the proximity of the counterfactaal explanations using FID scores 19 as described by Singla et al. [53] The scores are based on the target attributes "Smiling" and "Young", and are divided into categories: Present, Absent, and Overall. Present considers explanations for which the ML model outputs probability greater than 0.9 for the target attribute. Absent refers to explanations with a probability lower than 0.1. Overall considers all the successful counterfactuals, which changed the original prediction of the ML model.
We report these scores in for all categories. DiVE produces the best quality counterfactuals, surpassing PE by 6.3 FID points for the "Smiling" target and 19.6 FID points for the "Young" target in the Overall category. DiVE obtains lower FID than xGEM+ which shows that the improvement not only comes from the superior architecture of our method. Further, there are two other factors that explain the improvement of DiVE's FID. First, the 3-TCVAE decomposition of the KL divergence improves the disentanglement ability of the model while suffering less reconstruction degradation than the VAE. Second, the perceptual loss makes the image quality constructed by DiVE to be comparable with that of the GAN used in PE. In addition, Table the Appendix shows that DiVE is more successful at preserving the identity of the faces than PE and xGEM and thus at producing feasible explanations. These results suggest that the combination of disentangled latent features and the regularization of the latent features help DiVE to produce the minimal perturbations of the input that produce a successful counterfactual.
In Figure 5 in Appendix B we show qualitative results obtained by targeting different probability ranges for the output of the ML model as described in PE. As seen Figure 5 DiVE produces more natural-lookin facial expressions than xGEM+ and PE. Additional results for "Smiling" and "Young" are provided in Figures and and and in the Appendix B
4.3 COUNTERFACTUAN EXPLANATION SPARSITY
Explanations that produce sparse changes in the attributes of the image more probable to be actionable. In this section we quantitatively compare the amount of valid and sparse counterfactuals provided by different baselines. Table shows the results for classifier model trained the at-
7

---

Under review as a conference paper at ICLR 2021
tribute Young of the CelebA dataset [ The first row shows the number of attributes that each method change in average to generate a valid counterfactual. Methods that require to change less attributes likely to be more actionable. We observe that DiVE changes less attributes on average than xGEM+. We also observe that DiVErsherSpeccaall is the method that changes less attributes among all the baselines. To better understand the effect of disentangled representations, we also report results for a version of xGEM+ with the 3-TCVAE backbone (xGEM++). We do not observe significant effects on the sparsity of the counterfactuals. In fact, a fine-grained decomposition of concepts in the latent space could lead to lower the sparsity.
4.4 BEYOND TRIVIAL EXPLANATIONS
Previous works on counterfactual generations tend to produce trivial input perturbations to change the output of the ML model. That is, they tend to increase/decrease the presence of the attribute that the classifier is predicting. For instance, in Figure 5all the explainers put a smile on the input face in order to increase the probability for "smile". While that is correct, this explanation does not provide much insight about the potential weaknesses of the ML model. Instead, in this work we emphasize producing non-trivial explanations, that are different from the main attribute that the MI. model has been trained to identify. These kind of explanations provide more insight about the factors that affect the classifier and thus provide cues on how to improve the model or how to fix incorrect predictions.
To evaluate this, we propose a new benchmark that measures a method's 's ability to generate valuable explanations. For an explanation to be valuable, it should 1) be misclassified by the ML model (valid), 2) not modify the main attribute being classified (non-trivial), and 3) not have diverged too much from the original sample (proximal). A misclassification provides insights into the weaknesses of the model. However, the counterfactual is even more insightful when it stays close to the original image as it singles-out spurious correlations learned by the MI. model. Because it is costly to provide human evaluation of an automatic benchmark, approximate both the proximity and the real class with the VGGFace2-basee oracle. We choose the VGGFace2 model as it is less likely to share the same biases as the ML model, since was trained for different task than the ML model with an order of magnitude more data. We conduct a human evaluation experiment in Appendix D and we find a significant correlation between the oracle and the human predictions. For 1) and 2) we deem that an explanation is successful if the ML model and the oracle make different predictions about the counterfactual. F.g., the top counterfactuals in Figure [ are not deemed successful explanations because both the MI. model and the oracle agree on its class, however the two in the bottom row are successful because only the oracle made the correct prediction. These explanations where generated by DiV DiVerssherppemaall As for 3) measure the proximity with the cosine distance between the sample and the counterfactual in the feature space of the oracle.
We test all methods from Section a subset of the CelebA validation set described in Appendix E We report the results of the full hyperparameter search (see Appendix [E) in Figure 2a The vertical axis shows the success rate of the explainers. i.e. the ratio of valid explanations that are non-trivial. This is the misclassification rate of the MI. model on the explanations. The dots denote the mean performances and the curves are computed with Kernel Density Estimation (KDE). On average, DiVE improves the similarity metric over xGEM+ highlighting the importance of disentangled rep- resentations for identity preservation. Moreover, using information from the diagonal of the Fisher Information Matrix as described in Eq. g further improves the explanations as shown by the higher success rate of DiVErisher over DIVE and DiVERanom- Thus, preventing the model from perturbing the most influential latent factors helps to uncover spurious correlations that affect the ML model. Finally, the proposed spectral clustering of the full Fisher Matrix attains the best performance vali- dating that the latent space partition can guide the gradient-based search towards better explanations. We reach the same conclusions in Table 3, where we provide a comparison with PE for the attribute Young. In addition, we provide results for version of xGEM+ with more disentangled latent factos (xGEM++). We find that disentangled representations provide the explainer with a more precise control on the semantic concepts being perturbed, which increases the success rate of the explainer by 16%.
Out-of-distribun generalization. the previous experiments, the generative model of DiVE was trained on the same data distribution (i.e., CelebA faces) the ML model. We test the out-
The code and pre-trained models of PE are only available for the attribute Young.
8

---

Under review as a conference paper at ICLR 2021
XGEM+ DM
6.2
XGEM
XGEMS
DIVE
random DIVE fisher
2.93
DIVE
DIVEnner
24
6.6
6.33
0.41
33
erbudiing atrilurtt (FC0)
orbutiing (hammo122)
unbuadingg Steamei221
(a) CelebA (ID)
(b) Synbols (ID)
(c) Synbols (OOD)
Figure 2: Beyond trivial explanations. The rate successful explanations (y-axis) plotted against embedding similarity (x-axis) for all methods. For both metrics, higher is better, i.e., the most valu- able explanations are the top-right corner. For each method, we ran an hyperparameter sweep and denote the mean of the performances with dot. The curves are computed with KDE. The left plot shows the performance on CelebA and the other two plots shows the performance for in- distribution (ID) and out-of-distribution (OOD) experiments on Synbols. . All DiVE methods outper- form XGEM+ on both metrics simultaneously when conditioning on successful counterfactuals. In both experiments, DiVErister and DiVE pasheerpecca improve the performance over both Randdom and DiVE.
of of-distribution performance of DiVE by training its auto-encoder subset of the latin alphabet of the Synbols dataset [3.. Then, counterfactual explanations are produced for a different disjoint subset of the alphabet. To evaluate the effectiveness of DiVE in finding biases the MI model, we introduce spurious correlations in the data. Concretely, we assign different fonts to each of In-distribution (ID) results are reported in Figure 26 for reference, and OD results are reported in Figure 2c We observe that DiVE is able find valuable countefactuals even when the VAE was trained on the same data distribution. Moreover, results are consistent with the CelebA experiment, with DiVE outperforming xGEM+ and Fiser information based methods outperforming the rest.
the letters in the alphabet as detailed in Appendix [
LIMITATIONS AND FUTURE WORK
This work shows that a good generative model can provide interesting insights on the biases a ML. model. However, this relies properly disentangled representation. In the case where the gener- ative model would be heavily entangled would fail produce explanations with a sparse amount
of features. However, approach can still tolerate small amount of entanglement, yielding small decrease in interpretability. We expect that progress in identifiability 35] 28 will increase the quality of representations. With a perfectly disentangled model, approach could still miss some explanations or biases. E.g., with the spectral clustering of the Fisher, we group latent variables and only produce single explanation per group in order present explanations that are conceptually different. This may leave behind some important explanations, but the user can simply increase the number of clusters the number explanation per clusters for a more in-depth analysis.
In addition to the challenge of achieving disentangled representations, finding the optimal hyperpa- rameters for the VAE and their generalization Moreover, if the generative model is trained on biased data, one could expect the counterfactuals to be biased as well. However, as we show Figure [c our model still finds non-tu con-trivial explanations when applied out of distribution. In that way, it could be trained on a larger unlabeled dataset to overcome possible biases caused by the lack of annotated data.
of the training distribution an open problem.
Although the generative model plays important role to produce actionable counterfactuals in the computer vision domain domain, our work could be extended to other domains. For example, Eq. tabular data by directly optimizing the observed
could be applied to find non-trivial explanations features instead of the latent factors of the VAE. However, further work would be needed to adapt the DiVE loss functions produce perturbations on discrete and categorical variables.

---

Under review as a conference paper at ICLR 2021
REFERENCES
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv:1809.11000 2018
[3] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and Zisserman. Vggface2: A dataset for recog- nising faces across pose and age. In International Conference on Automatic Face and Gesture Recognition, 2018.
[4] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Leurning Representa- tions, 2019.
[5] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, 2018.
[6] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3): 273-297, 1995.
[7] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. arXiv preprint arXiv:170.007855 2017.
[8] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009.
[9] Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timmit Gebru. Detecting bias with generative counterfactual face attribute augmentation. arXiv preprint arXiv: 06439, 2019.
[10] Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, pp. 592-603, 2018.
[11] Ruth C Fong and Andrea Vedaldi. Interpretable explanations black boxes by meaningful perturbation. In International Conference Computer Vision, 2017.
[12] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. arXiv preprint arXiv: 10145, 2019.
[13] Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. In Advances in neural information processing systems, 2017.
[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
[15] Yash Goyal, Ziyan Wu, Jan Emstt, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint 4rXiv:1904.0745.. 2019.
[16] Riceardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. Factual and counterfactual explanations for black box decision making. IEEE Intelligent Systems, 34(6):14-23, 2019.
[17] Riceardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi. Black box explanation by learning image exemplars in the latent feature space. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 189-205. Springer, 2019.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition, 2016.
10

---

Under review as a conference paper at ICLR 2021
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
[20] Xianxu Hou, Linlin Shen, Sun, and Guoping Qiu. Deep feature consistent variational autoencoder. In Winter Conference on Applications of Computer Vision, 2017.
[21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely con- nected convolutional networks. In Computer Vision and Pattern recognition, 2017.
[22] Farhad Imani, Ruimin Chen, Evan Diewald, Edward Reutzel, and Hui Yang. Deep learning of variant geometry in layerwise imaging profiles for additive manufacturing quality control. Journal of Manafacturing Science and Engineering, 141(11), 2019.
[23] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
[24] Simon Jégou, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation. In Computer Vision and Pattern Recognition Workshops, 2017.
[25] Finn V Jensen et al. An introduction to Bayesian networks, volume 210. UCL press London, 1996.
[26] Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xgems: Generating examplars to explain black-box models. arXiv preprint arXiv:1806.08860 2018.
[27] Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. Model-agnostic coun- terfactual explanations for consequential decisions. In International Conference on Artificial Intelligence and Statistics, pp. 995-905, 2020.
[28] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp. 2207-2217. PMLR, 2020.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412,6900, 2014.
[30] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.66 14, 2013.
[31] Alexandre Lacoste, Pau Rodríguez López. Frédéric Branchaud-Charron, Parmida Atighe- hchian, Massimo Caccia, Issam Hadj Laradji, Alexandre Drouin, Matthew Craddock, Laurent Charlin, and David Vázquez. Synbols: Probing learning algorithms with synthetic datasets. Advances in Neural Information Processing Systems, 33, 2020.
[32] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied handwritten zip code recogni- tion. Neural Computation, 1(4):541-551, 1989.
[33] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision, 2015.
[34] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In International Conference on Machine Learning, 2019.
[35] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Tschannen Weakly suueris- disentanglement without compromises. arXiv preprint arXiv:2002.02886. 2020.
[36] James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding poste- rior collapse in generative latent variable models. 2019.

---

Under review as a conference paper at ICLR 2021
[37] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, 2017.
[38] Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classi- fiers through diverse counterfactual explanations. In Conference Fairness, Accountability, and Transparency, 2020.
[39] John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. Journal of the Royal Statistical Society: Series A (General), 135(3)3300-384, 1972.
[40] Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adap- tive metric for improved few-shot learning, Advances in Neural Information Processing Sys- tems, 31:721-731, 2018.
[41] Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, inter- pretable and robust deep learning, arXiv preprint
2018.
[42] Martin Pawelczyk, Klaus Broelemann, and Gjergji Kasneci. Learning model-agnostic coun- terfactual explanations for tabular data. Proceedings of The Web Conference 2020, pp- 3126-3132, 2020.
[43] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. arXiv preprint arXiv:1709.07871 2017.
[44] Rafael Poyiadzi, Kacper Sokol, Raul Santos- Rodrigu, Tijl De Bie, and Peter Flach. Face: feasible and actionable counterfactual explanations. In Proceedings of the AAAI/ACM Confer- ence on AI, Ethics, and Society, pp. 344-350, 2020.
[45] R. Quinlan. Induction of decision trees. Machine Learning, 1:81-106, 1986.
[46] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. Inter- national Conference on Learning Representations, 2018.
[47] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining, 2016.
[48] German Ros, Sebastian Ramos, Manuel Granados, Amir Bakhtiary, David Vazquez, and An- tonio M Lopez. Vision-based offfine-online perception paradigm for autonomous driving. In Winter Conference on Applications of Computer Vision, 2015.
[49] Chris Russell, Efficient search for diverse coherent explanations. Conference on Fairness, Accountability, and Transparency, 2019.
[50] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In International Conference on Computer Vision, 2017.
[51] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. arXiv preprint arXiv: 02600.0 2017.
[52] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint 1rXiv:1312.6034, 2013.
[53] Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In International Conference on Learning Representationss 2020.
[54] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
[55] Yu Stella and Jianbo Shi. Multiclass spectral clustering. In null, pp. 313. IEBE, 2003.
[56] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1607.00607 2016.
12

---

Under review as a conference paper at ICLR 2021
[57] Arnaud Van Looveren and Janis Klaise. Interpretable counterfactual explanations guided by prototypes. arXiv preprint arXivv1907.000 0254002844 2019.
[58] David Vázquez, Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrachh, Antonio M López, Adriana Romero, Michal Drozdzal, and Aaron Courville. A benchmark for endolu- minal scene segmentation of colonoscopy images. Journal of healthcure engineering, 2017.
[59] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object de- tectors emerge in deep scene cnns. arXiv preprint
2014.
[60] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Computer Vision and Pattern Recognition, 2016.
13

---

Under review as a conference paper at ICLR 2021
APPENDIX
A EXTENDED RELATED WORK
Counterfactual explanation lies inside a more broadly-connece body of work for explaining clas- sifier decisions. Different lines of work share this goal but vary in the assumptions they make about what elements of the model and data to emphasize as way of explanation.
Model-agnostic counterfactual explanation. Like 47 37 these models make no assumptions about model structure, and interact solely with its label predictions. Karimi et al. 27 develop model agnostic, as well as metric agnostic approach. They reduce the search for counterfactual ex- planations (along with user-provided constraints) into a series of satisfiability problems to be solved with off- shethh SAT solvers. Similar in spirit to 47), Guidotti et al. [16] first construct a local neighbourhood around test instances, finding both positive and negative exemplars within the neigh- bourhood. These are used to learn a shallow decision tree, and explanations are provided in terms of the inspection of its nodes and structure. Subsequent work builds on this local neighbourhood idea [77. but specializes to medical diagnostic images. They use VAE to generate both positive and negative samples, then use random heuristic search to arrive at a balanced set. The generated explanatory samples are used to produce saliency feature map for the test data point by considering the median absolute deviation of pixel-wise differences between the test point, and the positive and negative example sets.
Gradient based feature attribution. These methods identify input features responsible for the greatest change in the loss function, as measured by the magnitude of the gradient with respect to the inputs. Early work in this area focused on how methodological improvements for object detec- tion in images could be re-purposed for feature attribution [59, 60). followed by work summarized gradient information in different ways [52] 54 50]. Closer inspection identified pitfalls of gradient- based methods, including induced bias due to gradient saturation or network structure [I]. as well as discontinuity due to activation functions [5T]. These methods typically produce dense feature maps, which are difficult to interpret. In our work we address this by constraining the generative process of our counterfactull explanations.
Reference based feature attribution. These methods focus instead on measuring the differences observed by substituting observed input values with ones drawn from some reference distribution, and accumulating the effects of these changes as they are back paok-propagated to the input features. Shrikumar et al. [5T) use a modified pack-propagation approach to gracefully handle zero gradients and negative contributions, but leave the reference to be specified by the user. Fong & Vedaldi [ propose three different heuristics for reference values: replacement with constant, addition of noise, and blurring. Other recent efforts have focused on more complex proposals of the reference distribution. Chen et al. [5] construct probabilistic model that acts lower bound the mutual information between inputs and the predicted class, and choose zero values for regions deemed uninformative. Building on desiderata proposed by Dabkowski & Gal 77 Chang al. 4 use generative model to marginalize over latent values of relevant regions, drawing plausible values for each. These methods typically either do not identify changes that would alter a classifier decision, or they do not consider the plausibility of those changes.
Counterfactual explanations. Rather than identify a set of features, counterfactual explanation methods instead generate perturbed versions of observed data that result corresponding change in model prediction. These methods usually assume both more access to model output and pa- rameters, as well as constructing a generative model of the data to find trajectories of variation that elucidate model behaviour for given test instance.
Joshi et al. (26] propose a gradient guided search in latent space (via a learned encoder model), where they progressively take gradient steps with respect to a regularized loss that combines a term for plausibility of the generated data, and the loss of the ML model. Denton et al. [9] use a Generative Adversarial Network (GAN) [14] for detecting bias present in multi-label datasets. They modify the generator to obtain latent codes for different data points and learn a linear decision boundary in the latent space for each class attribute. By sampling generated data points along the vector orthogonal
14

---

Under review as a conference paper at ICLR 2021
to the decision boundary, they observe how crossing the boundary for one attribute causes undesired changes in others. Some counterfactual estimation methods forego a generative model by instead solving surrogate editing problem. Given original image (with some predicted class), and an image with desired class prediction value, Goyal et al. [15] produce a counterfactual explanation through series of edits to the original image by value substitutions in the learned representations of both images. Similar in spirit are Dhurandhar et al. [0O] and Van Looveren & Klaise [57]. The former propose a search over features highlight subsets of those present in each test data point that are typically present in the assigned class, as well as features usually absent in examples from adjacent classes (instances of which are easily confused with the label for the test point predicted by the model). The latter generate counterfactual data that is proximal to zyest, with a sparse set of changes, and close to the training distribution. Their innovation to use class prototypes to serve additional regularization term in the optimization problem whose solution produces counterfactual.
Several methods go beyond providing counterfactually generated data for explaining model deci- sions, by additionally qualifying the effect proposed changed between a test data point and each counterfactual. Mothilal et al. [3] focus tabular data, and generate sets of counterfactual expla- nations through iterative gradient based improvement, measuring the cost of each counterfactual by either distance in feature space, the sparsity of the set of changes (while also allowing domain expertise to he applied). Poyiadzi et al. 44 construct a weighted graph between each pair of data point, and identify counterfactuals (within the training data) by finding the shortest paths from test data point to data points with opposing classes, Pawelczyk et al. [42 focus on modelling the density of the data to provide ''tttaiaale'' counterfactuals, defined to be proximal test data points, yet not lying in low-density sub-spaces of the data. They further propose to weigh each counterfactual by the changes in percentiles of the cumulative distribution function for each feature, relative to the value of test data point.
15

---

Under review as a conference paper at ICLR 2021
B QUALITATIVE RESULTS
In Figure we show qualitative results obtained by targeting different probability ranges for the output of the ML model described in PE. Note that PE directly optimizes the generative model to take an input variable å E IR that defines the desired output probability ÿ f(x + 6. To ob- tain explanations at different probability targets, we train a second order spline on the trajectory of perturbations produced during the gradient descent steps of our method. Thus, given the set of perturbations {c.). Vt E L..T, obtained during т gradient steps, and the corresponding black- box outputs (f(v)e=)), the spline obtains the eg for target output by interpolation. As seen in Figure 5 DiVE produces more natural-looking facial expressions than xGEM+ and PE. Although DiVE is not explicitly trained to produce exemplars at intermediate target probabilities, expla- nations are more correlated with the target probabilities than PE. Additional results for "Smiling" and "Young" are provided in Figure
Figure 3 4 present counterfactual explanations for additional persons and attributes. The results show that DiVE achieves higher quality reconstructions compared to other methods. Further, the reconstructions made by DiVE are more correlated with the desired target for the MI. model output fax). In Figure 55] provide samples generated by method with a gender-biased classifier foiased and an unbiased classifier funbiased. We compare DiVE to PE and xGEM+. We found that gender changes with the "Smiling" attribute with foinsed while for funbissed it stayed the same. In addition, we also observed that for fbased the correlation between "Smile" and "Gender" is higher than for PE. It can also be observed that xGEM+ fails to retain the identity of the person x when compared to PE and our method. Finally, Figure 6 shows successful counterfactuals for different instantiations of DiVE.
16

---

Under review as a conference paper at ICLR 2021
Query Image
Generated Visual Explanations
Desired f(x)
[0-0.1)
[0.2-0.3)
(0.3-0.4)
[0.5-0.6)
(0.6-0.7)
(0.7-0.8)
(0.9-1.0)
Not-smiling
Smiling 1.0
f(x 1.0
(x)
0.04
0.39
0.47
0.63
0.84
0.95
PE
1.0
0.01
0.25
0.35
0.55
0.65
0.75
0.97
XGEM+
1.0
0.0
0.25
0.35
0.55
0.66
0.76
0.99
Ours
0.2
0.0
0.31
0.25
0.26
0.46
0.69
0.97
PE
0.2
0.26
0,99
XGEM+
0.96
Ours
Figure 3: Qualitative results of DiVE, Progressive Exaggeration (PE) 153 and xGEM 26 for the "Smiling" attribute. Each column shows the explanations generated for a target probability output of the MI. model. The numbers top of each show the actual output of the MI. model.
17

---

Under review as a conference paper at ICLR 2021
Query Image
Generated Visual Explanations
Desired f(x)
(0-0.1)
(0.2-0.3)
(0.3-0.4)
(0.5-0.6)
(0.6-0.7)
(0.7-0.8)
(0.9-1.0)
Old
Young 0.97
f(x) 0.96
(x)
0.11
0.18
0.28
0.46
0.45
0.75
PE
0.96
0.25
0,25
0.34
0.55
0.65
0.75
0.95
XGEM+
0.96
0.03
0.2
0.3
0.5
0.7
0.96
Ours
0.16
0.07
0.41
0.38
0.56
0.36
0.79
0.93
PE
0.16
0.07
0.21
0,3
0.5
0.6
0.7
0.96
XGEM+
0.16
0.07
0.21
0.3
0.5
0.6
0.7
0.96
Ours
Figure 4: Qualitative results of DiVE, Progressive Exaggeration (PE) [53]. and xGEM+ for the "Young" attribute. Each column shows the explanations generated for a target probability output of the ML model. The numbers on top of each row show the actual output of the ML model.
18

---

Under review as a conference paper at ICLR 2021
Query Image Desired f(x):
Generated Visual Explanations
[0.0-0.1)
[0.2-0.3)
(0.3-0.4)
[0.5-0.6)
(0.6-0.7)
(0.7-0.8)
[0.9-01.0] Smiling
Not-smiling
pamilin :..0
pemiin (R):
mman (x)::.0
orncle
(X) 0.0
0.0
0.0
0.0
0.0
0.01
0.0
XGEM
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.01
0.25
0.33
0.57
0.67
0.75
0.99
0.0
0.0
0.0
0.0
0,0
0.0
0.0
0.C0
smiling
[ 0.0
0.23
0.32
0.38
0.49
0.69
0.94
(x]: 0.0
Lenadal (8) 0.0
0.02
0.08
0.10
0.31
0.89
0.96
0.95
X
0.0
0.1
0.6
0.7
0.99
0.0
0.01
0.25
0.32
0.54
0.68
0.80
0.99
Ours
0.0
0.01
0.27
0.34
0.49
0.57
0.93
Figure 5: Bias detection experiment. Each column presents an explanation for target "Smiling" probability interval. Rows contain explanations produced by PE (53), xGEM+ and our DiVE. (a) of uender-unbiased classifier, and (b) corresponds to explanations of gender-biased "Smile" classifier. The classifier output probability displayed on top of the images while the oracle prediction for gender is displayed at the bottom.
19

---

Under review as a conference paper at ICLR 2021
f fsmile (x) =
DIVE w/o Liite
original
DIVE
DIVE-F
fsmile(x) =
0
DIVE-FS
Figure 6: Successful counterfactual generations for different instantions of DiVE, Here, the original
image was misclassified non-smiling. All methodologies were able to correctly add a smile to the woman.
20

---

Under review as a conference paper at ICLR 2021
C IDENTITY PRESERVATION
As argued, valuable explanations should remain proximal to the original image. Accordingly, per- formed the identity preservation experiment found in (53) benchmark the methodologies against each other. Specifically, use the VGGFace2-based [3] oracle to extract latent codes for the original images as well as for the explanations and report latent space closeness as the fraction of time the explanations' latent codes are the closest to their respective original image latent codes' compared the explanations on different original images. Further, we report face verification accuracy which consist of the fraction of time the cosine distance between the aforementioned latent codes is below 0.5.
Table presents both metrics for DiVE and its baselines on the "Smilling" and "Young" classifica- tion tasks. We find that DiVE outperforms other methods on the "Young" classification task and almost all the "Smiling" task.
CelehA:Smiling :Smiling
CelebA:Voung
xGEM
PE
xGEM+
(ours)
xGM
PE
xGEM-
DIVE (ours)
Latent Space Closeness
88.2
88.0
99.8
98.7
89.5
81.6
97.5
99.1
Face Verification Accuracy
0.0
85.3
91.2
97.3
0.0
72.2
97.4
98.2
Table 4: Identity preserving performance on two prediction tasks.
21

---

Under review as a conference paper at ICLR 2021
D IMPLEMENTATION DETAILS
In this Section, we provide provide the details to ensure the that our method is reproducible.
Architecfure details. DiVE's architecture is a variation BigGAN 2 as shown in Table We chose this architecture because it achieved impressive FID results on the ImageNet [ The decoder (Table (bb) simplified version of the 128 128 BigGAN's residual generator, without non- local blocks nor feature concatenation. We use InstanceNorm [56] instead of BatchNorm [23] to obtain consistent outputs at inference time without the need of an additional mechanism such as recomputing statistics [2]. All the InstanceNorm operations of the decoder are conditioned on the input code z in the same way as FILM layers 43 The encoder (Table/6a) follows the same structure as the BigGAN 128 128 discriminator with the same simplifications done to our generator. We use the Swish non-linearity 46 in all layers except for the output of the decoder, which uses a Tanh activation.
For all experiments we use a latent feature space of 128 dimensions. The ELBO has a natural princi- pled way of selecting the dimensionality of the latent representation. If d is larger than necessary, it will not enhance the reconstruction error and the optimization of the ELBO will make the posterior equal the prior for these extra dimensions. More can be found on the topic in [36]. In practice, we experimented with d = [64, 128, 256} and found that with d = 128 we achieved a slightly lower ELBO.
To project the 2d features produced by the encoder to a flat vector (H, log (o²)), and to project the sampled codes to a 2d space for the decoder, we use 3-layer MLPs. For the face attribute classifiers, we use the same DenseNet [21] architecture as described in Progressive Exaggeration 53
Optimization details. All the models are optimized with Adam 229 with a batch size of 256. During the training step, the auto-encoders optimized for 400 epochs with a learning rate of 4. 10-1. The classifiers are optimized for 100 epochs with a learning rate of 10-1 To prevent the auto-encoders from suffering KL vanishing, adopt the cyclical annealing schedule proposed by Fu et al. [12] on the third term of Equation
Counterfactual inference details. At inference time, the perturbations are optimized with Adam until the MI. model output for the generated explanation f(x) only differs from the target output ÿ by a margin 8 or when a maximum number of iterations the experiments since more than 90% of the counterfactuals are found after that many iterations. The different E, are initialized by sampling from a normal distribution A ~ (0, 0.01). For the sort by the magnitude of
reached. We set = 20 for all
DiVEpisher baseline, to identify the most valuable explanations, f = diag(F) Then, we divide the dimensions of the sorted e into N contiguous partitions of N where is the dimensionality of Z. Formally, let ef)) be e sorted by f. then e(f) is
size = D constrained follows,
if i (Ii k otherwise
where i E 1..N indexes each of the multiple c, and j E 1..D indexes the dimensions of As result we obtain partitions with different order of complexity. Masking the first partition results in explanations that are most implicit within the model and the data. On the other hand, masking the last partition results in explanations that more explicit.
To compare with Singla et al. (53] in Figure B values y of the output of the ML model classifier. One way to achieve this would be to opti- mize fcr for each of the target probabilities. However, these successive optimizations would slow down the process of counterfactual generation. Instead, we propose to directly maximize the target class probability and then interpolate between the points obtained in the gradient descent trajectory obtain the latent factors of the different target probabilities. Thus, given the set of perturba- tions ) VI. E 1..7, obtained during gradient steps, and the corresponding MI. model outputs (f(g)))) ). we obtain the EG for target output ÿ by interpolation. We do such interpolation by fitting a piccewise quadratic polynomial on the latent trajectory, commonly known as Spline in the computer graphics literature.
we produced counterfactuals at arbitrary target
22

---

Under review as a conference paper at ICLR 2021
E BEYOND TRIVIAL EXPLANATIONS EXPERIMENTAL SETUP
The experimental benchmark proposed in Section 4.4 is performed on subset of the validation set of CelebA. This subset is composed of 4 images for each CelebA attribute. From these 4 images, were correctly classified by the ML model, while the other 2 were misclassified. The two correctly classified images are chosen that one was classified with high confidence of 0.9 and the other one with low confidence of 0.1. The 2 misclassification were chosen with the same criterion. The total size of the dataset is of 320 images. For each of these images we generate & counterfactual explanations. From these counterfactuals, we report the ratio of successful explanations.
Here are the specific values we tried in our hyperparameter search: 7 E [0.0, 0.001, 0.1, 1.0), a [0.0, 0.001, 0.1, 1.0), À E [0.0001, 0.0005, 0.001], number of explanations to 15 and learning rate [0.05, 0.1]. Be xGEM+ doesn't have a 7 nor a parameter, we increased its learning rate span to [0.01, 0.05, 0.1] to reduce the gap in its search space compared with DIVE. We also changed the ranomd seeds and ran a total of 256 trials.
23

---

Under review as a conference paper at ICLR 2021
F
HUMAN EVALUATION
Human ML Classifier (real non-trivial)
Method
Correlation
p-value
XGEM+ [26
38.37%
0.37
0.000
DiVE
38.65%
0.25
0.002
DiVERandom
38. 89%
0.24
0.001
DiVErisher
40.56%
0.17
0.023
41.90%
0.23
0.001
Table 5: Human evaluation. The first column contains the percentage of non-trivial counterfactuals from the perspective of the human oracle. These counterfactuals confuse the ML classifier without changing the main attribute being classified from the perspective of a human. The second column contains the Pearson correlation between the human and the oracle's predictions. The third column contains the p-value for t-test with the null hypothesis of the human and oracle predictions being uncorrelated.
We built web-based human evaluation task assess DiVE is more successful at finding non- trivial counterfactuals than previous state of the art and the effectiveness of the VGG-based oracle, see Figure 7 For that, we present humans with valid counterfactuals and ask them whether the main attribute being classified by the ML model is present in the image not. We use subset of CelebA containing random sample of images per attribute, each one classified by the VGG oracle containing the attribute with the following levels of confidence: [0.1, 0.4, 0.6, 0.9). From each of these 160 images, we generated counterfactuals with xGEM+ [26] DiVE, DiVEgandm. DiVErs and FiihherSpeenn and show the valid counterfactuals the human annotators. Results are reported in Table In the left column we observe that leveraging the Fisher information results in finding more non-trivial counterfactuals, which confuse the ML model without changing the main attribute being classified. In the second column report the Pearson correlation between the oracle and the classifier predictions. statistical inference test reveals significant correlation (p-value<0.022
you the following facial attribute? Straight, Hair
Figure 7: Labelling interface. The user is presented with counterfactual image and has to choose the target attribute present or not in the image.
24

---

Under review as a conference paper at ICLR 2021
G MODEL ARCHITECTURE
Table presents the architecture of the encoder and decoder used DiVE.
Table DiCe architecture for 128 x 128 images. ch represents the channel width multiplier in each network.
RGB image æ E R 1218 128x3
R128
ResBlock down 3ch - 16ch
Linear 128ch - 128ch
ResBlock 16ch - 32ch
Linear 128ch - 128ch
ResBlock down 32ch - 32ch
Linear 128ch + 128ch x 4 x 4
ResBlock 32ch + 64ch
ResBlock up 128ch + 64ch
ResBlock down 64ch 64ch
ResBlock up 64ch - 32ch
ResBlock 64ch + 128ch
ResBlock 32ch 16ch
ResBlock down 128ch + 128ch
ResBlock up 16ch 16ch
ResBlock 128ch + 128ch
ResBlock 16ch -> 16ch
ResBlock down 128ch 128ch
ResBlock up 16ch + 16ch
IN, Swish, Linear 128ch x
128ch
ResBlock 16ch + 16ch
IN, Swish, Linear 128ch 128ch
IN. Swish, Cony 16ch + 3 tanh
IN, Swish, Linear 128ch > 128ch x 2
M(u E R 128
R 128))
(b) Decoder
(a) Encoder
25

---

Under review as a conference paper at ICLR 2021
H MODEL ALGORITHM
Algorithm D presents the steps needed for DiVE to generate explanations for a given ML model
using a sample input image.
Algorithm 1: Generating Explanations
Input Sample image ML model f(-)
Output Generated Conterfactuals
Initialize the perturbations matrix parameter of size n x d
Σ &- randd(f 0, 0.01)
Get the original outpul from the ML model y f(x)
Extract the latent features of the original input + (6(x)
Obtain fisher information on z
f= + Fz)
Obtain k partitions using spectral clustering
10 P x SppactrlClusteeiing(33.
11 Initialize counter
i =-0
13
while i < do
14
for each e,P () do
Perturb the latent features
16
t (z + c)
Pass the perturbed image through the ML model ÿ + f(x)
18
Learn to reconstruct Ý from Y
19
+ compute Eq.
Update while masking subset of the gradients ac
23
end
Update counter
25
i i +
26
end
26

---

Under review as a conference paper at ICLR 2021
1
OUT-OF-DISTRIBUTIIO EXPERIMENT
n
h
6
a
6
t
C
d
W
2
m
I
g
d
e
i
y
L
ā
y
C
9
u
b
a
f
i
0
C
K
E
R
0
X
9
é
y
1
4
1 p j
a
9
r
Z
C
N
n
9
K
û
g
f
h
q
ú
P
à
h
p
b
u
O
ñ
6
Figure 8: Sample of the synbols dataset.
We test the out-of-distribution (OOD) performance of DiVE with the Synbols dataset [3]] Syn- bols is an image generator with characters from the Unicode standard and the wide range of artistic fonts provided by the open font community. This provides us to better control on the features present in each set when compared to CelebA. We generate 100K black and white of 32x32 images from 48 characters the latin alphabet and more than IK fonts (Figure B). In order to train the VAE ing 32 training characters: {a, b, d e, E, g, i, 3, L, n, P, q, E, t, Y, 2, à, á, å, å, é, é, é, i, 8, 8, ú, dj. Counterfactuals are then generated for the remaining 16 characters: (c, b, k, or 3, U,
different disjoint character set, randomly select the follow-
x, a, 3, i, I, 6, 8, B..
i. i
1
i j
IILU L 00
X
u 0
h W Aabpa
0
0.0
Figure 9: Successful counterfactuals for four different synbols in the OOD regime. Each sample con- sists of the original image in bigger size, five different counterfactuals generated by DiV EishertSspeerr and the difference in pixel space with respect to the original image (gray background). The header in each sample indicates the target class, e.g., I, u, All the counterfactuals predicted by the ML model as belonging to the target class and differ from the oracle (non-trivial).
DiVE's objective is discover biases on the ML model and the data. Thus, we use the font attribute in order to bias each of the characters on small disjoint subsets of fonts. Font subsets are chosen
so that they are visually similar. order to assess their similarity, we train ResNet12 [40] to
classify the fonts of the 100K images and calculate similarity in embedding space. Concretely, use K-Means to obtain 16 clusters which are associated with each of the 16 characters used for counterfactual generation. The font assignments random counterfactuals are displayed in Figure 9 DiVeeesterppeetall successfuly confuses the ML model without changing the oracle prediction, revealing biases of the ML model.
reported in Table 7 Results for four different
27

---

Under review as a conference paper at ICLR 2021
Anonymous Pro, Archilects Daughter, BioRhyme Expanded. Brumo Act, Bruno Ace SC, Caniarell, Eligible, Eligible Suns, Futurespure, Happy Monkey, Levend Exa. Lexend
Mega. Lesend Tera, Lesend Zetia, Michnoma, Orbilnon. Questril, Resalia, Siimi Ulna Espunded Syncopate, Topwka, Turne Road
b Acis, Alassi, Alfa Slab One, Amsranth, Archivo Black, Atno, Balno Blai, Roves, Black Dys One, Bowlly One SC, Bubklegum Sans, Bungee. Bungee Inline. Callstngs Candal, Cantanatte, Carter Ose, Ceviche Ose, Changa Onn. Chan Philomese One, Chela One. Chivo, Concert One. Cookbook Tite, Days One, Enca One, Francous One, Fredaka Ose, Goedine, Gurajada. Halt, Hauning Spirits, Holtwoed
Fegiz One, Galinds,
SC, Imperial One. Jaldi, Jnckey Dnc, Jomberin, Joti Kanit, Kaveon, Keanin Kenia, Lilezar, Lemos, Lilin One, Loudrina Solid, Luckiest Cuyy. Mir, Monoton, Palanquin Dark, Paeseso One, Passion, Passion One. Patus Oue, Paytone One. Peace Sans, PoetsesOnne, Power, Pridi, Pranted Circuit Board, Rakkas, Kammetio One. Kunchers, Kowdies, Rubik Mona One, Kubik Une. Russo One, Saira Steneil One, Secular One. Seymour
One. Signor One, Skranji Squada One, Suez Chnc Teko, USSR STENCH, Ultra, Uniy Thfling Unlock, Viafom, Wattashhimms simplicity
Delius Unicase, Didact Gothic, Daru Sans, Dustismo,, Fauna One, Feronin, Plamenco. Gadaza, Glass Antiuu Gothic AI, Gons, Handize, IBM Flex Mono Light. IBM Plex Sans Condensed Light, IBM Plex Sans Light, Indic Flower. Jusefin Sans Sul, Kite One, Munjari, Metrophobic, Miriam Libre, News Cycke Offiide, Overlock, Overlock SC. Panefrescco 250wt, Pavanam, Pontane Sans, Post, No Bills Culumbo Medium, Puniaan Quatinocent Suns. Quicksand. Roda, Scupe One, Snippet. Sulpbar Puint, Tajawcal Light. Test Me One, Thabit, Unkemps, Varta, WeblySleek UI,
Abel, Anabeim, Antic Antic Slab, Armala, Barrectio, Blavoka, Buda Chilanka, Comforaas, Comic Neue, Cative Cuiiee Mons,
Yaldevi Colombo Light
Ablaya Libre Medium, Abyssinic SII,, Adamina, Aliet, Alike, Almenda SC, Amelhysn, Andada, Aquifer Ampey, Asir, Average, Rackervvilke, Brwwer, Canbo, Della Respira. Donegal One, Estebun, Farwood Text, Femis, Fjuni, GFS Didot, Cabricla Goady Bookletter 1911. Hubibs, HeadlamiOne, IM FELL French Canun, IM FELL French Canon SC. Iniks, Inmalhari, Karma, Knon, Komle. Lanceler, Lolges, Linden Linux Liertinee Display, Lasris. Mate. Mate Metamorphon, Millogo, Montaga, New Alhasa Unicods, OFI. Sonts Mill Goudy TT, Ovo, PT Serif Caption, Petonn, Poly, Prociono, Quando, Rosarivo, Sowarabi Mischo, Sodan, Sedan SC, Thean Didor, Theuno Modam, Themm Old Style, Trocchi. Trykker, Uchen
Abri Faracee Almendra, Atburus, Asset, Audhowide. Cnndeend,, Bi CodosoCoddeeeeiiaeee Bigshut One, Bluzium Buwiby One, Branerr Flane, Breken Glass, Bungee Outline, Bungee Shade, Butcherman, CAT Kaster, Cabin Sketch, Catenary Stamp, Chango, Charnonntonn Chenyy Bomb, Cherry Cream Soda, Chonburi, Cadystar Coiny, Corhen, Coustand, Creepize CriminalHaadd DMN Schablonicsshrift Crackdd Devonshire, Diplociata. Diplomata SC, Dr Sugiyana, DanvescaMan. Eater, Elsic, Emblema One, Faster One, Flavors Fonidiner Swanky. Fredeticka the Great, Frijole. Fruktur, Geostar, Goostur Fill. Goblin One. Gravitas One, Hanales Fill, Inab Graver, Kabinett Fraktur, Kitibeh, Koowave, Leckerli One. Lily Scripe One, Linelight, Linux Bialinam Outline, Lisux Bialinam Sladow, Lacies Schoenschrfty CAT. Meubra, Metal Masia, Milionian Tattoo, Modak, Mollc, Mortified Dtip. Mrs Sheppards, Multvaa, New Rocker, Niconne, Nouteer Netalie, Utco Script. Open Faam. Overhaul, Paper Cuts 2. Peralta, Piedza, Plaster, Poller One, Porter Sans Block. Press Start 2P Purple Purse. Racing Suns One. Remix, Ringling. Risalty. Rushan Display, Rye, Smil, Sanidana. Sarina, Sarpanch. Schulze Werbekrafl, Sentic Que, Sertzfar, Spicy Rice. Stalin One, SideeaakNo2, Sat Dried, Tangerine, Thickhend, Tipps, Titan One. Tomorroww, Trade Winds, VT323, Vampiro One. Vist Shadow, Video, Vipand Angular, Wallpoet, Yesteryear. Zilla Slab Highlight
Severely, Shojumaru, Strikhand, Slackey, Smokum,
u
Alel, Alegreya Sans, Almara, Archivo, Anmo, Aneual, Arya, Assistant, Asul, Averia Libre, Averia Sans Libre, Be Viremam, Bellema Cambay. Comme, Cousine, DM Sans, Darker Grotesque, DejeMs Sans Mono, Dhyuns, Expletus Suns, Hack, lstok Web, Kollo, Krub, Lato. Liberation Menu. Liberation Sans, Libre Frankinn Lina Suns, Marcellus, Murtel Sans, Meriwesther Sans, Monda, Mplus 1p. Makia, Mall, Nicamii, Noto Suns Open Suns, Open Sans Hebrew, PT Sans, PT Sans Captice. Raieway, Ramble, Roboto Monu, Rosario, Rounded Mplus Sancation, Saraun, Surala. Scoda, Sintony Taiawal, Tenor Sans, Voces, Yontramanay
Ales Yoth, Antar, Avena Graesa Libre, Biice Bhai 2, Bromuse Bonertly Kids, Caesar Dressing. Cagluostro, Capriola, Coreal, Caveat Brush, Cheises Market, Chewy, Class Coder, Convincing Pintte Crese, Crumicproductivve Comggraee Courier Prime, Covered Ry Your Grace. Damion, Dear Old Dad. Dekko, Damestic Manners, Dosit, Fricz Type, Frika Onaig, Espresso Doke. Farsan, Finger Paint, Frecile Face. Packin Gwesbwyfar Gloeia Hallelujah, Gochi Hand. Grand Hotel, Halagen, FELL DW Pica, FELL DW Again Down Here. Kolam, Kodchasan, Lacquer, Lorem lpsum, Love Ya Like A Sister, Macondo, Mali, Mansalva, Margarine Marmelad, Matins, Megr, Moriffed, Nunito (ljectiie. Oldenhorg Oreguno, Paeifion, Pangolin, Patrick Hand, Patrick Hand SC, Pocin,, Pannforma, Pocspiere, Rancho, Recnic Beasie. Rock Solt Ruge Boogle, Sacramento, Salso, Schooleell, Sedgnick Ave, Sadgwick Ave Display, Stadows lnto Light, Short Stack, Sirinttencll Sofadi One, Salway, Special Elite. Srinchh, Stalemate. Sue Ellen Francisos, Sunshincy, Supercomputer, Supermeraads, Swanky and Moo Moo, Sweet Spors, Varela Round, Vibur,
sc, FELL English SC, Ita, Juntor, Junior CAT. Just Another Hand,
Wilter Tumeoat, Warnes, Wellffees Yellownil, Zoyada
W
Alsa, Ali DIN 1451 Mitteschril Alic DIN 1451 Mitelschift geprang, Ambl, Arvo, Asup, Asp VP Bein, Alhiii Atomic Age. B812, 5812 Mono, Burkow, Barlow Semn Condensed, Basic. Blinker, Bree Scrif, Cairo, Cello Sans, Chakra Posch. Changa Medium, Cherry Swash. Crete Round, DejaVu Suns, Dogpie One. Droid Sans, Eliine Sans, Encode Sans, Emuode Sans Condensed, Encude Sans Expanded, Encude Sans Semi Condensed. Encode Sans Wide, Exo. Eso Federu, Fin
Fira Sans Condensed, Gonserrrt. HummenerihOne, Hand Drown Shapes. Harmatiam, Heebu, Hepts Shb, Hind, Hind Kochi, IBM Ples Mono. IBM Plex Mono Medium. IBM Ples Sans, I8M Plet Suns Condensed, loeland. Josefin Sans, Krona One, Livic, Mada, Magra, Maven Pro, Maven Pro VF Beta, Mizza Monserrat Monteeraa Altemases, Myanmar Khyay, NATS. Nobile. Oxanium, Play, Poppins, Prompt, Prosto One, Prosa Libre, Quantico. Red Hat Test, Reem Kuli. Renner" Righteous, Saira. Samn SemiCondensed, Semi, Stare. Signika, Soniano Sans Unicode, Source Code Pro, Source Sans Spartan, Viga. Yaaa One. Zilla Slab
Abhaya 1 libre Amita, Astic Didone, Anef Rugio, Arima Madurai, Rollefair, CAT Childs CAT 1anz, Cardo, Candes, Ginzel, Cormonat, Comorant Garamond Cemmanan SC, Commatant Unicase, Cormorast Upsight, Cuprum. Domine Dustismo Roman, EI Messii, Fahkwaag. FogrwoNas, Forum, Galatio SIL,, Gayathe. Gilde Dispay, Glegon. GlokMiser. Gpinks, Griffy, Gupler, Plex Serif Light. Taris Strff, Ialiana, Julson, Junge Libre Caslon Displa,
Biolinum Capitals, Linux Biolinum Slanied, Lihestine Capisals, Labster Two, Marcellus SC, Marte Meriendo, Modem Antqus Moetserra Sohmyada. Mountains of Chisstuss, Mystery Quest, Old Standand TT. Playfair Disple, Playfiir Disply SC. Portmunicau Proto. Proteel Prida66, Quattracemoo Resagnicio Risque, Rulina, Speciaal SC, Trirony, Viauda Libee, Wes, kawaszeh, okolaks
Accurtist. Advent Pro, Archive Narrow Aubrey. Cahin Condensed, Convergeace, Enccde Sans Compeessed, Firro, Pira Suis Extra Condensed, Gildeano, Gemanu Libre, Gemunu Libre Light, Geo. Codes, Homenaje, loeberg. Liberty Sans, Mohave. Nova Nova Flat, Nova Oval, Nova Round, Nosa Slim, NovaMono, Open Sans Condensed, Open Sans Heheew Condensed, PT Sans Narrow Port Ligat Sans, Port Liigat Slab, Pragan Narrow, Kajdhani, Rationale, Rabaso Condensed, Ropa Sans, Stim Condensed, Saim ondraeooneeeed, Share Tech, Share Tech Mono. Serait Strong. Taun. Ubuntu Condensed. Voltaare Yaldowi Colombo. Yaldevi Colembo Medium
Aladia, Alegre Sans, Allan, Amammee Anton, Antonio, Asap Condensed. Astloch, Sign, Bod Seriet Bahina, Bahianita Bangers, Barloestus Schait. Barlow Condensed, Barrio Bebus Nour, BenchNine. Berlin Email Strif, Berlin Emiil Serif Shadow, Berolina, Benholdr Maincer Frikiur, Bindemsier Kursiv, Big Shoulden Display, Big Shoulders Text, Bigelow Rules, Binioo IVE. Beosoo, Bangaloo, CAT FrankeaDesschh CAT liching Goiishh, Calligraserif, Cisa Sans, Chicle, Combo. Contrall One. Crshed. DN Tiling, Dagerotypos, Denk One. Diginal Numbers, Dorsa Economica, Eleventh Square, Engagement, Exphoria Script. Ewert, Fetle Mikado, Fallaa One. Flubyy. Friedolin, Galada, Germania One, Gianna, Grddatee Hanaki, heqqes Francois Shadow, Gutisch, Jolly Lodger. Jake, Kanzler Kavisanar, Kazmann Sans, Kelly Slab, Kland, Kminky Lemonada. Loved by the King. Maides Drange, Marck Seripe. ModicvalStrp. Modula One, Meriends One, Millonian, Mouse Memains Nova Seripe, Odticee Sans, Oswald, Paprika, Pabway Gothic One, Penguin Aluck, Pirata One. Pommern Gotishh. Post No Bills Colombo Princess Sulia, Redressed, Ribeye Martow, Rum Raisin, Sancreek, Saniretro, Sanituchtro, Sevillann, Six Caps, Slim Jim. Smythe, Solia, Sportop Stafliches, Stint Ulna Condensed, Tillaaa Tulpen One, Undemlog, Unica One, UnifrkkurMagnnii
One,
Yonone Kaffeesatz
Amiric SC: Bellota, Bellota Test, Bernando Moda, Bloklemers Balpes, Blokletias Poticed, Bubbler One. Bungee Harline, Coming Soon, Cratty Girls, Gessnu Libre Extral.ighn,
Give You Glory, Gold Plated, Gruppo, 1BM Piex Mono ExtraLigtt IBM Plex Mono Thin, IBM Plex Sans Condensed EstraLight, IBM Piet Sans Condensed Thin, IBM Plex, Serif Onc. Lazeiby Computer, Life Savers, Londrita Outline. Londinn Shadow, Mada Extnal Mada Light. Major Mono Display, Megrim, Nixe One, Noethanpoos, Over the Rainbow, Panefresco Iwt, Poiet One, Post Bills Colombbe Light, Raleway Den. KavenguikSans, Sansation Light. Shadows Inio
Estralight, IBM Plex Serif Thin, Julins
Light Two, Siem Nevada Read, Slimanif, Snowburst One, Trjinaal Extal Tominal Dosis, Thisodith, The Gid Nost Dooe, This Pencil Hardwriting Viles, Waiting for the
Sunrise, Wire One. Yaldevi Colanioo ExtraLighe
Draid Seri. Gammeral Eagle Lake, Fondamento, Frank Ruhl Libes, Golasio, Gentium Bisic, Gentium Book Busic, Ibarra Real Nova, Junicode, Liberation Serif, Libre Baskerville, Libre Caslon Text, Linux Biolinum, Linus Libetiine Linus Libertine Shatte, Lusitana,
Amiri, Buenarl, Calades, Charis SII, Crimson VejaVa Suiif, Dis
Manuale, Merriwealser, Nobeiia Test, PT Serif, Scheheraznde, Spectral, Taving Unna. Vesper Devanagani Libr
Aldrich, Alegona Sons SC, Alco, Amiko, Andika, Annic Use Your Telescope, Average Sans, Rai Jinjures, Rramanns Belgano, Rinkhyme, Biryani, Cabin, Cabin VF Beta, Calling Code. Carme, Canois Gothic, Carooia Gothie Catamarnn, Changs Light, Convincing Droid Sans Mano, Electrlizee Eaglehert Fresca, Neahel- lenic, Imprimaa Inconsolaaa, Indes, Inna Sans, Josetin Slab, K2D. Kada, Kulim Park, Leizon. Lexend Deca, Mako, MeLarea, Meem Inimal, Molesgo, Numans Orienta, Overpass, Overpass Mens. Osygen Mono, PT Mono, Panefresco 400but. Podkova, Podiooa VF Beta, Red Hai Display Rhodium L.Bve, Ruluko, Sunchez, Sani Trisie, Saunnihi Gothic, Sen. Shami, Slabo 1ps, Slabo 27ps. Sometype Mono Spece Mono, Spineaker, Teffy: Tuffylufam, Turly Scrie, Ulcan Mono, Varele
ARueZon,
Aguatina Scnpt, Akreni, Alex Beush, Anzona, Heth Ellen, Bilbo, Branspuulver. C Callgratimm Cedavvile Cunsive, Charm, Clicker Scapt, Condiment. Cookue, Dancing Script, Dawning of a New Day. Dynalight, Felipa. Great Vibes, Herr Von Muelemed Homemadee Apple. Italianoo Jim Nighsshade. Kanshan Serigt. Kristi, La Belle Aurore. League Scripi, Meddon, Meis Stript, Mervale Script, Miuma, Miniver, Miss Fojandss, Monsieur La Doulaise, Monier, Mr Dedfor, Mr Duloe, Mr De Haviland, Mrs Saint Delafiell Norican Nottang You Could Do. Parisieane, Pelt Formal Scnpt, Playon Script. Playball, Promocyya Quintessemial, Qwigley, Rochester, Romanesco, Rouge Scrige, Rutlie, Salisyy, Seaweed Script, Srisakdi, Vengeance
Aclonics, Alegreya, Alegnya SC, Ambula SC, Aritika, Averia Senf Libr, Balihwar, Benthum. Bister. Caniata One. Chimson Pro, Croisaant One, DM Serif Disphe, Eezan,
Finlly Candy, Firiquesa Fussina, Federant, Ginesol, Grenze, Halant Henny Penny, Hommeneus IBM Ples Senif, IRM Ples Serif Medium. IM FRLI Douisle Pica, IM
FELL. Deuble Pica SC, FELL Eaglish, FELL Gieat Primer SC, Inkaut Amiqus. lncques Francois, Indges, Kameron. Laila, Lateef, Literta, Locs. Maltree, Markwil Text, Marko One, Monteiro Lubato, Original Surfer, Pricopatulogin de la Vida Cotidiama, Radley, Rusi, Riboye. Roboto Slab, Rokkin, Rocha One, Sahitya, Sansiaa Simonella, Source Serif Pro, Spirax, Sunios Stencil, Stoke, Sumina, Uncial Anriqua Vidaioka, Volkhow, Valloorn, Vallkam SC
Table 7: Font clusters assigned to each character.
28

---

Under review as a conference paper at ICLR 2021
J DETAILS ON THE BIAS DETECTION METRIC
In Table we follow the procedure in first developped in [26] and adapted [33 and report confounding metric for bias detection. Namely, the "Male" and "Female" is the accuracy of the oracle on those class conditioned on the target label of the original image. For example, we can see that the generated explanations for the the biased classifier, most methods generated an higher amount of Non-smiling females and smiling males, which was expected. The confounding metric, denoted as overall, the fraction of generated explanations for which the gender was changed with respect to the original image. It thus reflect the magnitude of the the bias as approximated by the explainers. Singla et [53] consider that model is better than another if the confounding metric the highest on foinsed and the lowest funbiased-
This is however not entirely true. There is no guarantee that foiased will perfectly latch on the spurious correlation. In that case, an explainer's ratio could potentially be too high which would reflect an overestimation of the bias. We thus need to a way to quantify the gender bias in each model. To do so, we look at the difference between the classifiers accuracy "Smiling" when the image is of a "Male" versus a "Female". Intuitively, the magnitude of this difference approximates how much the classifier latched the "Male" attribute make its smiling predictions. We compute the same metric for in the non-smiling case. We average both of them, which we refer as ground truth in Table As expected, this value is high for the fuiased and low for funbiased. Formally, the ground truth is computed as
Ea-p(a) (egrp(z || - f())a - ail 1y f(z)aa - azll]
(10)
where a represents the attribute, in this case the gender.
29


Reviews:
[{"review_id": "u-dDvLckjqK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Reasons for score\n\nOverall, I really liked your proposed method and would appreciate seeing your paper published. However, as of now, it does not pass the acceptance threshold. If you address my questions and requests, I would be willing to change my score.\nI think that it is critical that you improve the experimental section in terms of writing and presentation of the experimental protocol, metrics, results and especially section 4.3. Also, I think that it is necessary that you improve the related work section and highlight the limitations of your method.\n\n\n## My background\n\nMy research is focused on detecting data biases (or spurious correlations) learned by deep neural networks using explainability methods. This is the exact scope of this paper. However, although I have a solid understanding of the attribution methods, this paper develops a counterfactual one which is related to but not directly within my area of expertise.\n\n## Summary\n\nContext:\nThe paper focuses on counterfactual explainability methods that aim at improving the reliability of machine-learning systems and help for model debugging (finding spurious correlations or model biases). Given an input example and a target prediction score, this class of methods generates counterfactual examples.\n\nProblem:\nThe authors identify and tackle issues of state-of-the-art methods, xGEM[23] and PE[41]:\n- they combine multiple biases of the model in a single counterfactual example (not disentangled),\n- they exaggerate or remove the presence of the attribute being classified (trivial and not valuable).\n\nSolution and novelty:\nThe proposed method generates $n$ counterfactual examples for a given input example and a target prediction score. It is composed of :\n- a pretrained encoder-decoder architecture with $\\beta$-TCVAE[5] that produces a disentangled vectorial representation of the input example,\n- an algorithm that produces $n$ perturbations of this representation that are decoded to generate counterfactual examples.\nThe perturbations are obtained by minimizing a loss composed of:\n- a binary cross-entropy loss to generate examples that match the target prediction score,\n- an L1 loss between the input example and each generated example to force small perturbations in input space,\n- an L1 loss on each perturbation to force small perturbations in latent space,\n- a structural mechanism based on the Fisher information matrix and spectral clustering to force diversity of perturbations\n\nClaim:\nThe proposed method generates counterfactuals that are diverse, non-trivial (i.e. not just exaggerate or remove an attribute), high quality (i.e. in distribution) and valuable explanations about the model's prediction (i.e. biases can be detected by humans).\n\nExperiments:\nThe method reaches state-of-the-art results on two existing benchmarks. The paper also introduces a new benchmark to evaluate how valuable the explanations are.\n\n## What I liked the most\n\n- meta-problem of explaining neural networks is critical\n- mostly well contextualized\n- mostly easy to read\n- mostly easy to understand\n- mostly well illustrated\n- relevant issues have been identified\n- novel, simple and interesting method to tackle them\n- novel experimental benchmark\n- I really liked the hat section of 4. Experimental Results where you describe the 3 different aspects that you evaluate and you point to the associated sections\n- improvements over state-of-the-art are significant\n- ablation study mostly validates each proposed components\n\n\n## What could be improved\n\nMy cons are expressed per section, but the listing is random (I did not write the critical cons at the top of each section).\n\n1. Abstract and introduction\n- I find it surprising to assume derivability for a black box in the context of explainability methods. I know PE[41] uses this definition, but at least they clearly state that they assume derivability. For me, it does not correspond to the commonly admitted definition of a black box (e.g. Wikipedia: \"implementation is opaque\" \"without any knowledge of its internal workings\"). See the influential Ribeiro et al. 2016 \"Explaining the predictions of any classifier\" (4000 citations) for a definition that implies no assumption about derivability (i.e. \"model-agnostic\").\n- I had to write a detailed summary of your paper to better understand it. It is not critical, but I think that your writing can be further improved/structured to better frame the issues of previous state-of-the-art that you tackle, the novelty of your contributions (for instance, is it novel to structure diversity with FisherMatrix and spectral clustering?) and the claims that you validate in the experiments.\n\n\n2. Related work\n- First of all, note that reading the extended related work in the supplementary material (which is not mandatory) did not address the points I am going to make. I am convinced that your related work section could be improved as follows: 1) I would briefly mention that many post-hoc explainability approaches to detect biases exist and that you focus on counterfactual ones. 2) I would write about counterfactual methods that are not generative (see Papernot et al. 2018 \"Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning\" which may be related). 3) I would write about state-of-the-art generative counterfactual methods and their limitations. 4) I would write about explainability methods that focus on diversity (you should mention that only a few exist if it is the case). \n- I do not agree with \"[attribution methods] do not explain how to modify [input features] to change the model outcome\". I think that is exactly what they do (by applying perturbations/masks). See Fong et al. 2017 \"Interpretable explanations of black boxes by meaningful perturbation\" that you already cite. However counterfactual generative methods of your kind apply perturbations that look real from a human standpoint or \"in-distribution\". I would like to see a discussion about the need for this kind of perturbations.\n\n3. Proposed Method\n- From a first read, it is not easy to map Dive, DiveFisher and DiveFisherSpactral onto their definition (around Eq. 7 and 8). Paragraphs Diversity loss and Beyond trivial counterfactual explanations could be better structured in this respect.\n- Figure1: You should make it clear that counterfactuals at the top are non-valuable to detect biases (because considered not bald by humans and by the model), whereas the bottom are valuable to detect biases (because considered bald by humans but not by the model).\n- Figure1: I do not understand why the bottom right has a black border.\n\n4. Experimental results\n- Overall, I do not find it clear what your training, validation and testing sets are made of. Do you use a testing set? How do you tune your hyperparameters? Is it standard?\n4.1\n- What does FID stand for? \n- I do not understand this: \"we train a second-order spline on the trajectory of perturbations produced during the gradient descent steps of our method\"\n- I think I understand, but it could be more clearly stated: \"even though DiVe is not explicitly trained to produce examples at intermediate target probabilities\". Actually, you can choose the target probability, isn't it?\n4.2\n- Table2: metric is not included in the caption\n- Table2: I do not understand the reason why some numbers are bold\n4.3\n- Overall I think it is critical to improving this section. I did not clearly understand how your novel experimental protocol can be used to validate \"the ability to identify diverse valuable explanations\".\n-\"Because it is costly to provide a human evaluation of an automatic benchmark, we approximate both the proximity and the real class with the VGGFace2-based oracle.\" Why do you think that we can trust the approximation?\n- Why don't you compare against PE and xGEM even tho it's not the same decoder as xGEM+ and DiVE?\n- I would make it clear what \"success rate\" means (you could make it bold where you define it)\n- Figure3: Overall, I should be able to mostly understand the Figure by reading the caption. It is not the case at all. I did not find it clear that you explore different hyperparameters.\n- \"We show results for all explanations in Figure 3a and only when the generated images are counterfactuals in Figure 3b.\" It is not clear. From what I understood your method generates counterfactuals only! Some of them are just valuable to detect biases (i.e. misclassified by humans) or not. What does \"successful counterfactuals\" mean?\n- It would be interesting to have a baseline with random masks to compare against your proposed method (Fisher and FisherSpectral).\n- There are no experiments on Dive-Fisher or Dive-FisherSpectral except for Figure 3 (and even for Figure 3. You don’t explain the difference between results for the different versions of Dive). I did not understand which version of your method works best. Also, you should define Dive-F and Dive-FS or not use them. I find that using Dive_Fisher and Dive_FisherSpectral improves clarity.\n\n5. Conclusion section\n- I would like to see a discussion about the limitation of your method and issues that need to be addressed in future works. In particular, what happens if your encoder-decoder is biased? What happens when your encoder-decoder is not able to produce disentangled representations (it seems often the case on real and complex datasets)? How to calibrate the many hyperparameters (to train the encoder-decoder and to weigh your loss functions)?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q6pPx7-CYF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n\nIn this paper, the authors present a method based on counterfactuals that learns a perturbation using constraints to ensure diversity in explanations.\n\nThe authors argue that explanations produced by their method are more “actionable, diverse, valuable and proximal than the previous literature”. However,\nit is unclear how they quantitatively measure these attributes, given that FID scores only captures the similarity of generated images to real ones.\n\nI would like to understand the motivation on using the perceptual reconstruction loss. The authors should clarify the usage of this loss in their method and\nhighlight its importance on their explanatory method. The author briefly mentioned the gains in terms of image quality, when compared with GANs in PE.\nHowever, I would like to see a more deeper discussion. \n\nSince interpretability is closely related to users/humans, it is difficult to assess the quality of the generated explanations without human evaluations.\nAn initial setup could be the one used in PE.\n\nOverall, assuming the above limitations, the experiments help to understand the contributions of the article.\n\nTypos:\n \n- Sec. 3.3: “Since these mask are …” -> “Since these masks are…” \n- Sec. 4.2: “In Figure 2b, …” -> “In Figure 2, …”", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q4nCq0JATVc", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors propose interpreting the decision of a black-box (BB) image classifier using diverse counterfactual explanations. The proposed model consists of a pre-trained β-TCVAE, which learns to extract a disentangled latent representation for the input image. To generate explanations for a given image, the model optimizes to find n latent perturbations. Each decoded output from β-TCVAE is similar to the original image and produces a desired outcome from the BB classifier. To ensure the diversity among the n latent perturbations, the model minimizes the pairwise similarity loss between the latent perturbations. The model further performs spectral clustering to partition the latent space into different attributes. Thus, at inference time, for the same input image, multiple counterfactual images can be generated as explanations by changing different dimensions of the latent space. The experiments demonstrate the realistic quality of the explanations and their ability to discover bias in the BB classifier.  \n\n\n•\tThe idea of generating multiple images as counterfactual explanations is interesting. If multiple explanations differ in multiple attributes, it can help identify un-wanted correlation or biases in the model and datasets. \n•\tThe paper lacks detailed experiments to quantify the importance of diverse explanations. The experiment should quantify what attributes, apart from trivial counterfactual changes, differ across the explanation images. The successful explanation experiment confirms differences in the explanation images, but ``what\" is different is not apparent.\n•\tThe author's definition of a valuable explanation is misleading. In the introduction, the authors describe a valuable explanation as an explanation that is proximal, i.e., it is much similar to the input image, and actionable i.e., it can be derived by performing feasible changes to the input image. In the experiment section on \"Beyond trivial explanations,\" authors defined a valuable explanation as the one for which the BB model and human or its proxy (an oracle network) have different outcomes. For the two definitions to be consistent, the authors assume that the oracle network can uniquely identify feasible features in an image and consider such features in its classification decision. \n•\tFeasible features in an image are hard to define. For example, adding/removing sunglasses is a trivial example of a feasible change. While changing some pixels around the face's lips, to add/remove the smile is a more complex change whose feasibility is hard to define. As the change in the region around the lips may add/remove an expression from a face that is hard to quantify. Also, the authors didn't perform any experiments to show that the explanations generated by their method correspond to feasible changes in the input image, in contrast to other methods like PE or xGEM. Since all the methods involve a generative process, which is prone to perform un-realistic changes to the image, the author's claim of restricting perturbation to only feasible changes is vague. \n•\tIt is not clear what training data is being used to train the encoder-decoder in the proposed model. If the method uses a different dataset from the training dataset of the BB classifier, please state that explicitly. Also, to compare against the existing methods, the authors can design an experiment where they consider a dataset (explain-dataset) different from the dataset used for training the BB classifier (BB-dataset). They can then use the explain-dataset to train and compare the different explanation models (DiVE, PE, xGEM). \n•\tThe authors consider a perceptual reconstruction loss instead of a standard pixel-wise reconstruction. An ablation study to compare the different reconstruction losses is required to justify the proposed model's additional dependency on a pre-trained network R.  \n•\tThe term \"adversarial loss\" is misleading. The adversarial loss defined in equation 5 constrains the model to learn a perturbation that results in the desired probabilistic outcome. Hence, the name should reflect this constraint and its dependence on the BB classifier. As mentioned in the text, there are no adversaries here; hence no min-max game to be solved.  \n•\tThe authors claim that the sparsity constraint on the latent perturbation results in proximal and actionable explanations. The explanations' actionability is defined in terms of sparsity in the number of attributes that are modified in the latent perturbation. Since β-TCVAE is trained in an unsupervised manner, the latent space is not explicitly disentangled in measurable attributes (e.g. presence of sunglasses). Hence, the disentangled attributes learned by β-TCVAE, and discovered by the spectral clustering, may not correspond to discrete human-understandable concepts (e.g., sunglasses). An actionable explanation enables the human end-users to modify discrete concepts (e.g., remove sunglasses from the face) and observe changes in the BB's behavior. Experiments are required to quantify the actionability of the explanations. \n•\tA replication of the \"Beyond trivial explanations\" experiment on real images can also demonstrate the disagreement between the prediction from the BB classifier and the oracle classifier. It's not clear how experimenting on the counterfactual images provided any more/different information than the same experiment performed on real images.\n•\tA valid counterfactual image should produce an opposite prediction from the BB classifier compared to the input image There are no experiments to quantify the validity of counterfactual explanations.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7N1X9q6gTDT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The present work proposes an explanation method returning actionable, proximal, diverse, and not trivial counterexamples as explanation. The work is well written even though various concepts are detailed only in the Appendix.\nThe proposal is interesting, sound in the formulation, and valuable from the experiments reported. The examples reported are nice and quite convincing. The bias detection case study is effective and well presented.\nHowever, the paper lacks some major points that make it not ready for publication.\nFirst, even though theoretically the proposed DIVE method can be employed on any type of data it is developed and tested only on image data. The future work discussion is missing and the possibility to employ it on other data types is not treated. The fact that experiments are reported for a unique dataset is a limitation. Various simple datasets with more simple features can be adopted (mnist, cifar10, fashion mnist) or cifar100 imagenet by considering categories and specific classes for the different features.\nSecond, the paper misses various work on counterfactual explanations (some of them are listed in the following) and consequently also a comparison against them. In particular, it would be interesting to test DIVE against methods using a different logic for finding the counterexamples than against methods using a similar approach. Furthermore, in DICE (cited) are presented many evaluation measures not adopted in this paper and it is not justified why.\nFinally, the trivial/valuable explanations, which are the main motivation for this wor, are not formally defined in Section 4, nor in Section 5.\n\nMinor issues:\n- The optimization problem solved by DIVE to find the exemplars is not sufficiently detailed in the main paper.\n- It is not clear (or not easy to find) the dimension of the latent space of the VAE and how it affects the performance\n\nMissing Related\nKarimi, Amir-Hossein, et al. \"Model-agnostic counterfactual explanations for consequential decisions.\" International Conference on Artificial Intelligence and Statistics. 2020.\nPoyiadzi, Rafael, et al. \"FACE: feasible and actionable counterfactual explanations.\" Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 2020.\nGuidotti, R., Monreale, A., Matwin, S., & Pedreschi, D. (2019, September). Black Box Explanation by Learning Image Exemplars in the Latent Feature Space. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 189-205). Springer, Cham.\nPawelczyk, M., Broelemann, K., & Kasneci, G. (2020, April). Learning Model-Agnostic Counterfactual Explanations for Tabular Data. In Proceedings of The Web Conference 2020 (pp. 3126-3132).\nDhurandhar, A., Chen, P. Y., Luss, R., Tu, C. C., Ting, P., Shanmugam, K., & Das, P. (2018). Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems (pp. 592-603).\nVan Looveren, A., & Klaise, J. (2019). Interpretable counterfactual explanations guided by prototypes. arXiv preprint arXiv:1907.02584.\nGuidotti, R., Monreale, A., Giannotti, F., Pedreschi, D., Ruggieri, S., & Turini, F. (2019). Factual and counterfactual explanations for black box decision making. IEEE Intelligent Systems, 34(6), 14-23.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose to use counterfactual (a.k.a as contrastive, as they do not account for causal mechanism) explanations  to explain the errors of an already trained predictive model with images as input data. To this end the authors rely on the manipulation of the latent space of  a VAE with disentangled representations. In general the idea is simple (and based on approaches from prior works) and extends work on counterfactual explanations which have been broadly studied in other domains like decision making where the input data have often semantic meaning (in contrast with the pixel of an image).  While the technical contribution is quite limited, I believe that the general approach of the paper is interesting. 

However, even after the rebuttal and reading the updated version, it is still unclear what exactly means key concepts for the paper like trivial and actionable, and more importantly how to use the proposed approach in practice, beyond checking/correcting for potential gender bias in  the data (given that you have access to gender information). In particular, it is not clear how you measure "non-triviality" for the predictive task when you do not have additional knowledge (like the gender) or when the disentangled latent representation do not correspond to semantical features (which as far as I understand they do not need to).  Similarly, while actionability and diversity have been broadly discussed in the decision making domain, it is again not clear what an actionable feature means here to me. Is it just that you can perturb the latent space? 

Furthermore, I believe it is worth exploring the connection to approaches for adversarial examples. As it has already discussed in the literature, in terms of formulation,  counterfactual explanations resemble the problem of adversarial examples, but it seems substantially different semantically. At times when reading the paper, it feels that it is indeed more related to adversarial examples than to counterfactual explanations, as the explainability part seems quite superficial. Thus, I would encourage the authors to better position their paper. 

In summary, I believe that the paper requires further work before being ready for publication. In particular, the paper would significantly from: i) a better positioning of paper with respect to the literature; ii) formally introduce  key concepts like actionability, diversity and triviality, explaining what they mean in this context, and how to measure them; and more importantly, iii)  explaining how the proposed approach (which by the way involves training a generative model)  can be used in general to 'understand' a model. On a final note, I believe that the paper would benefit from from bringing back to the main body of the paper the experiments that were moved to the appendix during the rebuttal. 

Decision: Reject