Explanation  by Progressive  Exaggeration
Authors: Sumedha Singla
Brian Pollack
Junxiang Chen
Kayhan Batmanghelich
Venue:  (2020)
Keywords: Explain
deep learning
black box
GAN
counterfactual

Abstract: As machine learning methods see greater adoption and implementation in high stakes applications such as medical image diagnosis, the need for model interpretability and explanation has become more critical. Classical approaches that assess feature importance (eg saliency maps) do not explain how and why a particular region of an image is relevant to the prediction. We propose a method that explains the outcome of a classification black-box by gradually exaggerating the semantic effect of a given class. Given a query input to a classifier, our method produces a progressive set of plausible variations of that query, which gradually change the posterior probability from its original class to its negation. These counter-factually generated samples preserve features unrelated to the classification decision, such that a user can employ our method as a ``tuning knob'' to traverse a data manifold while crossing the decision boundary.  Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.

Published as a conference paper at ICLR 2020
EXPLANATION BY PROGRESSIVE EXAGGERATION
Sumedha Singla
Brian Pollack, Junxiang Chen Department of Biomedical Informatics University of Pittsburgh
Department of Computer Science University of Pittsburgh
Kayhan Batmanghelich
Department of Biomedical Informatics Department of Computer Science Intelligent Systems Program
University of Pittsburgh
ABSTRACT
As machine learning methods see greater adoption and implementation in high stakes applications such as medical image diagnosis, the need for model inter- pretability and explanation has become more critical. Classical approaches that assess feature importance (e.g., saliency maps) do not explain how and why a par- ticular region of an image is relevant to the prediction. We propose method that explains the outcome a classification black-box by gradually exaggerat- ing the semantic effect of given class. Given query input classifier, our method produces progressive set of plausible variations of that query, which gradually changes the posterior probability from its original class to negation. These counter- factually generated samples preserve features unrelated to the clas- sification decision, such that user can employ our method as "tuning knob" to traverse data manifold while crossing the decision boundary. Our method is model agnostic and only requires the output value and gradient of the predictor with respect to its input.
INTRODUCTION
With the explosive adoption of deep learning for real-world applications, explanation and model interpretability have received substantial attention from the research community (Kim, 2015; Doshi- Velez & Kim, 2017; Molnar, 2019; Guidotti al., 2019). Explaining an outcome of a model in high stake applications, such as medical diagnosis from radiology images, is of paramount importance to detect hidden biases in data (Cramer et al., 2018), evaluate the fairness of the model (Doshi- Velez & Kim, 2017), and build trust in the system (Glass et al., 2008). For example, consider evaluating a computer-aide aided diagnosis of Alzheimer's discase from medical images. The physician should be able to assess whether or not the model pays attention age-related disease-related variations in an image in order to trust the system. Given a query, our model provides an explanation that gradually exaggerates the semantic effect of one class, which equivalent traversing the decision boundary from side to another.
Although not always clear, there are subtle differences between interpretability and explana- tion (Turner, 2016). While the former mainly focuses on building or approximating models that are locally or globally interpretable (Ribeiro et al., 2016), the latter aims at explaining a predictor a- posteriori. The explanation approach does not compromise the prediction performance. However, a rigorous definition for what a good explanation is elusive. Some researchers focused on providing feature importance (e.g., in the form of a heatmap (Selvaraju et al., 2017)) that influence the outcome of the predictor. In some applications (e.g., diagnosis with medical images) the causal changes are spread out across a large number of features (i.e., large portions of the image are impacted by disease). Therefore, heatmap may not be informative or useful, as almost all image features are highlighted. Furthermore, those methods do not explain why predictor returns outcome. Others have introduced local occlusion or perturbations to the input (Zhou et al., 2014; Fong & Vedaldi, 2017) by assessing which manipulations have the largest impact on the predictors. There is also

---

Published as a conference paper at ICLR 2020
recent interest in generating counterfactual inputs that would change the black box classification de- cision with respect to the query inputs (Goyal et al., 2019; Liu et al., 2019). Local perturbations of query guaranteed to generate realistic plausible inputs, which diminishes the usefulness of the explanation, especially for end users (e.g., physicians). We argue that the explanation should depend not only the predictor function but also the data. Therefore, is reasonable to train model that learns from data as well as the black-box classifier (e.g., (Chang et al., 2019; Dabkowski & Gal, 2017; Fong & Vedaldi, 2017)).
Our proposed method falls into the local explanation paradigm. Our approach is model agnostic and only requires access the predictor values and its gradient with respect the imput. Given query input to black-box, we aim at explaining the outcome by providing plausible and progressive variations to the query that can result in a change to the output. The plausibility property ensures that perturbation is natural-looking. A user can employ our method as a "tuning knob" to progressively transform inputs, traverse the decision boundary from one side the other, and gain understanding about how the predictor makes decision. We introduce three principles for explanation function that can be used beyond our application interest. We evaluate our method on set of benchmarks well real medical imaging data. Our experiments show that the counterfactually generated samples are realistic- -ookiig and in the real medical application, satisfy the external evaluation. We also show that the method can be used to detect bias in training of the predictor.
METHOD
Consider black box classifier that maps an input space X' (e.g.. images) to an output space y (e.g., labels). In this paper, we consider binary classification problems where y = +1]. To model the black-box, we use fx) P(ylx) denote the posterior probability of the classification. We assume that f is differentiable function and we have access to its value as well as its gradient with respect to the input [x)...
f(x)
Heal/Fake
Ma
Input
X
f(x)
fixil
(KL( (a1-)
Flx
11 #l1
||e
#|1
Ma
B00))
(a)
(b)
Figure 1: (a) The schematic of the method: is the black-box function producing the posterior probability. 8 is the required change in black-box's output fxx). I, (x, 5) an explainer function for f. which shifts the value of f(x) by 8. The E(-) )) an encoder that maps the data manifold M2 to the embodding manifold M,, Xs is abbreviation for I, (x, 8). (b) The architecture of model: E is the encoder, G denotes the conditional generator G(, ey (z, 8)), f the black-box and D the discriminator. The cireles denote loss functions.
We view the (visual) explanation of the black-box as a generative process that produces an input for the black-box that slightly perturbs current prediction (f(x) + 6) while remaining plausible and realistic. By repeating this process towards each end of the binary classification spectrum, we can traverse the prediction space from one end to the other and exaggerate the underlying effect. We conceptualize the traversal from one side of the decision boundary to the other walking across data manifold, M,. We assume the walk has a fixed step size and each step of the walk makes change the posterior probability of the the classifier, f. Since the output of f is bounded between
2

---

Published as a conference paper at ICLR 2020
[0, 1|, we can take at-most | steps. Each positive (negative) step increases (decreases) the posterior probability of the previous step. We assume that there is a low-dimensional embedding space (M,g) that encodes the walk. An encoder, E MM + MM, maps input, x, from the data manifold, Mz, to the embedding space. A generator, G : MM + My. takes both the embedding coordinate and the number of steps and maps it back to the data manifold (see Figure
We use (;, .) to denote the explainer function. Formally, (x( (X,R) - - X' is a function that takes two arguments: a query image x and the desired perturbation 8. This function generates perturbed image which is then passed through function f. The difference between the outputs of f given the original image and the perturbed image should be the desired change i.e., fxx)) fxx 6. We use X5 to denote Z 8). This formulation enables us to use ó as a knob to exaggerate the visual explanations of the query sample while it crossing the decision boundary given by function f. Our proposed interpretabilityy function 7 should satisfy the following properties:
1. Data Consistency: perturbed samples generated by I should lie on the data manifold, Mg. to be consistent with real data. In other words, the generated samples should look realistic when compared other samples.
2. Compatibility with f: changing the second argument in (x,.. should produce the de- sired outcome from classifier J. i.e., (I, 6)) ~ (xx + 6.
3. Self Consistency: Applying reverse perturbation should bring x back to its original form i.e., (x, -5) x. Also, applying setting ō to zero should return the query, i.e. (x, = x.
Each criterion is enforced via a loss function which are discussed in the following sections.
2.1
DATA CONSISTENCY
We adopt the Generative Adversarial Networks (GANs) framework for our model (Goodfellow et al., 2014). The GANs implicitly model the underlying data distribution by setting up min-max game between generative (G) and discriminative (D) networks:
LGAN(D, G) = Pxx) log (D(x))] + Fa [log (1 D(G(
where z and Pa are the noise distribution and the corresponding canonical distribution. There has been sigmificant progress toward improving GANs stability as well as sample quality (Brock et al., 2019; Karras al., 2019). The advantage of GANs is that they produce realistic-looking samples without an explicit likelihood assumption about the underlying probability distribution. This prop- erty is appealing for our application.
Furthermore, we need to provide the desired amount of perturbation to the black-box, f. Hence, we use a Conditional GAN (cGAN) that allows the incorporation of context as condition the GAN (Mirza & Osindero, 2014; Miyato & Koyama, 2018). To define the condition, we fix the step size, 6, and descritize the walk which effectively cuts the posterior probability range of the predictor (i.e., [0, 1]) into
I equally-sized bins. Hence, one can view the perturbation from f(x) to f(x) + 8
as changing the bin index from the current value cx(x, 0) to (x, where cx(x, 8) returns the bin index of fxx + 8. We use cx(x, d) as a condition to the cGAN.
The cGAN optimizes the following loss function:
LOGAN(D, G) = Ex,o-P(xx.)) loo (D(x,c) c))] + Pee [log (1 D(G(z, c)).
(1)
where e denotes a condition. Instead of generating random samples from Pz. we use the output of an encoder, E(x), as input to the generator. Finally, the explainer function is defined as:
(x 6) = G(E(x), (x 6))
(2)
Our architecture is based on Projection GAN (Miyato & Koyama, 2018), a modification of cGAN. An advantage of the Projection GAN is that it scales well with the number of classes allowing 8 0. The Projection GAN imposes the following structure on the discriminator loss function:
LOGAN (D, ()) c) = log Paaa (c|x)
+ loog (x)a(x)
r(clx) + (+(G()))
q(c|x)
q(x)
where COGAN DD, G) indicates the loss function in Eq. when & is fixed, ((-) and (b(-) are networks producing vector (feature) and scalar outputs respectively. The (c(xx) is a conditional ratio function which will be discussed in Section 2.2.
3

---

Published as a conference paper at ICLR 2020
2.2 COMPATIBILITY WITH THE BLACK BOX
In our model, the condition c is an ordered variable i.e., cx 61) < cf( 02) when 6n < 82. There- fore, adapt the first term in Eq. account for ordinal multi-class regression by transforming (x into [ binary classification terms (Frank & Hall, 2001):
r(c = x = Σ v ((x), KZ
(4)
where (-)) is the feature network in Eq. 3 and v are parameters. We also need to ensure that plugging X; into f(-) yields f(x) + 6 (i.e., compatible with f). This condition enforce by
KullbacLeeibler (KL) divergence loss term. Adding the KL loss and the conditional ratio function we arrive at the following loss:
Cy(D,G) = r-((xx)) + DKL (f(x) + (x)
While the first term is a function of both G and D, the second term influences only the generator G.
2.3 SELF CONSISTENCY
We use reconstruction loss term to enforce encoder-decoder consistency and satisfy the identity constraint of x (x, 0),
= (B(())=)
(5)
We also require that the perturbation is reversible (i.e., I, (I, 8), -) x). We use cycle- consistency (Zhu et al., 2017) loss to reconstruct the input from its corresponding perturbed image,
Laye = ||x G(E(xs), (x 0))l
Note that the conditions for the generators in Eq. 5 and 6 are the same. However, the former, we are reconstructing the input x from its latent space, but in the latter, we perturb xş from the bin index ey (x, 6) back to original bin index cy (x, 0).
2.4 OBJECTIVE FUNCTIONS
We adapted the hinge version of the adversarial loss for LeGAN D).
CoCAN(D) Expppm (0 - (xx,
Ex cy (x).)
- DiI(OO--1DDGGEE().c((//o 8)))
LOGAN (G, Dww.
(D(G(E())) 8))
(8)
The overall objective function is
min max AGGAN AyC, (CC XreCrooGG)) + XxceCeye
(9)
E.G
where ACGAN, Ay, Ars are the hyper-parameters that balance the importance the loss terms.
RELATED WORK
Our work broadly relates literature in interpretation methods that are designed to provide a visual explanation of the decisions made by a black-box function f. for given query sample x.
Perturbation-based methods: These methods provide interpretation by showing what minimal changes are required in x to induce desirable output of f. Some methods employed image manip- ulation via the removal of image patches (Zhou et al., 2014) or the occlusion of image regions (Zhou et al., 2014) to change the classification score. Recently, the use of influence function, proposed by (Koh & Liang, 2017) are applied as a form of data perturbation to modify classifier's 's response.
The authors in (Fong & Vedaldi, 2017) proposed the use of optimal perturbation, defined as remov- ing the smallest possible image region in x that results the maximum drop in classification score. In another approach, (Chang et al., 2019) proposed generative process to find and fill the image
4

---

Published as a conference paper at ICLR 2020
regions that correspond the largest change in the decision output of a classifier. To switch the decision of a classifier, (Goyal et al., 2019) suggested generating counterfactuals by replacing the regions of x with patches from images with a different class label. All of the aforementioned works perform pixel- or patch-level manipulation to x, which may not result in natural-looking images. In contrast, our model enforces that the perturbed data be consistent with the unperturbed data to ensure that the perturbation is plausible. Furthermore, our method can be applied to general data and is not restricted to the imaging domain.
Saliency map-based methods: Saliency maps explains the decision of f on x by highlighting the relevant regions of z. Some earlier work in this direction(Simonyan et al., 2013; Springenberg al., 2015: Bach et al., 2015) focuses on computing the gradient of the target class with respect to x and considers the image regions with large gradients as most informative. Building on this work, the class activation map (CAM) (Zhou et al., 2016) and its generalized version Grad-CAM Selvaraju et al. (2017) and other variants such as LPR (Bach et al., 2015) use a linear or non-linear combination of the activation layers to derive relevance score for every pixel in image. These -aased methods are not model-agnostic and require access to intermediate layers. Recently, Adebayo et al. (2018) have shown that some saliency methods are independent both of the model and of the data generating process. We used their propose evaluation to validate our interpretation model. The saliency maps also prone to adversarial attacks as shown by Ghorbani al. (2019) and Kinder- mans et al. (2017). Furthermore, the causal effect of a class is distributed across an image, which the case radiology images, the saliency approaches highlight large sections of the image, which greatly reduce the usefulness of the interpretation.
Generative explanation- methods: These are interpretation models that uses generative process to produce visual explanations. The contrastive explanations method (CEM) (Dhurandhar al., 2018) generates explanations that show minimum regions in x which must be present/absent for particular classification decision. In another work, (Liu et al., 2019; Joshi et al., 2019; Saman- gouei et al., 2018) generates explanations that highlight what features should be changed in x so that the classifier confidence in the prediction is strengthen (prototype) or weakened (counterfac- tual). Our approach is aligned with these latter lines of work, although our method and model architecture is different. Our method allows for the gradual change of the class effect, and our con- sistency criteria result in high-quality feasible perturbation in x. We rigorously evaluate our method on real medical imaging applications, in addition the curated computer vision datasets.
4 EXPERIMENTS
We set up four experiments evaluate our method. First, we assess if our method satisfies the three criteria of the explainer function introduced in Section 2. We report both qualitative and quantitative results. Second, we apply our method on medical image diagnosis task. We use external domain knowledge about the disease to perform a quantitative evaluation of the explanation. Third, we train two classifiers on biased and unbiased data and examine the performance of our method in identifying the bias. While our method does not produce a saliency map, in our last experiment, use the two counterfactual samples on the boundary [0, 1] to generate a saliency map and compare it with the other methods. In Appendix A, we show further experiments to evaluate model in human experiments, to demonstrate its compatibility with a multi-labe classifier and, an ablation study, to show the relative importance of each of the three criteria of the explainer function.
Our experiments are conducted on the CelebA (Liu et al., 2015) and CheXpert (Irvin et al., 2019) datasets. CelebA contains 200K celebrity face images, each with forty attribute labels. We consid- ered binary classifier trained on the "smiling" and "young" attributes. CheXpert a medical dataset containing 224K chest x-ray images from 65K patients and has labels for fourteen aadio-graphi ob- servations. We considered Cardiomegaly as the target class for generating explanations. All images are re-sized 128 128 before processing.
4.1 EVALUATING THE CRITERIA OF THE EXPLAINER
Figure 2 reports the qualitative results three datasets. Given a query image inference time, our model generates a series of images X5 as visual explanations, which gradually increase the posterior probability fxxs) (top label). We show results for three prediction tasks: smiling not-smiling.
5

---

Published as a conference paper at ICLR 2020
young old, and Cardiomegaly healthy, The values the top of each figure report the f(xs) For Cardiomegaly, we show the outlines of the heart as well as its normalized size (values inside the parenthesis), which is indicative of the disease.
Query Image
Generated Visual Explanations
Desired f(x)):
(0.0-0.1)
[0.2-0.3)
(0.3-0.4)
(0.5-0.6)
[0.6-0.7)
(0.7-0.8)
[0.9-010]
Not-smiling
Smiling
f(x) 100
(xs): 0.04
0.39
0.47
0.63
0.84
0.95
1.0
CelebA: Smiling
0.0
0.31
0.25
0.26
0.69
0.97
Old
Young 0.97
0.96
0.11
0.18
0.28
0.46
0.45
0.72
CelebA: Young
0.16
0.07
0.41
0.38
0.56
0.36
0.79
0.93
Healthy
Cardiomegaly
0.81(1.07)
0.45(1.0)
0.47(1.02) 0.47(1.11) 0.63(1.1) 0.65(1.14) 0.7(1.17) 0. 0.78(1.25)
X-Ray: Cardiomegaly
0.12(0.99)
0.07(1.0) 0.23(1.07) 0,24(1.09) 0.320.11) 11) 0.71(1.25) 0.64(1.27) 0.87(1.3)
Figure 2: Visual explanations generated for three prediction tasks: smiling/not-smiilee young/old face (middle two rows) and Cardiomegalyhhellty chest x-ray (bottom two rows). The first column shows the query image, followed by the corresponding generated explanations. The values above each image the output the classifier f. For Cardiomegaly. show the segmentation of the heart (yellow edge) and report normalized heart size (values parenthesis), which indicative of disease.
(first two rows),
Data Consistency: The generated explanations are synthesized variations the query image. To quantitatively compare their visual quality, we consider Fréchet Inception Distance (FID) (Heusel et al., 2017). We compared our results against the counterfactual explanations produced by XGEM (Joshi al., 2018). The details of the xGEM model are given in appendix A.2. We di- vided the real and fake (i.e,, generated explanations) images into two groups (on either boundary of fxx E j0, 1)) and reported the FID for each group and the overall score. Our method signifi- cantly outperforms xGEM, producing crisper and more realistic-looking images. xGEM is based on variational autoencoder (VAE) which are known to produce blurry images (see Figure 7).
Compatibility with the black-box f: the desire perturbation we plotted the expected outcome f(x) + against the actual response of the classifier for the generated explanations, fxxs). Figure 3 shows how our model performs when generating a series of explanations starting from a wide range of initial query images. The perfor- mance is almost perfect for Young/Old, but less for more challenging classification problems such as Smiling or Cardiomegaly. The plot also validates that we are producing perturb images covering the entire classification range, [0, 1). Appendix A.3 shows additional result from CelebA dataset.
To quantify whether the generation process is aligned with

---

Published as a conference paper at ICLR 2020
Celibn:Smiling
CelebA:
Young
Xray:Cardiomegaly
Target Class
xGEM
Ours
xGEM
Ours
xGEM
Ours
Present (x(x)) [0.9, I)
111.0
46.9
115.2
67.6
368.6
82.9
Absent (f(xs) 00 0.1)
112.9
56.3
170.3
74,4
394.6
84.8
Overall (/(xs) E [0, 1)
106.3
35.8
117.9
53.4
326.3
58.1
Table I: The Fréchet Inception Distance (FID) score, measuring the quality of the generated explanations for the three prediction tasks. Lower FID corresponds better image quality. Top (bottom) row corresponds the top (bottom) 10% of the dccision interval.
CelebA(Smiling)
CelebA(Young)
XRay|Cardiamagely)
1.0
Query f(x)
0.6
0.0-0.2
0.2-0.4
0.4-0.6
0.6
0.2
0.8-1.0
0.0 0.2
0.4
0.6
0.8
1.0.0.0
0.2
0.4
0.6
0.8
1.0
0.0
+
+
((x] +
Figure 3: Plot of the expected outcome from the classifier. f(x) + against the actual response of the classifier on generated explanations, fxx). The monotonically increasing trend shows positive correlation between f(x) + 8 and fxx.). and thus the generated explanations consistent with the expected condition.
Identity preservation: The generated explanations should differ only in semantic features associ- ated with the target class, while retaining the identity of the query image. We extracted the latent embedding for real images (E(x)) and their corresponding explanations (E(xs)). )), for different val- ues of 8. We calculated latent space closeness the percentage of the times, X5 is closest to the query image x as compared to other generated explanations
vO, X,
[[ E(xs)lla
min
|E((s) E(m)lla.
(10)
(X
the of explanations generated for all the real images excluding the query image x. Another, popular approach to quantify identity of two face images, is to perform face verification. We used state-of-the-art face recognition model trained on VGGFace2 dataset (Cao et al., 2018) as feature extractor for both real images and their corresponding fake explanations. For face verification, we calculated the closeness between real and fake image as cosine distance between their feature vectors. The faces were considered as verified i.e., fake explanation have same identity as real image, if the distance below 0.5. Table summarizes the results.
where, m E
Our method achieved high performance on localized attribute "smiling", which alters relatively small region of the face image as compare to attribute "age" which affects the entire face. Medical images like chest x-ray have very fine grain details which are difficult to preserve in the generative process of GAN. Our explainer function preserves the high level features like shape and size of the lung, but it struggles retain the low level features like anatomy of the breast and shape of the collar bones. Also, it should be noted that both the datasets have multiple images for same person, but we ignore this information in our analysis and treat each image different identity. We compared our performance against XGEM (Joshi et al., 2018). VAE explicitly minimizes for latent space closeness. The generated explanation by xGEM were blurry version of the query image. Hence, although they were close to query image in latent space, but they didn't 'tt preserve the identity of the individual as shown in face verification task and evident Figure in appendix A.2. comparison, our model achieved good performance on both the tasks.
4.2 COUNTERFACTUAL EVALUATION ON MEDICAL DATA
Cardiomegaly refers to an abnormal enlargement of the heart (Brakohiapa et al., 2017). To un- derstand the explanations derived for Cardiomegaly target class, overlaid the heart segmentation

---

Published as a conference paper at ICLR 2020
CemiingSmiling
CelebA:Young
Xray:Cardiomegaly
xGEM
Ours
xGEM
Ours
xGEM
Ours
Latent Space Closeness
88.2
88.0
89.5
81.6
2.2
27.9
Face Verification Accuracy
0.0
85.3
0.0
72.2
Table 2: Identity preserving performance on three prediction tasks.
over the x-ray image and visualize the gradual change in heart size. The heart segmentation is shown as outlines in Figure 2, with their corresponding heart size (top values in parentheses). The heart segmentation derive by training a UNet (Ronneberger et al., 2015) model on the segmentation in chest radiograph (SCR) dataset (van Ginneken et al., 2006). We registered with its associated X; and applied the resulting transformation to the heart masks of x to derive the heart masks for X5-
For population-level analysis, we plotted the average heart size of X5 the condition used for generation (x( + 5) in Figure (a). The plot shows a positive correlation between the heart size and the response of the classifier fxx), which agrees with the definition of Cardiomegaly. To better understand the results, we divided the population into two groups, the first group (x"; f(x") < 0.1)
consists of real images of healthy x-rays, and the second group (x"; f(x") 0.9) contains real images of abnormal X-rays positive for Cardiomegaly. For x" we generated counterfactual as x such that (xx] > 0.9. Similarly, counterfactuals for x are derived as x such that I' f(x²) < 0.1. In Figure (b), we show the distribution of heart size in the four groups. We reported the dependent t-test statistics for paired samples (x" and x5). (x" and x²).. A significant p-value < 0.001 rejected the null hypothesis (i.e., that the two groups have similar distributions). We also reported the independent two-sample t-test statistics for healthy (x" and x p-value > 0.01) and abnormal (xe and x3. p-value < 0.01) populations. Given higher p-values, we cannot reject the null hypothesis of identical average distributions with high confidence. Our model derived explanations successfully captured the change in heart size while generating counterfactual explanations.
3.5
1.0
1.8
Grad- Read-Cam
2.6
1.6
p.c.0.00
Accracy of the clasoffirr
DeepLIFT (Rescale)
Fpeloo-180
Ours
1.4
Random
2.2
1.2
07
Normalized
0.0
1.0
0.8
0.6
0.6
Daal Healthy
Counterfactual
Counterfactual Abnormal Healthy
Xx]
Abnormal XS
pertrubed
(b)
(c)
Figure 4: Cardiomegaly discase associated with large heart size. we show the positive correlation between the heart size and the response of the classifier fxx). (b) Comparison distribution of the heart size in the four groups. (c) Plot to show the drop in accuracy of the classifier as we perturb the most relevant pixels (relevance calculated from saliency map) in the image.
4.3 SALIENCY MAP
Saliency maps show the importance of each pixel of an image in the context of classification. Our method is not designed to produce saliency maps as a continuous score for every feature of the input. We extract an approximate saliency map by quantifying the regions that changed the most when comparing explanations at the opposing ends of the classification spectrum. For each query image, we generated two visual explanations corresponding to the two extremes of the decision boundary (f(x.) = 0 and f(xs) = 1). The absolute difference between these explanations is our saliency map. Figure shows the saliency map obtain from our method and its comparison with popular gradient based methods. We restricted the saliency maps obtained from different methods to have positive values and normalize them to range [0,1]. Subjective, the saliency maps produced by method are very localized and are comparable to the other methods.
8

---

Published as a conference paper at ICLR 2020
We adapted the metric introduced in (Samek et al., 2016) to compare the different saliency maps. In an iterative procedure, we progressively replace a percentage of the most relevant pixels in an image (as given by the saliency map) with random values sampled from uniform distribution. We observe the corresponding change in the classification performance as shown in Figure 4 (c). All the methods experienced drop in the accuracy of the classifier with increase in the fraction of perturb pixels. The saliency maps produced by our model is significantly better than random maps and are comparable to the other saliency map methods. It should be noted that, there are many ways to quantify important regions in image, using the series of explanations generated by our method. We didn't 't optimize to find the best saliency map and showed results for one such method.
Original
Grad-CAM
Epsilon-LRP
DeepLIFT (Rescale)
Ours
Figure 5: Our comparison with popular gradient- saliency map producing methods on the prediction task of identifying smiling faces in CelebA dataset.
4.4
BIAS DETECTION
Our model can discover confounding bias in the data used for training the black-box classifier. Con- founding bias provides an alternative explanation for an association between the data and the target label. For example, a classifier trained to predict the presence of disease may make decisions based on hidden attributes like gender, race, or age. In simulated experiment, we trained two clas- sifiers to identify smiling not-smiling images in the CelebA dataset. The first classifier fBiased is trained on a biased dataset, confounded with gender such that all smiling images are of male faces. We train second classifier fNo-biased on unbiased dataset, with data uniformly distributed with respect to gender. Note that we evaluate both the classifiers on the same validation set. Addition- ally, we assume access proxy Oracle classifier SGender that perfectly classifies the confounding attribute i.e, gender. As shown in Cohen et al. (2018), if the training data for the GAN is biased, then the inference would reflect that bias. In Figure 6, we compare the explanations generated for the two classifiers. The visual explanations for the biased classifier change gender as it increases the amount of smile. We adapted the confounding metric proposed Joshi al. (2018) to summarize )). we quantify that
our results in Table 3. Given the data D {(x;, Vi. a:). X; E X yi, classifier is confounded by an attribute a if the generated explanation âs has a different attribute a, as compared to query image when processed through the Oracle classifier fGender- The metric is formally defined as Ep (g(((())) * DD) For biased classifier, the Oracle function predicted the female class for the majority of the images, while the unbiased classifier is consistent with the true distribution of the validation set for gender. Thus, we the fraction of generated explanations that changed the confounding attribute "gender" was found to be high for the biased classifier.

---

Published as a conference paper at ICLR 2020
Target Label
Black-box classifier
Smiling
Not-Smiling
/iised
Male: 0.52
Male: 0.18
Female: 0.48
Female: 0.82
Overall: 0.12
Overall: 0.35
No-biassed
Male: 0.48
Male: 0.47
Female: 0.52
Female: 0.53
Overall: 0.07
Overall: 0.08
Table 3: Confounding metric for biased detection. For target label "Smiling" and "Not-Smiling", the explana- 0.1 respectively. The Male and Female
tions generated using condition f(x) + > 0.9 and f(x) values quantifies the fraction of the generated explanations classifier as male female, respectively by oracle classifier fomder. The overall value quantifies the fraction the generated explanations who have different gender as compared to the query image. small overall value shows least hias.
Query Image
Generated Visual Explanations
Desired f(x):
[0.0-0.1]
(0.2-0.3)
(0.3-0.4)
(0.5-0.6)
[0.6-077]
(0.7-0.8)
[0.9-01.0]
Not-smiling/Femall 0.0)
Smillng/Malle
flxlo.0
0.23
0.32
0.38
0.49
0.89
0.94
Biosad
0.0
0.0
0.21
0.28
0.34
0.48
0.74
0.96
No di esed
0.0
0.0
0.19
0.31
0.36
0.45
0.72
0.95
Blazed
0.0
1.0
No Biesed
0.0
0.01
0.01
0.01
0.03
0.06
0.1
0.0
Figure 6: The visual explanations for two classifiers, both trained to classify "Smiling" attribute on CelebA dataset. For each example, the top row shows results from "Biased" classifier whose data distribution is con- founded with "Gender". The bottom row shows explanations from "No-Biased" classifier with uniform data distribution gender. The top label indicates output of the classifier and the bottom label the output of an oracle classifier for con-foure con-founding attribute gender. The visual explanations for the "Biased" classifier changes the gender as it adds smile on the face.
CONCLUSION
In this paper, we proposed a novel interpretation method that explains the decision of black-box classifier by producing natural-looking, gradual perturbations of the query image, resulting in an equivalent change the output of the classifier. We evaluated our model on two very different datasets, including a medical imaging dataset. Our model produces high-quality explanations while preserving the identity of the query image. Our analysis shows that our explanations are consistent with the definition of the target disease without explicitly using that information. Our method can also be used to generate a saliency map in a model agnostic setting. In addition to the interpretability advantages, proposed method also identify plausible confounding biases in classifier.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, pp- 9505-9515, 2018.
Sebastian Bach, Alexander Binder, Grgoire Montavon, Frederick Klauschen, Klaus-Robert Muller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
Edmund Kwadwo Kwakye Brakohiapa, Benard Ohene Botwe, Benjamin Dabo Sarkodie, Eric Kwesi Ofori, and Jerry Coleman. Radiographic determination of cardiomegaly using cardiotho- racic ratio and transverse cardiac diameter:
size fit all? Part one. The Pan African medical journal, 27:201, 2017. ISSN 1937-8688. doi: 10.11604/pamj.200 2017.27.201. 12017. URL http: www. ncbi nlm. nih. gov/ tww.gpooe//aagoo pttpp//pm///44//mm/// pubmedcentrale nih. gov/art erender. fagi Part www.iidd=cc5559933
www.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. In International Conference on Learning Representations, 2019. URL
https: /opp:://spenreetii new feee/forumin d= x
Q Cao, I. Shen, W Xie, O M Parkhi, and A Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In International Conference Automatic Face and Gesture Recognition, 2018.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining Image Clas- sifiers by Counterfactual Generation, 2019. URL.htts::///peeeeete /ttp:////ppe/eeeeee/e/ee net forum?ide B1MXz20cYQ.
Joseph Paul Cohen, Margaux Luck, and Sina Honari. Distribution matching losses can hal- lucinate features in medical image translation. In Lecture Notes in Computer Science (in- cluding subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformat- ics), volume 11070 LNCS, pp. 529-536. Springer Verlag, 2018. ISBN 9783030009274. doi: 000100799778---0000092--------
Henriette Cramer, Jean Careia-Gathrige Aaron Springer, and Sravana Reddy. Assessing and ad- dressing algorithmic bias in practice. interactions, 25(6):58-66) 66, 2018.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances Neural Information Processing Systems, pp. 592-603, 2018.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint 1w0..v1100..00. 08600, 2017.
Ruth C. Fong and Andrea Vedaldi. Interpretable Explanations Black Boxes by Meaningful Per- turbation. In Proceedings the IEEE International Conference Computer Vision, 2017. ISBN 9781538610329. doi: 10. 11.11////CC.221...37..
Eibe Frank and Mark Hall. A simple approach to ordinal classification. In European Conference Machine Learning, pp. 145-156, 2001.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Alyssa Glass, Deborah L McGuinness, and Michael Wolverton. Toward establishing trust in adaptive agents. In Proceedings of the 13th international conference on Intelligent user interfaces, pp- 227-236. ACM, 2008.
11

---

Published as a conference paper at ICLR 2020
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Z Ghahramani, M Welling, C Cortes, N D Lawrence, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URLhttp: http///apprs... paper/5423-geneee
cc/
www.vvvevavvaaaaveeeeeaaaaa wwweerssrrrlleettsseeeeee pdf.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual Visual Explanations. International Conference on Machine Learning, pp. 2376-2384, 2019. URL
http://arxiv org/ /t.bs/1904 004551.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5):93, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus. Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew Lungren, and Andrew Y. Ng. CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison, 2019. URL http://axxx..oraaa//// org/bb// 1901. 07031.
Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xGEMs: Generating Exam- plars to Explain Black-Box Models. arXiv preprint haw::8806.8888 2018.
Shalmali Joshi, Oluwasanmi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. To- wards Realistic Individual Recourse and Actionable Explanations Black-Bor Decision Making Systems. CoRR, htts/1907.09615. 2019. URL http: http:::///xvvvv... ort.ass//o////a/////a////.. aaa/bb///// abss/190..99e
Tero Karras, Samuli Laine, and Timo Aila, A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 4401-4410, 2019.
Been Kim. Interactive and interpretable machine learning models for human machine collabora- tion. PhD) thesis, Massachusetts Institute of Technology, 2015.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280, 2017.
Pang Wei Koh and Percy Liang Understanding black-box predictions via influence functions. In 34th International Conference on Machine Learning, ICML 2017, 2017. ISBN 9781510855144
Shusen Liu, Bhavya Kailkhura, Donald Loveland, and Yong Han. Generative Counterfactual Intro- spection for Explainable Deep Learning. arXiv, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
Mehdi Mirza and Simon Osindero. Conditional Generative Adversarial Nets. CoRR, abs/1411.178.. 2014. URL http: org/ 1784.
Takeru Miyato and Masanori Koyama. cGANs with Projection Discriminator. In International fformm?
Conference on Leurning Representations, 2018. id=ByS1VpgRR.
Christoph Molnar. Interpretable Machine Learning. Github, 2019. https: h/trss//hhhristph.mm github. ic/ wt//nteerpeeabll... -bbok//k
12

---

Published as a conference paper at ICLR 2020
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-1144 ACM, 2016.
Olaf Ronneberger, Philiipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomed- ical Image Segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015. URL http: / aaaaiiiiiv...... wtt///gg///b/aoo..//.../..
abs 105.. 04597.
Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. Explaingan: Model explanation via decision boundary crossing transformations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 666-681, 2018.
Wojciech Samek, Alexander Binder, Grgoire Montavon, Sebastian Lapuschkin, and Klaus-Rober Müller. Evaluating the visualization what a deep neural network has learned. IEEE transactions neural networks and learning systems, 28(1 (661)):660-2673 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. Computing Research Repository, 2013.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A Riedmiller. Striving for Simplicity: The All Convolutional Net. In ICLR (workshop track), 2015.
Ryan Tumer. A model explanation system. In 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1-6. IEEE, 2016.
B van Ginneken, M B Stegmann, and M Loog. Segmentation of anatomical structures in chest radiographs using supervised methods: comparative study on a public database. Medical Image Analysis, 10(1):19 40, 2006.
Anthony J Viera, Joanne M Garrett, and others. Understanding interobserver agreement: the kappa statistic. Fam med, 36653360 2005.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. Computing Research Repository, 2014.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired Image-to-lmage Transla- tion using Cycle-Consisted Adversarial Networks. In Proceedings of the IEEE international con- ference computer vision, pp. 2223-2232, 2017. URL http: at/paaxxvvv...../... org/ wttpo/ggg/bb///////// 1703. 10593.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The architecture for the generator and discriminator is adapted from Miyato & Koyama (2018). 8). The image encoding learned by encoder E(x) is fed into the generator. The condition cy 5) is passed to each resnet block in the generator, using conditional batch normalization. The generator has five resnet blocks, where each block consists of EN-ReLL-Conw3-BR -N--RLL-CCoov3. BN is batch normalization, ReLU is activation function, and Conv3 is the convolution filter. The encoder function uses the same structure but downsamples the image. The discriminator function has five resnet blocks, each of which has the form Rel.-v--e...--oooeee
13

---

Published as a conference paper at ICLR 2020
A.2 XGEM IMPLEMENTATION
We refers to Joshi et al. (2019) for the implementation of xGEM First. VAE is trained to generate face images. The VAE used available athttps//gtthub.omm/// waw.bb/oooomyyy///aa/aea. L All settings and architectures were set to default values. The original code generates image of dimension 64x64. We extended the given network to produce an image with dimensions 128x128. The pre- trained VAE is then extended to incorporate the cross-entropy loss for flipping the label of the query image. The model evaluates the cross-entropy loss by passing the generated image through the classifier. Figure shows the qualitative difference between the explanations generated by our proposed method and xGEM.
A.3 EXTENDED RESULTS FOR EVALUATING THE CRITERIA OF THE EXPLAINER
Here, we provide results for four more prediction tasks celebA dataset: no-beard or beard, heavy makeup or light makeup, black hair or not back hair, and bangs or no-bangs. Figure shows the qualitative results, extended version of results in Figure 2. We evaluated the results from these prediction tasks for compatibility with black-box f (see Figure 9), data consistency and self consis- tency (see Table 4).
Prediction Task
Data Consistency (FID)
Self Consistency
Present
Absent
Overall
LSC
FVA
Smiling vs Not-smiling
46.9
56.3
35.8
88.0
85.3
Young vs Old
67.5
74.4
53.4
81.6
72.2
No beard vs Beard
79.2
72.3
45.4
89.6
83.3
Heavy makeup vs Light makeup
64.9
98.2
39.2
89.2
75.3
Black hair vs Not black hair
55.8
72.8
34.8
79.4
81.6
Bangs vs No bangs
54.1
57.8
40.6
76.5
87.3
Table 4: Our model results for six prediction tasks on CelebA dataset. FID (Fréchet Inception Distance) score measures the quality of the generated explanations. Lower FID is better. LSC (Latent Space Closeness) quantifies the fraction of the population where generated explanation is nearest to the query image than any other generated explanation in embedding space. FVA (Face verification accuracy) measures percentage of the times the query image and generated explanation have same face identity per model trained on VGGFace2. Higher LSC and FVA is better.
A.4 HUMAN EVALUATION
We used Amazon Mechanical Turk (AMT) to conduct human experiments to demonstrate that the progressive exaggeration produced by our model is visually perceivable to humans. We presented AMT workers with three tasks. In the first task, we evaluated if humans can detect the relative order between
explanations produced for given image. We ask the AMT workers, "Given two images of the same person, in which image is the person younger (or smiling more)?" (see Figure 10). We experimented with 200 query images and generated two pairs of explanations for each query image (i.e., 400 hits). The first pair (easy) imposed the two images are samples from opposite ends of the explanation spectrum (counterfactuals), while the second pair (hard) makes no such assumption.
In the second task, we evaluated humans can identify the target class for which our model has provided the explanations. We ask the AMT workers, "What is changing in the images? (age, smile, hair-style or beard)". We experimented with 100 query images from each of the four attributes (i.e., 400 hits). In the third task, we demonstrate that our model can help the user to identify problems like possible bias in the black-box training. Here, we used the same setting as in the second task but also showed explanations generated for biased classifier. We ask the AMT workers, "What is changing in the images? (smile or smile and gender)" (see Figure 10). We generated explanations for 200 query images each, from biased-classife classifier (aae)) explainer (i.e., 400 hits). In all the three tasks, we collected eight votes for each task, evaluated against the ground truth, and used the majority vote for calculating accuracy.
(fBlased) explainer from Section 4.4 and unbiased
14

---

Published as a conference paper at ICLR 2020
We summarize results in Table 5. In the first task, the annotators achieved high accuracy for the easy pair when there was a significant difference among the two explanation images, as compared to the hard pair when the two explanations were successful in identifying the relative order between the two explanation images.
have very subtle differences. Overall, the annotators
In the second task, the annotators
generally successful in correctly identifying the target class. The target class "bangs" proved to be the most difficult to identify, which was expected. The gen- erated images for "bangs" were qualitatively, the most subtle. For the third task, the correct answer was always the target class i.e., "smile". In the case of biased classifier explainer, the annotators selected "Smile and Gender" 12.5% of the times. The gradual progression made by the explainer for a biased classifier was very subtle and was changing large regions of the face compared to the unbiased explainer. The difference much more visible when we compare the explanation gener- ated for the same query image for biased and no-biased classifier, as in Figure 6. But in realistic scenario, the no-biased classifier would not be available to compare against. Nevertheless, the an- notators detected bias at roughly the same level of accuracy our classifier (Table 3). Future work could improve upon bias detection.
Annotation Task
Overall
Sub categories
Accuracy 83.5%
k-statistic
Category
Accuracy
k-statistic
Task-1 (Age)
0.41 (Moderate)
Hard
73%
0.31 (Fair)
Easy
94%
0.51 (Moderate)
Task-1 (Smile)
77.5%
0.28 (Fair)
Hard
66%
0.23 (Fair)
Easy
89.5%
0.32 (Fair)
Task-2 (Identify Target Class)
77%
0.35 (Fair)
Age
72%
Smile
99 %
Bangs
50%
Beard
87%
Task-3 (Bias Detection)
93.75%
0.14 (Slight)
Biasse
87.5%
0.09 (Slight)
No hiased
100 %
0.02 (Slight)
Table 5: Summarizing the results of human evaluation. The x -stttistics measure inter-rater agree- ment for qualitative classification of items into some mutually exclusive categories. One possible interpretation of x as given in Viera et al. (2005) is < 0.0: Poor, 0.01 - 0.2: Slight, 0.21 0.40: Fair, 0.41 - 0.60: Moderate, 0.61 0.80: Substantial and 0.81 1.00: Almost perfect agreement.
A.5 EVALUATING CLASS DISCRIMINATION
In multi-label settings, multiple labels can be true for given image. In this test, we evaluated the sensitivity of our generated explanations to the class being explained. We consider a classifier trained to identify multiple attributes: young, smiling, black-hair, no-beard and bangs in face images from CelebA dataset. We used our model to generate explanations while considering one of the attributes as the target. Ideally, explanation model trained to explain target attribute should produce explanations consistent with the query image on all the attributes beside the target. Figure 11 plots the fraction of the generated explanations, that have flipped in source attribute as compared to the query image. Each column represents one source attribute. Each row is one run of our method to explain given target attribute.
A.6 ABLATION STUDY
Our proposed model has three types of loss functions: adversarial loss from cGAN, KL loss, and reconstruction loss. The three losses enforce the three properties of our proposed explainer function: data consistency, compatibility with f. and self-conssitency, respectively. In the ablation study, we quantify the importance of each of these components by training different models, which differ in one syper-paramete while rest equivalent (AaaN 1, Xy 1 and Axe 100). For data consistency, we evaluate Fréchet Inception Distance (FID). FID score measures the visual quality of the generated explanations by comparing them with the real images. We show results for two
15

---

Published as a conference paper at ICLR 2020
groups. In the first group, consider real and fake images where the classifier has high confidence in presence of the target label i.e. f(xs), f(x) E [0.9, 1.0). In second group, the target label is absent i.e., fxx)), fxx E [0.0, 0.1). We also report overall score by considering all the real and generated explanationn together. For compatability with f we plotted the desired output of the clas- sifier i.e., [(x + 5 against the actual output of the classifier f(xx) for the generated explanations. For self' consistency, we calculated the Latent Space Closeness (LSC) measure and Face verification accuracy (FVA). LSC quantifies the fraction of the population in which the generated explanation is nearest to the query image than any other generated explanation in embedding space. FVA mea- sures the percentage of the instances in which the query image and generated explanation have the same face identity as per the model trained on VGGFace2. For the ablation study, we consider the prediction task of young vs old on the CelebA dataset. Figure 12 shows the results for compatibility with f. Table 6 summarizes the results for data consistency and self-consistency.
Configuration
Data Consistency (FID)
Self Consistency
AGAAN
A
Arec
Present
Absent
Overall
LSC
FVA
100
69.7
105.7
67.2
96.1
99.8
1
1
100
67.5
74.4
53.4
81.6
72.2
10
1
100
89.4
105.2
63.0
68.0
82.7
100
1
100
71.6
80.6
44.26
75.3
18.0
0
100
66.2
66.2
44.9
77.2
99.4
1
100
67.5
74.4
53.4
81.6
72.2
10
100
95.5
90.4
62.4
71.83
96.8
100
100
77.4
73.1
71.2
55.4
42.23
0
116.2
118.9
72.2
16.6
0.0
63.0
78.6
61.6
32.2
5.5
1
10
87.6
83.6
65.7
71.5
88.8
67.5
74.4
53.4
81.6
72.2
Table 6: Our model with ablation on prediction task of young vs old on CelebA dataset. FID (Fréchet Inception Distance) score measures the quality of the generated explanations. Lower FID better. LSC (Latent Space Closeness) quantifies the fraction of the population where generated explanation nearest to the query image than any other generated explanation in embedding space. FVA (Face verification aceuracy) measures percentage of the times the query image and generated explanation have same face identity as per model trained on VGGFace2. Higher LSC and FVA better.
16

---

Published as a conference paper at ICLR 2020
Query Image
Generated Visual Explanations
Not-smiling
Smiling
Ours
CelebA: Smiling
XGEM
Ours
L
L
L
XGEM
Young
CelebA: Young
XGEM
Healthy
Cardiomepaly
Ours
X-Ray: Cardionegaly
XSEM
Ours
XGEM
Figure 7: Visual explanations generated for three prediction tasks on CelebA dataset. The first column shows the query image, followed by the corresponding generated explanations.
17

---

Published as a conference paper at ICLR 2020
Query Image
Generated Visual Explanations
Desired fix)
(0.0-0.1) [0.1-0.2] (0.2-0.3) (0.3-0.4) (0.4-0.5) (0.5-0.6) [0.6-0.7] (0.7-0.8) (0.8-0.9) [0.9-01.]]
Not-smiling
Smillng
10
Exal=004
0.22
0.39
0.47
0.63
0.84
0.82
0.95
0.91
1.0
CelebA: Smiling
fix--0.2
A.3-0000
Old
Young
Ax1=0.96
Axal=0.11
0.46
CelebA: Young
2x1m0.16
Axel=0.07
0.28
0.41
0.
0.56
36
0.72
0.79
0 0.85
0.93
Beard
No-Ben No-Beard
Axx)-014
0.58
CelebA: No Beard
fixi=0.0
0.0%
0.15
0.57
0.87
0.96
Light Makeup
Heavy Makeup
CelebA: Heavy Makeup
Gx1=0.73
0.14
0.37
0.42
0.53
0.64
0.85
Ki 24
nxc1=0.06
0.17
0..66
0.52
0.66
0.75
0.84
0.96
0.99
Black Hair
Black Hair 0.99
8.1=10
#0x)=0
0.25
0.5
0.66
0.64
0.78
0.84
0.92
0.96
CelebA: Black Hair
fxx=00122
mxJ=0.05
No
Bangs 1.0
1.0
nx01-0033
0.19
0.28
0.51
0.64
0:77
0.91
0.96
0.99
CelebA; Bangs
=000
0.07
0.12
0.37
0.69
0.77
0.85
0
..99
0.93
0.95
Figure 8: Visual explanations generated for six prediction tasks CelebA dataset. The first column shows the query image, followed by the corresponding generated explanations. The values above each image the output of classifier f.
18

---

Published as a conference paper at ICLR 2020
1.0
CelebiNNo Beard)
CelebN/HHevyy Mol
CelebA/Blaciddar)
CelebA[Bangs)
0.0
Query flx)
0.6
0.0-0.2
0.2..44
0.4
0.4-0.6
0.2
0.4
0.6
0.8
1.0
0.4
[x) 4
750
Figure 9: Plot of the expected outcome from the classifier, f(x) + 6, against the actual response of the classifier on generated explanations, f(xs). The monotonically increasing trend shows a positive correlation between f(x) + and / (xs), and thus the generated explanations are consistent with the expected condition.
Task-1 (Age)
Task-2 (identify Target Class)
which Image the person looking young?
changing images?
Select option
Select option
HI
Age
(Bange)
Task-1 (Smile)
Task-3 (Bias Detection)
which Image the person smiling more?
What is changing the images
Select an option
Select option
Gender
Smile and Gender
Not
The interface for the human evaluation done using Amazon Mechanical Turk (AMT). Task-1 evaluated humans can detect the relative order between two explanations. Task-2 evaluated if humans identify the target class for which model has provided the explanations. Task-3 demonstrated that our model can help the user to identify problems like possible bias in the black- box training.
Figure 10:
0.92
0.08
0.07
0.04
0.08
0.8
Smillng
0.11
0.89
0.08
0.04
0.07
Targel Altribute
0.6
Blockrlair
0.16
0.18
0.58
0.08
0.16
0.4
No- Re-eterd
0.13
0.1
0.07
0.46
0.04
0.2
Jangs
0.15
0.08
0.17
0.07
0.32
0.0
Young
Smillng
BlackMair
No-Beard
Baris
Figure 11: Each cell is the fraction of the generated explanations, that have flipped in source attribute as compared to the query image. The x-axis is source attribute and y-axis is the target attribute for which explanation is generated. Note: This is not confusion matrix.
19

---

Published as a conference paper at ICLR 2020
10
Asss O, Ar 1,vvc 100
Aveuu A LAnc 1.00
Assan 10, Arw 1. 100
Asseu 100, Aym 1.Acc 1.0
0.
0.6
663
0.4
0.0
02
04
0.6
0.8
1.0
0.0
0.2
0.8
1.0 0.0
Anay Jy D.Ace
100
Anan
12
Query f(x)
0.0-0.2
0.4-0.6
0.6-0.8
0.8-1.0
1,00.0
0.2
0.6
0.6
0.8
1.0
0.0
n2
86
0.8
1.000
02
04
0.6
0.8
1.0
+
Rx1+0
Rxt
100
1.4
Araay Arm L.vee
Aspru 1, Aym 1.
100
3.0
0.4
0.0
1.0 0.0
0.2
0.4 0.6
0.8
1.0
0.0
02
3.4 3.6
0.8
3.0
[x)
0x)
100
Figure 12: Ablation study to show the effect of KL loss term. Plot of the expected outcome from the classifier, f(x) + 5, against the actual response of the classifier generated explanations, f(xs).
20


Reviews:
[{"review_id": "rJgq6kTS5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for explaining the output of black box classification of images.  The method generates  gradual perturbation of outputs in response to gradually perturbed input queries. The rationale is that, by looking at these, humans can interpret the classification mechanics.\n\nThe presentation is clear. The coverage of prior work is sufficient (although references should point to the published work instead of arxiv entries, when the former is available).\n\nOne question that is not addressed is how efficient is this method, in terms of computational cost. This is a method that increases the amount of input data (through perturbation). What is the minimum amount of input data that needs to be perturbed in this way, before the method can become human interpretable? \n\nAlso, ideally any work on human interpretability of ML should be evaluated on humans. If not, it is an approximation, and it should be presented and reasoned as such (with a discussion of limitations and caveats, for instance).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "H1eKOkenFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Here are the claims I could find in the intro:\n\"Given a query input to a black-box, we aim at explaining the outcome by providing plausible and progressive variations to the query that can result in a change to the output\"\n > This is well supported as the model generates these and it is very reasonable that it can.\n\"the counterfactually generated samples are realistic-looking\"\n> The images seem to support this.\n\"the method can be used to detect bias in training of the predictor\"\n> Section 4.4 makes it really clear that, at least in the described setting, it works.\n\nI think the idea could be presented in a better way. The general concept of exaggerating a feature that represented a class seems novel and exciting. Just based on the novelty of that alone I think this is worth accepting. I would imagine there would be a cleaner way of achieving all this but maybe it is all necessary.\n\nI don't understand Figure 1a. I don't think this helps to illustrate the point. M_z seems to just be a bottleneck but the writing makes it seem like it is more.\n\nSection 4.2 is a bit hard to read. It is not clear for me what is the goal of this section.\n\nSection 4.4 seems very similar to the idea in this work https://arxiv.org/abs/1805.08841 which studied how bias in CycleGANs can be seen when you vary the bias which I think should be cited here.\n\nTypos:\n\"our application of interested\"\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an idea for interpolating between two points in the decision-space of a black-box classifier in the image-space, while producing plausible images along the interpolation path. The presentation is clear and the experiments support the premise of the model.
While the proposed technique can be used to help understanding how a classifier works, I have strong reservations in calling the generated samples "explanations". In particular, there is no reason for the true explanation of how the classifier works to lie in the manifold of plausible images. This constraint is more of a feature to please humans rather than to explain the geometry of the decision boundary.
I believe this paper will be well-received and I suggested acceptance, but I believe it will be of limited usefulness for robust understanding of the decision boundary of classifiers.
Decision: Accept (Spotlight)