DISSECT: Disentangled Simultaneous Explanations via Concept Traversals
Authors: Asma Ghandeharioun
Been Kim
Chun-Liang Li
Brendan Jou
Brian Eoff
Rosalind Picard
Venue: ICLR 2022 Poster (2022)
Keywords: Explainability
Interpretability
Counterfactual generation
Generative Adversarial Network
Variational Autoencoder

Abstract: Explaining deep learning model inferences is a promising venue for scientific understanding, improving safety, uncovering hidden biases, evaluating fairness, and beyond, as argued by many scholars. One of the principal benefits of counterfactual explanations is allowing users to explore "what-if" scenarios through what does not and cannot exist in the data, a quality that many other forms of explanation such as heatmaps and influence functions are inherently incapable of doing. However, most previous work on generative explainability cannot disentangle important concepts effectively, produces unrealistic examples, or fails to retain relevant information. We propose a novel approach, DISSECT, that jointly trains a generator, a discriminator, and a concept disentangler to overcome such challenges using little supervision. DISSECT generates Concept Traversals (CTs), defined as a sequence of generated examples with increasing degrees of concepts that influence a classifier's decision. By training a generative model from a classifier's signal, DISSECT offers a way to discover a classifier's inherent "notion" of distinct concepts automatically rather than rely on user-predefined concepts. We show that DISSECT produces CTs that (1) disentangle several concepts, (2) are influential to a classifier's decision and are coupled to its reasoning due to joint training (3), are realistic, (4) preserve relevant information, and (5) are stable across similar inputs. We validate DISSECT on several challenging synthetic and realistic datasets where previous methods fall short of satisfying desirable criteria for interpretability and show that it performs consistently well. Finally, we present experiments showing applications of DISSECT for detecting potential biases of a classifier and identifying spurious artifacts that impact predictions.

Published as a conference paper at ICLR 2022
DISSECT: DISENTANGLED SIMULTANEOUS EXPLA- NATIONS VIA CONCEPT TRAVERSALS
Asma Ghandeharioun
Been Kim
Google Research / MIT
Google Research
aghandeha aghandeharioun@eeee rinnerroourn@gggoeeeoooooooo
beenkim@googl e
Chun-Liang Li, Brendan Jou, Brian Eoff
Rosalind W. Picard
Google Research
MIT
fchunliang, bjou, wwo.f/eooogge..ccom
picard@media mit edu
ABSTRACT
Explaining deep learning model inferences is a promising venue for scientific un- derstanding, improving safety, uncovering hidden biases, evaluating faimess, and beyond, as argued by many scholars. One the principal benefits of counterfac- tual explanations is allowing users explore "what-if" scenarios through what does not and cannot exist in the data, quality that many other forms of explana- tion such as heatmaps and influence functions are inherently incapable doing. However, most previous work on generative explainability cannot disentangle im- portant concepts effectively, produces unrealistic examples, or fails to retain rel- evant information. We propose novel approach, DISSECT, that jointly trains generator, a discriminator, and concept disentangler overcome such chal- lenges using little supervision. DISSECT generates Concept Traversals (CTs), defined as sequence of generated examples with increasing degrees concepts that influence classifier's decision. By training generative model from a clas- sifier's s signal, DISSECT offers a way to discover a classifier's inherent "notion" of distinet concepts automatically rather than rely user-predefined concepts. We show that DISSECT produces CT's that (1) disentangle several concepts, (2) are influential
classifier's decision and coupled to its reasoning due joint training (3), are realistic, (4) preserve relevant information, and (5) are stable across similar inputs. We validate DISSECT several challenging synthetic and realistic datasets where previous methods fall short of satisfying desirable criteria for interpretability and show that performs consistently well. Finally, we present experiments showing applications of DISSECT for detecting potential biases of a classifier and identifying spurious artifacts that impact predictions.
INTRODUCTION
Explanation of the internal inferences of deep learning models remains challenging problem that many scholars deem promising for improving safety, evaluating fairness, and beyond [13, 16, 23, 49]. Many efforts in explainability methods have been working towards providing solutions for this challenging problem. One way to categorize them is by the type of explanations, some post hoc techniques focusing on the importance of individual features, such as saliency maps [19, 47, 76, 79], some on importance of individual examples [33, 34, 39, 87], some on importance of high- level concepts [35]. There has been active research into the shortcomings of explainability methods explanation [84].
(e.g. [1, 30, 61, 70, 84]) and determining when attention can be used
While these methods focus on information that already exists in the data, either by weighting fea- tures or concepts in training examples or by selecting important training examples, recent progress in generative models [12, 29, 36, 40, 45] has lead to another family of explainability methods that provide explanations by generating new examples or features [14, 32, 68, 73]. New examples or features can be used to generate counterfactuals [82] allowing users to ask: what if this sample were to be classified as the opposite class, and how would it differ? This way of explaining mirrors the

---

Published as a conference paper at ICLR 2022
(a) Heatmaps
(b)
(c)
(d)
(0)
Query: Benign
Segmentation Masks
Retrieval Based
Counterfactual Generation
DISSECT: Multiple
Counterfactual Generations
Figure I: Examples of explainability methods applied to a melanoma classifier. Explanation by (a) heatmaps (e.g. [19, 47, 76, 79]). (b) segmentation masks (e.g. [22, 69)), (c) sample retrieval (e.g. 172D, (d) counterfactual generation (e.g- [68. 73)), (e) and multiple counterfactual generations such as DISSECT. Multiple counter- factuals could highlight several different ways that changes in skin lesion could reveal its malignancy and overeome some of the blind spots
other approaches. For example, they can demonstrate that large le- sions, jagged borders, and asymmetrical shapes lead to melanoma classification. They can also show potential biases of the classifier by revealing that surgical markings can spuriously lead to melanoma classification.
way humans reason, justify decisions [3], and learn [4, 6, 83]. Additionally, users across visual, auditory, and sensor data domains found examples the most preferred means of explanations [31].
To illustrate the benefits of counterfactual explanations, consider dermatology task where an ex- planation method used to highlight why certain sample is classified as benign/mallinnat (Fig. 1). Explanations like heatmaps, saliency maps, or segmentation masks only provide partial information. Such methods might hint at what is influential within the sample, potentially focusing on the lesion area. However, they cannot show what kind of changes in color, texture, inflammation could trans- form the input at hand from benign to malignant. Retrieval-based approaches that provide examples that show concept
not enough for answering "what-if" questions either. A retrieval- -bsed tech- nique might show input samples of malignant skin lesions that have similarities to a benign lesion in patient A. but from different patient B. potentially from another body part even different skin tone. Such examples do not show what this benign lesion in patient A would have to look like if it were classified malignant instead. On the other hand, counterfactuals depict how to modify the input sample to change its class membership. A counterfactual explanation visualizes what a malignant tumor could look like in terms of potential color texture, inflammation changes on the skin. Better yet, multiple counterfactuals could highlight several different ways that changes in skin lesion could reveal its malignancy. For classifier that relies on surgical markings [86] as well as meaningful color/texture/size changes to make its decision, single explanation might fail to reveal this flaw resulting from dependence on spurious features, but multiple distinct explana- tions shed light on this phenomenon. Multiple explanations are strongly preferred in several other domains, such as education, knowledge discovery, and algorithmic recourse generation [17, 54].
In this work, we propose generation-based explainability method called DISSECT that generates Concept Traversals sequeenes of generated examples with increasing degrees of concepts' influence a classifier's decision. While current counterfactual generation techniques fail sat- isfy the most consistently agreed-upon properties desired for an explainability method simultane- ously [50, 60, 68, 73, 74], DISSECT aims overcome this challenge. CT's are generated by jointly training a generator, a discriminator, and a CT disentangler to produce examples that (1) express one distinct factor [50]
time; (2) are influential [73, 74] to a classifier's decision and are coupled to the classifier's ' reasoning, due to joint training; (3) are realistic [73]: (4) preserve relevant informa- tion [50, 60, 68]; and (5) are stable across similar inputs [21, 50, 60]. Compared other approaches that require a human to identify concepts priori and find samples representing them, e.g. [9, 35], DISSECT uncovers them automatically. We compare DISSECT with several baselines, some of which have been optimized for disentanglement, some used extensively for explanation, and some that fall between. DISSECT is the only technique that performs well across all these dimensions. Other baselines either have hard time with influence, lack fidelity, generate poor quality and un- realistic samples, or do not disentangle properly. We evaluate DISSECT using 3D Shapes [7], CelebA [44], and new synthetic dataset inspired by real-world dermatology challenges. We show that DISSECT outperforms prior work in addressing all of these challenges. We discuss applications to detect a classifier's 's potential biases and identify spurious artifacts using simulated experiments.
This paper makes five main contributions: 1) presents a novel counterfactual explanation approach that manifests several desirable properties outperforming baselines; 2) demonstrates applications
2

---

Published as a conference paper at ICLR 2022
through experiments showcasing the effectiveness of this approach for detecting potential biases of a classifier; 3) presents a set of explainability baselines inspired by approaches used for generative disentanglementt 4) translates desired properties commonly referred to in the literature across forms of explanation into measurable quantities for benchmarking and evaluating counterfactual genera- tion approaches; 5) releases new synthetic dermatology dataset inspired by real-world challenges.
RELATED WORK
Our method relates the active research of post hoc explainability methods. One way to categorize them is by explanation form. While met with criticisms [1. 75], many feature-based explainability methods are shown to useful [47, 79J, which assign a weight to each input feature to indicate their importance in classification [10, 57]. Example-based methods are another popular category [33, 34, 39, 87] that instead assign importance weights to individual examples. More recently, concept-basee methods have emerged that attribute weights to concepts, i.., higher-level representations of features [22, 25, 35] such as "long hair".
Our work leverages recent progress in generative modeling, where the explanation is presented through several conditional generations [52, 56]. Efforts for the "discovery" of concepts are also related to learning disentangled representations [12, 29, 35, 36], which has been shown to be chal- lenging without additional supervision [45]. Recent findings suggest that weak supervision in the following forms could make the problem identifiable: how many factors of variation have changed [71], using labels for validation [46]. or using sparse labels during training [46]. While some tech- niques like Conditional Subspace VAE (CSVAB) [38] began to look into conditional disentangle- ment by incorporating labels, their performance has not yet reached their unconditional counterparts. Our work uses new form of weak supervision to improve upon existing methods: the posterior probability and gradient of the classifier-under-esss.
Many explainability methods have emerged that use generative models modify existing exam- ples or generate new examples [14, 15, 26, 32, 48, 68, 73). Most of these efforts use pre-trained generators and generate new example by moving along the generator's embedding. Some aim to generate samples that would flip the classifier's 's decision [15, 32], while others aim to modify particular attribution of the image and observe the classifier's decision change [14]. More recent work allows the classifier's predicted probabilities ing training [68], and visualize the continuous progression
gradients to flow through the generator dur-
key differential feature [73]. This progressive visualization has additional benefits, as Cai et al. [9] showed that pathologists could iterate more quickly using a tool that allowed them to progressively change the value of a concept, compared to purely using examples. Additionally, continuous exaggeration of a concept facilitated thinking more systematically about the specific features important for diagnosis.
Most of these approaches assume that there is only one path that crosses the decision boundary, and they generate examples along that path. Our work leverages both counterfactual generation techniques and ideas from the disentanglement literature to generate diverse sets of explanations via multiple distinct paths influential to the classifier's decision, called Concept Traversals (CT). Each CT is made of sequence of generated examples that express increasing degrees of a concept.
METHODS
Our key innovation is developing an approach to provide multiple potential counterfactuals. We show how to successfully incorporate the classifier's signal while disentangling relevant concepts. We review existing generative models for building blocks of our approach, and follow Singla et al. [73] for DISSECT's backbone. This choice is informed by the good conditional generation capa- bility of [73] and being strong interpretability baseline. We then introduce a novel component for unsupervised disentanglement of explanations. Multiple counterfactual explanations could re- veal several different paths that flip the classification outcome and shed light on different biases of the classifier. However, single explanation might fail to reveal such information. Disentangling distinct concepts without relying on predefined user labels reduces the burden on the user side. Ad- ditionally, it can surface concepts that the user might have missed otherwise. However, unsupervised disentanglement is particularly challenging [45]. To address this challenge, we design a disentan- gler network. The disentangler, along with the generator, indirectly enforces different concepts to
3

---

Published as a conference paper at ICLR 2022
be distinct possible. This section summarizes our final design choices. For more details about baselines, DISSECT, and ablations studies, see $A.1, $A.2, and $A.6, respectively.
Notation. Without loss of generality, we consider binary classifier f: X + Y = (-1,1} such that f(z) = ply = 1|z) where x ~ Px. We want to find K latent concepts that contribute to the decision-making of f- Given Σ, a E [0, 1]. and k, want generate image, by perturbing the latent concept kt such that the posterior probability f()) = a. In addition, when conditioning on and a concept by changing a, we hope for a smooth change in the output image σ. This smooth- ness resembles slightly changing the degree of a concept. Putting it together, we desire a generic gen- eration function that generates I, defined as ((z, a, k; f) : X x [o, 1] x {0, 1......., K 1} + X, where (G((z, k; f)) x a. The generation function G conditions on x and manipulates the con- cept such that monotonic change the ((g( 0, :.).)..)))(.. (z 1, k)) achieved.
3.1 ENCODER- DEDDDR ARCHITECTURE
We realize the generic (conditional) generation process using an encoder-decoder framework. The input is encoded into an embedding (xx)), before feeding into the generator G, which serves as a decoder. In addition to E(x), we have a conditional input c(a, k) for the kth latent concept is the desired prediction of f we want to achieve in the generated sample by perturbing the hhth latent concept. The generative process G can be implemented by G(E(z)) c(a, k)), where we can manipulate e for interpretation via conditional generation.
are going manipulate with level
There are many advances in generative models, which can be adopted to train G(E(z), c(a,,))) For example, Variational Autoencoders (VAE) explicitly designed for disentanglement [11, 29, 38, 41]. However, VAE-based approaches often generate blurry images which requires customized architec- ture designs [63, 81]. In contrast, GANs [24] tend to generate high-quality images. Therefore, we consider an encoder-decoder design with GANs [5, 55, 58, 64, 66, 89] to have flexible controllability and high quality generation for explainability.
There are different design choices for realizing the conditioning code c. A straightforward option is multiplying a K --dimensional one-hot vector, which specifies the desired concept k, with the proba- bility a. However, conditioning on continuous variables is usually challenging for conditional GANs compared with their discretized counterparts [52]. Therefore, we discretize a into [0, 1/N,.... 1}, and parametrize e as binary matrix [0, 1](N+1)*K for N + 1 discrete values and concepts, fol- lowing Singla et al. [73]. Therefore, c(a, k) denotes that we set the elements (m, k) to be 1, where m is an integer and m/N S a < (m + 1)/N. See & A.2 for more details.
We include the following elements in our objective function following Singla et al. [73], and further modify them to support multiple concepts:
Interpreting classifiers. We align the generator outputs with the classifier being interpreted !: (G) = a log (f, (z, k)))) + (1- a) log (1 - (G (a,, c(a, k)))
Reconstruction. We adopt & reconstruction loss in our encoder-decoder design [55, 89), Given x ~ Px and classifier f, the ground-truth for all concepts is f (x):
K
GG) = 1
Σ k-O
(()))(f()))))))]
(2)
Cycle consistency. Define Fa,k = G(r,, c(a, k)).
We add a cycle consistency loss [89]:
| G(E,, (f(x), k))],
(C)
E.
(3)
GAN loss. We use spectral normalization [53] with its hinge loss variant [52] to compare the distribution ~ Px and the generated data:"
LonN(D)= -ENPx min -0. (0. D(=))
Eewpy Ban Bu.u
min (eG,-D (O-
(4)
LOGNN(G) = - -EuwPx Ba. ((G(a)c(a))))
(5)
Note that trained having access to f's signal. The Loglicas between the desired probability a and the predicted probability of the perturbed image is minimized, Thus, the model supports fidelity by design. One can use any GAN loss [2, 24, 27, 42, 53, 88]. Generated data comes from uniformly sampling k & a.
4

---

Published as a conference paper at ICLR 2022
3.2 ENFORCING DISENTANGLEMENT
Ideally, want the K concepts capture distinctively different qualities influential the classifier being explained. Though G is capable of expressing K' different concepts, the above formulation does not enforce distinctness among them. To address this challenge, we introduce a concept disen- tangler Rt inspired by the disentanglement literature [8, 11, 12, 29, 38, 41, 43]. Most disentangle- ment measures quantify proxies of distinctness of changes across latent dimensions and similarity of changes within latent dimension [45]. This is usually achieved through latent code traversal, keeping a particular dimension fixed, or keeping all but one dimension fixed. Thus, we design R to encourage distinctness across concepts and similarity within concept.
The K concepts can be viewed as "knobs" that can modify a sample along a conceptual dimension. Without loss of generality, we assume modifying only one "knob" at time. Given pair of images (ar, x'). R tries to predict which concept k, k E {O...... K 1}. has perturbed query æ to produce x'. To formalize this, let Tk G(I, c(a, k)) and Tg G(ER,a e(f(x), k)). The loss function is
C, (G, R) CIE(e,, R(x, + CE(CK: R(xx,
(6)
where CE is the cross entropy and eμ is one-hot vector of size K for the hthh concept. Note that while the ftth "knob" can modify a sample query with a degree, a [0, 1/N1/...... 1]. R tries to solve a multi-class binary prediction problem. Thus, for any a, a # f(z), R(x, 5x) is trained to predict a one-hot vector for the hth concept. Furthermore, note that no external labels are required to distinguish between concepts. Instead, by jointly optimizing R and G. we penalize the generator G if any concept knob results in similar- moriar- samples other concept knobs. Encouraging noticeably distinct changes across concept knobs indirectly promotes disentanglement. In summary, the overall objective function of our method is:
min max G, D
GG) A Ancc A, (G, (7)
(D.)
We call our approach DISSECT as it disentangles simattaneoussupllaaaaions via goncept traversals.
EXPERIMENTS
3D Shapes. We first use 3D Shapes [7] purely for validation and demonstration purposes due to the controllability of all its factors. is a synthetic dataset available under Apache License 2.0 composed of 480K 3D shapes procedurally generated from 6 ground-truth factors of variation. These factors are floor hue, wall hue, object hue, scale, shape, and orientation.
Second, we create a new dataset, SynthDerm (Fig. 2). Real-world characteris- ties of melanoma skin lesions in dermatology settings inspire the generation of this dataset [80]. These characteristics include whether the lesion is asymmetrical, its border is irregular jagged, is unevenly colored, has a diameter more than 0.25 inches, or is evolving in size, shape, or color over time. These qualities are usually referred as the ABCDE of melanoma [65]. We generate SynthDerm algorithmically by varying several factors: skin tone, lesion shape, lesion size, lesion location (vertical and horizontal), and whether there are surgical markings present. We randomly assign one of the following to the lesion shape: round, asymmetrical, with jagged borders, or multi- colored (two different shades of colors overlaid with salt-and-pepper noise). For skin tone values, we simulate Fitzpatrick ratings [20]. Fitzpatrick scale is commonly used approach to classify the skin by its reaction to sunlight exposure modulated by the density of melanin pigments in the skin. This rating has six values, where 1 represents skin with lowest melanin and 6 represents skin with highest melanin. For our synthetic generation, we consider six base skin tones that similarly resem- ble different amounts of melanin. We also add a small amount of random noise to the basc color to add further variety. SynthDerm rm includes more than 2,600 64x64 images.
SynthDerm.
CelebA. We also include the CelebA dataset [44] that is available for non-commercial research purposes. This dataset includes "identities [of celebrities] collected from the Internet" [78], is re- alistic, and closely resembles real- world settings where the attributes are nuanced and not mutually independent. Celeba includes more than 200K images with 40 annotated face attributes, such as smiling and bangs.
5

---

Published as a conference paper at ICLR 2022
Fitapatrick Scale Samples Characteristics
Fitepatrick Scale Samples Charackaristics
Benign
IV
Jagged borders
II
Asymmetrionl
V
Uneven colors
Asymmetrieal with surgica markings
VI
than 0.25*
Figure 2: Illustration of SynthDerr dataset that algorithmically generated. Fitzpatrick scale of skin classification based on melanin density and corresponding examples the dataset are visualized.
4.1 BASELINES
We consider several baselines. First, we modify a set of VAEs explicitly designed for disentan- glement [8, 11, 29, 41] incorporate the classifier's signal during their training processes and en- courage the generative model to learn latent dimensions that influence the classifier, i.e., learning Important CTs. While these approaches are naturally designed to explain external classifier, they have been designed to find distinct factors of variation. We refer to these baselines with mod postfix, e.g. 3-VAE-mod. Second, we include CSVAE [38] that aims solve unsupervised learning of features associated with specific label using low-dimensional latent subspace that can be independently manipulated. CSVAE, without any modification, the closest alternative in the literature that tries to discover both Important and Distinct CTs. Third, we include Explanation by Progressive Exaggeration (EPE) [73]. GAN-based approach that learns to generate one series of counterfactual and realistic samples that change the prediction of f. given data and the classifier's signal. EPE is the most similar baseline to DISSECT in terms of design choices; however, it is not designed to handle distinct CTs. We introduce EPE-mod by modifying EPE to allow learning multiple pathways by making the generator conditional on the CT dimension (more details in $A.1).
4.2 EVALUATION METRICS
This section provides a brief summary of our evaluation metrics. We consider Importance, Real- ism, Distinctness, Substiutaaillity and Stability, which commonly appear as desirable qualities in explainability literature." See more in 5A.3.
Importance. Explanations should produce the desired outcome from the black-box classifier f. Pre- vious work has referred to this quality using different terms, such as importance [22], compatibility with classifier [73], and classification model consistency [74]. While most previous methods have relied on visual inspection, we introduce quantitative metric to measure the gradual increase of the target class's 's posterior probability through a CT. Notably, we compute the correlation between a and f[1(x, a, k; f)) introduced in $ 3. For brevity, refer to f([(x, der of the paper. We also report the mean-squared error and the Kullback-Leibe between and f We also calculate the performance of f on the counterfactual explanations. Specifically, we replace the test set of real images with their generated counterfactual explanations and quantify the performance of the pre-trained black-box classifier f the counterfactual test set. Better performance suggests that counterfactual samples are compatible with f and lie on the cor- rect side of the classifier's boundary. Note that Importance scores can be calculated per concept and provide ranking across concepts, or they can be aggregated to summarize overal performance. See 5A.7 for more information about concept ranking using individial concept Importance scores. Realism. We need the generated samples that form a CT to look realistic to enable users to identify concepts they represent. means the counterfactual explanations should lie on the data manifold.
k; f)) (ā) (z)) in the remain- (KL) divergence
This quality has been referred to realism or data consistency [73]. We use Fréchet Inception Distance (FID) [28] as a measure of Realism. Additionally, inspired by [18], we train post hoc classifier that predicts whether sample is real or generated. Note that is essential do this step post hoc and independent from the training procedure because relying on the discriminator's accu- racy in adversarial training framework Distinctness. Another desirable quality for explanations to represent inputs with non-overlapping concepts, often referred as diversity [50]. Others have suggested similar properties such as co-
be misleading [18].
Fgg. 1 encourages Importance, Fq. 5 encourages Realism, Fq. 6 encourages Distinctness, Eq. and Eq. 3 indirectly help with Stability. All the elements of Eq. contribute improving Substitutability.

---

Published as a conference paper at ICLR 2022
Queev
EPS-aad
Query
DISSECT
DISSECT
Plana
Crant
Examples from 3D Shapes. EPE and mod converge to finding the same concept, despite EPE-mod's ability to express multiple pathways to switch classifier outcomes from False to True. DISSECT discovers the two ground-truth concepts: CT₁ flips the floor color to cyan and CT2 flips the shape color to red.
Figure 3:
Quantitative results on 3D Shapes. DISSECT performs significantly better par with the strongest baselines in all evaluation categories. Modified VAE variants perform poorly in terms of Impor- tance, worse than CSVAE, and significantly worse than EPE, EPE-mod, and DISSECT. CSVAE and other VAE variants do not produce Sugh-tuaitty images, thus have poor Realism scores; meanwhile, EPE, EPE-mod, and DISSECT generate realistic samples indistinguishable from real images. While the aggregated metrics for Importance order across EPE, EPF-mod, and DISSECT. Our approach greatly improves Distinctmess, especially compared to EPF-mod. EPE inherently incapable of doing this, and the extension EPFE-mod does, but poorly. For the Substituiability scores, note the classifier's precision, recall, and accuracy when training on actual data is 100%. (*Some methods only generate samples with f()) = 0.0. Correlation with constant value is undefined.)
Table 1:
useful for discarding VAE baselines with poor performance, they do not show consistent
RT.
MSE
TCF
Axc
Pax
TCF
Ro
Amo
PPee
Ree
Ace
Piaz
Ieeami
F5a
Rc:
(seau)
Rs:
Sab
Sab
Sub
MSE
150
601
6m
670
300
633
66
423
023
wmb
82
191
2197
Ney
K0w
6m
200
01
10
100
100
209
133
103
130
353
131
-01
X175
63
323
103
10
16
w7
109
100
5127
11000
135
10301
100
19
100
11000
herency, meaning examples of a concept should be similar but different from examples of other concepts [22]. To quantify this in a counterfactual setup, we introduce distinctness measure cap- turing the performance of secondary classifier that distinguishes between CTs. We train classifier post hoc that given a query image x. generated image x', and K number of CTs, predicts CT& that has perturbed x to produce x', k E {1,..... .......K) K).
Substitutability. The representation of sample in terms of concepts should preserve relevant in- formation [50, 60). Previous work has formalized this quality for counterfactual generation contexts through a proxy called substitutability [68]. Substitutability measures an external classifier's perfor- mance real data when it is trained using only synthetic images.* 4 A higher substitutability score suggests that explanations have retained relevant information and are of high quality.
Stability. Explanations should be coherent for similar inputs, quality known as stability [21, 50, 60]. To quantify stability in counterfactual explanations, we augment the test set with additive random noise to each sample æ and produce S randomly augmented copies x;- Then, we generate counterfactual explanations T and 7,, respectively. We calculate the mean-squared dif- ference between counterfactual images F and & and the resulting Jensen Shannon distance (JSD) between the predicted probabilities f f(z) and f(z").
4.3 CASE STUDY I: VALIDATING THE QUALITIES OF CONCEPT TRAVERSALS
Considering 3D Shapes [7], we define an image as "colored correctly" if the shape hue is red or the floor hue is cyan. We train a classifier to detect whether a sample image is "colored correctly" or not. In this synthetic experiment, only these two independent factors contribute to the decision of this classifier. Given a not "colored correctly" query, we would like the algorithm find CT related to the shape color and another CT associated with the floor color-two different pathways that lead to switching the classifier o
*This metric has been used in other contexts outside of explainability and has been called Classification Accuracy Score (CAS) [62]. CAS is more broadly applicable than Fréchet Inception Distance [28] & Inception Score [67] that only useful for GAN evaluation.
In this scenario, these two ground ground-truth concepts do not directly apply to switching the classifier outcome from True to False. For example, if an image has a red shape and a cyan floor, both of these colors need to be changed to switch the classification outcome. We still observe that DISSECT finds CT's that change different combinations of colors, but the baseline methods converge the same CT. See 5A.5. for more details.
7

---

Published as a conference paper at ICLR 2022
LaPU-moo
DISSUCT
Query FP
Query
Cowrr
Ciiler
Booder" Anger
No change
Coloe
Diameter
Bunder"
Booder
Suzaical
Cooor
Figure 4: Examples from SynthDerm comparing DISSECT with the strongest baseline, EPE-mod. We illustrate three queries with different Fitzpatrick ratings [20] and visualize the two most prominent concepts for each technique. We observe that EPE-mod converges on a single concept that only vaguely represents meaningful ground-truth concepis. However, DISSECT successfully finds concepts describing asymmetrical shapes, jagged borders, and uneven colors that align with the ABCDE of melanoma [65]. DISSECT also identifies concepts surgical markings that spuriously impact the classifier's decisions.
Table 2: Quantitative results on SynthDerm. DISSECT performs consistently best in all categories. For anchoring the Substinutahility scores, note that the precision, recall, and accuracy of the classifier when training on actual data 97.7%, 100.0%, and 95.4%, respectively.
Relbo
Dhitkteco
1
1M88
TCF
TCF
TC
LPDD
Hec
1.0
PIc:
Bes
Fux
Rac
1235
1131
1.78
011
325
318
16
351
200
15.1
0705
BFEE
1087
0225
13
010
80.
30
863
238
351
15.1
833
80
6.11
0/01
176
asst
072
092
008
243
369
351
111
163
07
743
607
0.339
6002
Tab. summarizes the quantitative results on 3D Shapes. Most VAB variants perform poorly in terms of Importance, CSVAE performs slightly better, and EPE, EPE-mod, and DISSECT perform best. Our results suggest that DISSECT performs similarly to EPE that has been geared explic- itly toward exhibiting Importance and its extension, EPE-mod. Additionally, DISSECT still keeps Realism intact. Also, it notably improves the Distinctness of CTs compared to relevant baselines.
shows the examples illustrating the qualitative results for EPE, EPE-mod, and DISSECT. Our results reveal that EPE converges to finding only one of these concepts. Similarly, both CTs generated by EPE-mod converge to finding the same concept, despite being given the capability to explore two pathways to switch the classifier outcome. However, DISSECT finds the two distinct ground-truth concepts through its two generated CTs. This is further evidence for the necessity of the disentanglement loss to enforce the distinctness of discovered concepts. For brevity, three sample queries are visualized (See more $A.5.1). Note that given the synthetic nature this dataset, the progressive paths may not be informative. Thus, we only visualize the far end of the spectrum, i.c., the example that flips the classification outcome all the way. See $4.5 for the progressive exaggeration of concepts in more nuanced experiment.
Fig.
4.4 CASE STUDY II: IDENTIFYING SPURIOUS ARTIFACTS
A "high performance" model could leam to make its decisions based on irrelevant features that only happen to correlate with the desired outcome, known as label leakage [85]. One the applica- tions of DISSECT is to Motivated by real-world examples that revealed classifier dependency tifying melanoma [86], we design this experiment. Given the synthetic nature of SynthDerm and how it has been designed based on real-world characteristics of melanoma [65, 80], each sample has deterministic label of melanoma or benign. If the image asymmetrical, has jagged borders, has different colors represented by salt-and- pepper noise, or has large diameter (i.e., does not fit a 40 square), the sample melanoma. Otherwise, the image represents the benign class. Similar to in-situ dermatology images, melanoma samples have surgical markings more frequently than benign samples. We train a classifier to distinguish melanoma vs. benign lesion images.
such spurious concepts and allow probing black-box classifier.
surgical markings in iden-
Given benign query, would like to produce counterfactual explanations that depict how to modify the input sample to change its class membership. We want DISSECT to find CTs that disentangle meaningful characteristies of melanoma identification in terms of color, texture, or shape [65], and identify potential spurious artifacts that impact the classifier's predictions.
Tab. 2 summarizes the quantitative results SynthDerm. Our method performs consistently well across all the metrics, significantly boosting Distinctness and Substitutability scores and making meaningful improvements Importance scores. Our approach has higher performance compared
8

---

Published as a conference paper at ICLR 2022
EESARCT
Crrstated Gernctrath
Cencptt::
Sealing
Figure 5: Examples from Ce CelebA. A biased classifier has been trained to predict smile probability, where a training dataset was sub-sampled so that smiling co-occurs only with "bangs" and "blond hair" attributes. EPE does not support multiple CTs. EPE-mod converges on the same concept. despite having the ability to express various pathways change (a) through CT, and CT2. However, DISSECT discovers distinct pathways: CTi mainly changes hair color to blond, and CT2 does not alter hair color but tries to add bangs.
Table 3: Quantitative results CelebA, DISSECT performs hetter than or par with the baselines all categories. Notably, DISSECT greatly improves the Distinctness of CTs and achieves a higher Realism score, suggesting disentangling CTs does not diminish the quality of generated images and may even improve them. The classifier's precision, recall, accuracy when training on actual data is 95.4%, 98.6%, 92.7%, respectively.
8
1M
KL
LMSE
Acc
Pec
TL
Rec
FFD
Pao
RR
1Ao
insoo)
The
Thee unagel
Tis
(reamo
Beb
Sue
bab
MSE
1500
CSNA
100
261
151
123
3737
276
136
000
021
001
101
153
302
153
DISSECT
1 1954
033
1997
912
163
111
82
10
35
988
351
161
961
110
363
675
9300
to EPE-mod and EPE baselines and substantially improves upon CSVAE. Our method's high Dis- tinctness and Substitutability scores show that DISSECT covers the landscape of potential concepts very well and retains the variety seen in real images strongly better than all the other baselines.
Fig. 4 illustrates a few examples to showcase DISSECT's improvements over the strongest baseline, EPE-mod. EPE-mod converges to finding a single concept that only vaguely represents meaningful ground-trume concepts. However, DISSECT successfully finds concepts describing asymmetrical shapes, jagged borders, and uneven colors that align with ABCDE of melanoma [65]. DISSECT also identifies surgical markings as a spurious concept impacting the classifier's decisions. Overall, the qualitative results show that DISSECT uncovers several critical blind spots of baselines.
4.5 CASE STUDY III: IDENTIFYING BIASES
Another potential use case of DISSECT is to identify biases that might need to be rectified. Since our approach does not depend on predefined user concepts, may help discover unkwown biases. We design experiment to test DISSECT in such setting. We sub-sample Cel ebbA to create training dataset such that smiling correlates with "blond hair" and "bangs" attributes. In particular, positive samples either have blond hair have bangs, and negative examples are all dark-haired and do not have bangs. We use this dataset to train a purposefully biased classifier. We employ DISSECT to generate two CTs. Fig. 5 shows the qualitative results, which depict that DISSECT automatically discovers the two biases, which other techniques fail to do. Tab. 3 summarizes the quantitative results that replicate finding from Tab. in 8 4.3 and Tab. 2 in & 4.4 in real-world dataset, confirming that DISSECT quantitatively outperforms all the other baselines in Distinctness without negatively impacting Importance or Realism.
CONCLUDING REMARKS
We present DISSECT that successfully finds multiple distinct concepts by generating series of realistic-looking, counterfactually generated samples that gradually traverse a classifier's decision boundary. We validate DISSECT via quantitative and qualitative experiments, showing significant improvement over prior methods. Mirroring natural human reasoning for explainability [3, 4, 6, 83], our method provides additional to deployment, especially in high-stakes tasks. DISSECT helps identify whether the model-under- test reflects practitioners' domain knowledge and discover biases not previously known to users.
aidtss-nn--aaalaaeee for practitioners to probe a trained model prior
One avenue for future explainability work is extending earlier theories [45, 71] that obtain disentan- glement guarantees. While DISSECT uses an unsupervised approach for concepts discovery, it is possible to modify the training procedure and the concept disentangler network to allow for concept supervision, which can be another avenue for future work. Furthermore, extending this technique to domains beyond visual data such time series text-based input broaden this work's impact. While we provide an extensive list of qualitative examples in the Appendix, numan-subject studies to validate that CT's exhibit semantically meaningful attributes could further strengthen findings.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
We emphasize that our proposed method does not guarantee finding all the biases of classifier, ensures semantic meaningfulness across all found concepts. Additionally, it should not replace procedures for promoting transparency in model evaluation such as model cards [51]; instead, we recommend be used in tandem. We also acknowledge the limitations for Fitzpatrick ratings used in SynthDera I'm dataset in capturing variability, especially for darker skin tones [59, 77], and would like consider other ratings for future.
REPRODUCIBILITY STATEMENT
The code for all the models and metrics is publicly available at btttps://gtti asmadotgh/ hwwaaadtggh/issset...oo Additiollll,. we released the new dataset, which is publicly available at Satts://affeee ceaeeettteediii media mit eddu/u/ dissect/////// s/sstddheenn See $A.3.6 and $A.4 4 for ad- ditional details about evaluation metrics, experiment setup, and hyperparameter selection.
com/
ACKNOWLEDGMENTS
We thank Ardavan Saeedi, David Sontag, Zachary Lipton, Weiwei Pan, Finale Doshi-Velez, Emily Denton, Matthew Groh, Josh Belanich, James Wexler, Tom Murray, Kree Cole-Melaughlin, Kim- berly Wilber, Amirata Ghorbani, and Sonali Parbhoo for insightful discussions, Kayhan Batmanghe- lich and Sumedha Singla for the baseline open-source code, and MIT Media Lab Consortium for supporting this research.
REFERENCES
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, pp. 9505-9515, 2018. 3,
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net-
works. International conference on machine learning, pp. 214-223. PMLR, 2017. 4
[3] Maximilian Augustin, Alexander Meinke, and Matthias Hein. Adversarial robustness in-
and out-distribution improves explainability, arXiv preprint arXiv:2003. 2020. 2, 9
[4] Sarah R Beck, Kevin Riggs, and Sarah Gorniak. Relating developments in children's counterfactual thinking and executive functions. Thinking & reasoning, 15(4))337-354, 2009.
2,
[5] David Berthelot, Thomas Schumm, and Luke Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint 1rXiv:1703.10717 2017.
[6] Daphna Buchsbaum, Sophic Bridgers, Deena Skolnick Weisberg, and Alison Gopnik. The power of possibility: Causal learning, counterfactual reasoning, and pretend play. Philosophi- cal Transactions of the Royal Society B: Biological Sciences, 367(1599):202-2275 2012. 2,
9
[7] Chris Burgess and Hyunjik Kim. 3D shapes dataset, 2018. URL /ttptttts///gtthubooo
deepmind/ /ddh/pddssppssssss dataset. 2,5,
[8] Christopher Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in beta-VAB VAE. Advances in neural information processing systems: Workshop on Learning Disentangled Representattons, 2017. 5, 6, 15
[9] Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp. Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado. Martin C Stumpe, al. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 CHI Conference on Human Factors Computing Systems, pp. 1-14, 2019. 2.3
[10] Runjin Chen, Hao Chen, Jie Ren, Ge Huang, and Quanshi Zhang, Explaining neural networks semantically and quantitatively. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9187-9196, 2019.
10

---

Published as a conference paper at ICLR 2022
[11] Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K. Duvenaud. Isolating sources of disentanglement in variational autoencoders. In Advances in neural information processing systems, pp. 2610-2620, 2018. 4, 4,5,0 6, 15
[12] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Info- GAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pp. 2172-2180, 2016. 3,
[13] Henriette Cramer, Jean Garcia- Garria-Gathright, Aaron Springer, and Sravana Reddy. Assessing and addressing algorithmic bias in practice. Interactions, 25(6):58-63, 2018.
[14] Emily Denton, Ben Hutchinson, Margaret Mitchell, and Timnit Gebru. Detecting bias with generative counterfactual face attribute augmentation, 2019. 1,3
[15] Amit Dhurandhar, Pin-Yu Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in neural information processing systems, pp. 592-603, 2018. 3
[16] Finale Doshi-Veles and Been Kim. Towards rigorous science of interpretable machine learn- ing. arXiv preprint arXiv: 1w0..70000.0000 2017. 1
[17] Michael Downs, Jonathan L Chu, Yaniv Yacoby, Finale Doshi-Velez, and Weiwei Pan. Cruds: Counterfactual recourse using disentangled subspaces. ICML Workshop on Human Inter- pretability in Machine Learning, 2020. 2. 16
[18) Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In Conference on Empirical Methods in Natural Language Processing, pp. 11-21, 2018. 20
[19] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. University of Montreal, 1341(3):1, 2009. 1.2
[20] Thomas B Fitzpatrick. The validity and practicality of sun-reactive skin types through vi. Archives of dermatology. 124(6):869-8)) 871, 1988.
[21] Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation neural networks fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019. 2, 7, 20
[22] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept- based explanations. In Advances in Neural Information Processing Systems, volume 32, 2019.
2, 3, 6, 7, 19, 20
[23] Alyssa Glass, Deborah L McGuinness, and Michael Wolverton. Toward establishing trust in adaptive agents. In Proceedings of the 13th international conference on Intelligent user interfuces, pp. 227-236, 2008
[24] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672-2680, 2014. 4, 17
[25] Yash Goyal, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (CaCE), arXiv preprint 1907.0197..0766. 2019.
[26] Yash Goyal, Ziyan Wu, Jan Emst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In International Conference on Machine Learning, pp. 2376-2384, 2019.
3
[27] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans, arXiv preprint I 00028, 2017. 4
[28] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6633 2017. 6, 7, 19, 20
[29] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual con- cepts with a constrained variational framework. International Conference on Learning Repre-
sentations, 265);6, 2017. 1, 4, 5, 6, 15

---

Published as a conference paper at ICLR 2022
[30] Sarthak Jain and Byron C Wallace. Attention is not explanation. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, volume 1, pp.
2019.
[31] Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can explain this to you? an empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33, 2020.
[32] Shalmali Joshi, Oluwasammi Koyejo, Warut Vijitbenjaronk, Been Kim, and Joydeep Ghosh. Towards realistic individual recourse and actionable explanations in black-box decision making systems. arXiv preprint arXiv: (907:099700991. 2019. 1,
[33] Rajiv Khanna, Been Kim, Joydeep Ghosh, and Sanmi Koyejo. Interpreting black box predic- tions using Fisher kernels. In International Conference on Artificial Intelligence and Statistics,
pp. 008133323339 3390, 2019. 3,3
[34] Been Kim, Cynthia Rudin, and Julie A Shah. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in neural information pro- cessing systems, pp. 1952-1960, 2014. 1,3
[35] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference Machine Learning, pp.
2018. 1,
3.3
[36] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2654-2663, 2018. 1,
[37] Diederik P Kingma and Jimmm Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015. 19
[38] Jack Klys, Jake Snell, and Richard Zemel. Learning latent subspaces in variational autoen- coders. In Proceedings of the 32nd International Conference on Neural Information Process- ing Systems, pp. 6445-6455, 2018. 3,4, 4, 5, 6, 16
[39] Pang Wei Koh and Percy Liang. Understanding black-bor predictions via influence functions. In International Conference on Machine Learning, volume 70, pp. 1885-1894, 2017. 1,3
[40] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convo- lutional inverse graphics network. In Advances in neural information processing systems, pp. 2539-2547, 2015.
[41] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018. URL
net www.:/.eeeeoouiiiicooo
HIKG7GZAR. 4, 5, 6, 15
[42] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng. Yiming Yang. and Barnabás Póczos. Mmd gan: Towards deeper understanding of moment matching network. arXiv preprint arXiv:1705.08588 2017. 4
[43] Zinan Lin, Kiran Thekumparampil, Giulia Fanti, and Sewoong Oh. InfoGAN-CR and ModelCentraliit: Self- selfisupervised model training and selection for disentangling GANs. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp- 6127-6139. PMLR, 13-18 Jul 2020. URL htthtt/// http::///cccccceng..oooo mir. 1in20e. h
[44] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In IEEE international conference on computer vision, pp. 3730-3738, 2015. 2,5
[45] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised leamn- ing of disentangled representations. In International Conference on Machine Learning, pp-
4114-4124, 2019. 1, 3, 5, 9, 15, 22
[46] Francesco Locatello. Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference Learning Representations, 2019. 3
12

---

Published as a conference paper at ICLR 2022
[47] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, 2017. 2,
[48] Ronny Luss, Pin- Chen, Amit Dhurandhar, Prasanna Sattigeri, Yunfeng Zhang, Karthikeyan Shanmugam, and Chun-Chee Tu. Leveraging latent features for local explanations. In Pro- ceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp- 1139-1149, 2021. 3
[49] Charles T Marx, Richard Lanas Phillips, Sorelle A Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. Disentangling influence: Using disentangled representations to audit model predictions. arXiv preprint 10Xxv::90660865522 2019. 1
[50] David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self self-explaining neural networks. In Advances in Neural Information Processing Systems, pp. 7775-7784,
2018. 2, 6, 7, 20
[51] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pp. 220-22 220-229,,00 2019. 10
[52] Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In Interna- lional Conference on Learning Representations, 2018. URL
net/
3, 17
[53] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normaliza- tion for generative adversarial networks. arXiv preprint
2018. 4
[54] Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning clas- sifiers through diverse counterfactual explanations. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020. 2
[55] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images latent space. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4467-4477, 2017. 4
[56] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier GANs. In International Conference on Machine Learning, volume 70, pp- 2642-2651, 2
[57] Jose Oramas, Kaili Wang, and Tinne Tuytelaars. Visual explanation by interpretation: Im- proving visual feedback capabilities of deep neural networks. International Conference on Learning Representations, 2018. 3
[58] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Con- text encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016. 4
[59] Latrice C Pichon, Hope Landrine, Irma Corral, Yongping Hao, Joni A Mayer, and Kather- ine Hoerster. Measuring skin cancer risk in african americans: the fitzpatrick skin type classification scale culturally sensitive. Ethn Dis, 20(2): 2010. 10
[60] Gregory Plumb, Maruan Al-Shedivat, Angel Alexander Cabrera, Adam Perer, Eric Xing, and Ameet Talwalkar. Regularizing black-box models for improved interpretability. Advances in Neural Information Processing Systems, 33, 2020. 20
[61] Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary Lipton. Learn- ing to deceive with attention- based explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4782 4793, 2020. 1
[62] Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In Advances in Neural Information Processing Systems, volume 32, pp. 12268-12279. Curran Associates, Inc., 2019. URL httpss://poceeeinnnteet///////
cc/ pappr/ .df. 7,20
[63] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with 2.. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpp::s///pocceediineeo
pctppppppppp./////////p/ http://www...aaaaaappaaaapp////p 8d1 wcd.bbcadfllccc cd9
13

---

Published as a conference paper at ICLR 2022
[64] JH Rick Chang, Chun-Liang Li, Barnabas Poczos, BVK Vijaya Kumar, and Aswin C Sankara- narayanan. One network to solve them all-solving linear inverse problems using deep projec- tion models. In Proceedings of the IEEE International Conference on Computer Vision, pp-
5888-5897, 2017. 4
[65] Darrell S Rigel, Robert J Friedman, Alfred W Kopf, and David Polsky. Abede an evolving concept in the early detection of melanoma. Archives of dermatology, 143(04:002-1104,
2005. 5, 8, 9
[66] Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks arXiv preprint orXiv:1706.04900 2017. 4
[67] Tim Salimans, Ian J Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and
Xi Chen. Improved techniques for training GANs. In NIPS, 2016. 7, 19,20
[68] Pouya Samangouei, Ardavan Saeedi, Liam Nakagawa, and Nathan Silberman. ExplainGAN: Model explanation via decision boundary crossing transformations. In European Conference
on Computer Vision, pp. 666-681 2018. 1, 2, 3, 7, 20
[69] Alberto Santamaria- Panii James Kubricht, Aritra Chowdhury, Chitresh Bhushan, and Peter Tu. Towards emergent language symbolic semantic segmentation and model interpretability. In International Conference on Medical Image Computing and Computer-Assise Intervention,
pp. 326-334. Springer, 2020. 2
[70] Soñia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931-2951, 2019. 1
[71] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Representattions 2020. URL https: wwwp./eeeee.....cpoo/ppp.e.....e net/ / forum? wtw.mmmmnnnnnggggggneeeeee 3, 9, 15, 22
[72] Wilson Silva, Alexander Poellinger, Jaime S Cardoso, and Mauricio Reyes. Interpretability- guided content-based medical image retrieval. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 305-314. Springer, 2020. 2
[73] Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In International Conference on Learning Representations, 2020. xFWgrFPS. 1id 1, 2, 3, 4, 6. 17, 19, 21
URL https: opwnreview net/ / forum? iid
[74] Sumedha Singla, Brian Pollack, Stephen Wallace, and Kayhan Batmanghelich. Explaining the black-box smoothly-a counterfactual approach. arXiv preprint arXiv:2101.042.. 2021. 2, 6,
19
[75] Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why modified bp attribution fails. arXiv preprint arXiv:1912.098166 2019.
[76] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- Grad: Removing noise by adding noise. ICML Workshop on Visualization for Deep Learning, 2017. 1.2
[77] Marilyn S Sommers, Jamison D Fargo, Yadira Regueira, Kathleen M Brown, Barbara L Beacham, Angela R Perfetti, Janine S Everett, and David J Margolis. Are the fitzpatrick skin phototypes valid for cancer risk assessment in a racially and ethnically diverse sample of women? Ethmicity & disease, 29(3)5505, 2019. 10
[78] Yi Sun, Yuheng Chen, Xiaogang Wang. and Xiaoou Tang. Deep learning face representation In Proceedings of the 27th International Conference on
by joint identification-veriicaai Neural Information Processing Systems-Volume 2, pp. 1988-1996, 2014. 5
[79] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, volume 70, pp. 3319-3328, 2017. 2,
[80] Hensin Tsao, Jeannette M Olazagasti, Kelly M Cordoro, Jerry D Brewer, Susan C Taylor, Jeremy S Bordeaux, Mary- Marr-Mar Chren, Arthur J Sober, Connie Tegeler, Reva Bhushan, et al. Early detection of melanoma: reviewing the abcdes. Journal of the American Academy of Dermatology, 72(4):117-723, 2015. 8,8
[81] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Proceedings of the 3st International Conference on Neural Information Process- ing Systems, pp. 6309-6331 -618, 2017. 4
14

---

Published as a conference paper at ICLR 2022
[82] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL & Tech., 31:841, 2017. 1
[83] Deena S Weisherg and Alison Gopnik. Pretense, counterfactuals, and bayesian causal models: Why what not real really matters. Cognitive science, 37(7):138-1138) 1381, 2013. 2.
[84] Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, 2019.
[85] Jenna Wiens, Suchi Saria, Mark Sendak, Marzyeh Ghassemi, Vincent X Liu, Finale Doshi- Velez, Kenneth Jung, Katherine Heller, David Kale, Mohammed Saeed, et al. Do no harm: A roadmap for responsible machine learning for health care. Nature medicine, pp. 1-4, 2019.
[86] Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann-Wellenhor, Luc Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al. Asso- ciation between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA dermatology, 155(10): 2019. 2,
[87] Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Advances in neural information processing systems, 2018. 1,
[88] Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprini urXivv 00/9003326.,.0 2016. 4
[89] Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image trans- lation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
A APPENDIX
A.1 BASELINES
A.1.1 BASELINE I: MOLTI-MODAL EXPLAINABILITY THROUGH VAE-BASED DISENTANGLEMENT
Disentanglement approaches have demonstrated practical success in learning representations that correspond to factors for variation in data [71]. though some gaps between theory and practice re- main [45]. However, the extent to which these techniques can aid post hoc explainability in conjunc- tion with an external model is not well understood. Thus, we consider a set of baseline approaches based on VAEs explicitly designed for disentanglement: 3-VAE [29], Annealed- VAE [8]. B-TCVAE [11], and DIPVAE [41]. We extend each of them to incorporate the classifier's ' signal during their training processes for fair comparison with DISSECT. Intuitively speaking, this encourages the generative model to learn latent dimensions that could influence the classifier, i.e., learning Influen- rial CTs. Note that it is necessary to also cover data generative factors that are independent of the external classifier. This means that, along with having good quality reconstruction and high likeli- hood, we need to promote sparsity in sensitivity of the external classifier to the latent dimensions of the generative model.
More formally, consider vanilla VAE that has an encoder eg with parameters 0, decoder de with parameters 6, and the M -ddimensional latent code z with prior distribution p(z). Recall that denotes the input sample. The objective of a VAE is to minimize the loss:
L VAE
leo (x(2)) + (L(e(())))))
We introduce an additional loss term for incorporating the black-box classifier's signal: We impose this only for the first dimensions the latent space
"Without loss of generality, the additional term can be applied to the first K dimensions, and there is no need consider (K) potential selections.
15

---

Published as a conference paper at ICLR 2022
in other words, the number of desired CTs, K < M. Minimizing this term provides CT&S, E , },, with negative (f(z)/0z, with a high (0f(z)/Ozz) The final loss is:
K= /f(x)/0ax
L
VAE 6.,
K
where À is hyyer-parameter We apply this modification to the four aforementioned VAE-based approaches and refer to them with a -mod postfix, e.g. B-VAE-mod."
Development process. To promote discovering Important CTs, introduced Laux Eis 0f(z)/8zz which incorporated the directional derivative of f with respect to the latent di- mensions of interest into loss function of VAE. Despite experimentation with many variants of Laux, observed two common themes.
First, a monotonic increase of fi) through traversing one latent dimension and keeping the rest static was hardly achieved. Second, while the purpose of Laux was promote exerting Importance only in the first K dimensions of the latent space, 0f(z) /Oza for d E {K+ + K 2, ... .M were impacted similarly. Having strongly correlated dimensions is failure in achieving the very goal of disentanglement approaches. Table 4 summarizes a subset of the variants of L studied.
Table 4: Summary of a subset of Lasx iterations. The development goal is to make the first K dimensions of the latent space Important. In some iterations, encouraged the remaining M K dimensions not to be Important to reduce potential correlation across latent dimensions.
Laxx
Ei Of(a)/Ook
Σx Df(w)/0s
EM KI 10f(z)/Ozal M-K
K
Em=n 0f(a)/Oz
Dd=k+1l (05
3
M-K
Eda @0f(x)/8zal
4
M-K
E 1 10s())
5
M K End (af(a)/ /
where (1,2. ... (1,,....,K),d {K+ +,K +2, -........ .M]
Ko(M MM K)
En. Lool log()00(a))/a//ooo /00ll ( where k E ,K, {K+ 1,,K2,,+,
7
K
A.1.2 BASELINE 2: MULTI-MODAL EXPLAINABILITY THROUGH CONDITIONAL SUBSPACE VAE
Another relevant area of work is conditional generation. In particular, Conditional subspace VAE (CSVAE) is method aiming to solve unsupervised learning of features associated with a specific label using a low-dimensional latent subspace that can be independently manipulated [38]. CSVAE partitions the latent space into two parts: w learns representations correlated with the label, and covers the remaining characteristics for data generation. An assumption of independence between and w made. To explicitly enforce independence information between Y and Z. CSVAE has proven successful in providing counterfactual scenarios to reverse unfavorable decisions an algorithm, also known as algorithmic recourse [17]. To adjust CSVAE to explain the decision-making of external classifier f, treat the predictions of the classifier as the label of interest.
the learned model, we minimize the mutual
More formally, the generative model can be summarized as:
wly N( 1), Bern(y),
w. N(do, (w, 2), 2 N(0,0
Conducting inference leads to the following objective function:
WW build these baselines top of the open-sourced implementations provided in https://gitths. com/goog research di sentangl ement_] under Apache License 2.0.
16

---

Published as a conference paper at ICLR 2022
M1 = pi (x.ol))))))))) plog (z/bn, =)]+ p (1t bog p(v)]
M2 = (+(()) (2x) / (ulz) log (5(p)) dy)
Ms = q(x (08 (y|2)]
min 31 Mi B2M2; max My 0,
A.1.3 BASELINE 3: MULTI-MODAL EXPLAINABILITY THROUGH PROGRESSIVE EXAGGERATION
Explanation by Progressive Exaggeration (EPE) [73] is a recent successful generative approach that learns to generate one series of counterfactual and realistic samples that change the prediction of f. given data and the classifier's 's signal. It is particularly relevant to our work as it explicitly optimizes Influence and Realism. EPE type of Generative Adversarial Network (GAN) [24] consisting of a discriminator (D) and a generator (G) that is based on Projection GAN [52]. It incorporates the amount of desired perturbation a on the outcome of f as:
LoGAN(D) = -Epppdata [min (0, -1 + (xx, 0))] Ex-paste [min(0, -1 D(G((za)), (8)
(G) -aa [D(G(z, a)]
A Kullback-Leible divergence (KI.) term in the objective function between the desired perturbation (a) and the achieved one ((((z,)))) promotes Importance [73]:
D) = r(D, G(z, a)) (Laa)((((x
(10)
where the first term is the likelihood ratio defined in projection GAN [24], and [73] uses an ordinal- regression parameterization of it.
reconstruction loss and a cycle loss promote self self-consistency in the model, meaning that applying reverse perturbation perturbation should reconstruct the query sample:
Liee (G) || G(x, (()))]
(11)
GCy || G(G((r, a), (z))]]
(12)
Thus, the overall objective function of EPE is the following:
min max LaiAN G) + XyL + Arechree )) + Arec (G),
(13)
G D
where AEAAN Ay. and Are the hyper-parameters.
Note that EPE only finds one pathway to switch the classifier's 's outcome. We argue that classifiers learned from challenging and realistic datasets will have complex reasoning pathways that could enhance model explainability if revealed. Decomposing this complexity is needed to make reasoning comprehensible for humans. We thus create a more powerful baseline, EPE-variant, EPE-mod. EPE-mod learns multiple pathways by making the generator conditional on another variable: the CT dimension. More formally, EPE-mod updates G(---) to G(.,.k) Eq. equation 8-equation 12, while Eq. equation 13 remains unchanged. We compare DISSECT to both EPE and EPE-mod.
A.2 DISSECT DETAILS
We build our proposed method on EPF-mod and further promote distinctness across CT's by adding disentangler network, R. The disentangler is classifier with K classes. Given pair of (æ, x') images, It tries to predict which CT has perturbed query æ to produce x'. Note that R can return close to 0 probability for all classes if x' is just reconstruction of x, indicating no tweaked dimensions. The disentangler also penalizes the generator if any CTs use similar pathways to cross the decision boundary. See the appendix for schematics of our method.
To formalize this, let:
ft = ((,,)(x)
ft a(z,a,k)
T = f(x),))
17

---

Published as a conference paper at ICLR 2022
EPE
Discriminater
Generator
Real Fake?
Comnerfoctual
dimenssoo
EPE-mod
Discriminator
Generator
CT Dimension
CT
Real Fake?
CT ditnension
DISSECT
Discriminator
CT Disentangler
CT Dimension
CT Dimension 2
Real Fake?
CT Dimension?
Figure 6: Simplified illustration EPE, EPE-mod, and DISSECT. EPE-mod can be viewed as an ablated version of DISSECT. Orange, Green, and Blue show elements related to the discrimina- tor, generator, and CT disentangler, respectively. The novelty of DISSECT the disentanglement components.
Note that 2t and 2n are reconstructions of x while Fy is perturbed to change the classifier output from f() to a. Therefore, æ, Fg and ă3 form a cycle, and & represents CT₃. R(...) is the predicted probabilities of the perturbed concept given pair of examples, which is vector of size K, where each element is value in [o, 1]. We define the following cross entropy loss that is a function of both R and G:
C, C.(ek, R(x, 3x)) + CE(Ck, R(Tx,
K
(14)
Exropaate
18

---

Published as a conference paper at ICLR 2022
Here, E₁ refers to one-hot vector of size K where the k-th element is and the remaining elements are zero. This term enforces R to identify no change when receiving reconstructions the same image as input and utilizes the cycle and promotes determining the correct dimension when non-zero change has happened, either increasing or decreasing the outcome of f. In summary, the overall objective function of our method is:
min max(AcaNNccanNDDCC -yyy))y + Arec Crecre + Arec GRR
(15)
+ GG, R))
For this adversarial min-mar max optimization, we use the Adam optimizer [37]. See Figure 6 for a for a visualization of DISSECT We open-source our implementation at https: / / sigttti thub. com/ asmadotgh. ssect under MIT license, which builds on top of the open-source implementa- tion of EPE [73] 8,
A.3 EVALUATION METRICS DETAILS
In this section, we provide more details about the evaluation metries. First, we provide evidence supporting these criteria desirable qualities for explainability. Then, we summarize how these qualities have been measured in prior work. Finally, we explain when we can use prior formulation of these metrics as is, and when we need to modify them to make them applicable to the counterfac- tual explanation case. We justify how these formulations capture the desired criteria.
A.3.1 IMPORTANCE
Motivation. Explanations should produce the desired outcome from the black-box classifier f- Previous work has referred to this quality using different terms, such as importance [22], compati- bility with classifier [73], and classification model consistency [74].
Quantification. Most previous methods have relied on visual inspection. Singla et al. [74] have plotted expected outcome from the classifier, against the actual classifier prediction of the gener- ated explanations. Ghorbani et al. [22] have plotted prediction accuracy vs. adding or removal of discovered concepts.
We introduce a quantitative metric to measure the gradual increase the target class's posterior probability through CT. Notably, compute the correlation between and f(](x, a; f)) in- troduced in $ 3. For brevity, we refer to [I[xx, a. k; f)) as f in the remainder of the paper. We also report the mean-squared error and the Beilbackk-eibble (KL) divergence between and fø)) This is one way to summarize information in the plots used by Singla et al. [74] for Importance evaluation.
We also calculate the performance of f on the counterfactual explanations. Specifically, we replace the
real images with their generated counterfactual explanations and quantify the perfor- mance of the pre-trained black-box classifier f on the counterfactual test set. Better performance suggests that counterfactual samples are compatible with f and lie on the correct side of the classi- fier's is boundary. This is one way to summarize the information in the plots used by Ghorbani et al. [22] for Importance evaluation.
A.3.2 REALISM
Motivation. We need the generated samples that form CT to look realistic to enable users to identify concepts they represent. It means the counterfactual explanations should lie on the data manifold. This quality has been referred to realism data consistency [73].
Quantification. variety of inception scores have perviously been used quantify visual quality of generated images, such as Fréchet Inception Distance [28] and Inception Score [67].
hhtpp:: //gi thub. com batmaattannnn.aa 1tt.mmmanoo//ppaatttononnoaoo available under MIT License.
19

---

Published as a conference paper at ICLR 2022
Similar to prior work, use FID [28]. However, since FID is based Inception V3 network trained on ImageNet, it could be a better fit for more realistic data that has similar distribution to Im- ageNet. This is not the case, especially for 3d Shapes and SynthDeras in work. Therefore, we include an additional metric inspired by [18]. We train a post hoc classifier that predicts whether sample is real generated. We report performance of this classifier as a proxy for Realism.
A.3.3 DISTINCTNESS
Another desirable quality for explanations is to represent inputs with non-overlapping concepts, often referred to as diversity [50]. Others have suggested similar properties such as co- herency, meaning examples of concept should be similar but different from examples of other concepts [22].
Motivation.
Quantification. To quantify this in counterfactual generation setup, introduce distinctness measure capturing the performance of secondary classifier that distinguishes between CTs. We train a classifier post hoc that given a query image x and generated image x' and K number of CTs, predicts CT that has perturbed x to produce z', k {1...., K}. This
A.3.4 SUBSTITUTABILIT
Motivation. The representation of a sample in terms of concepts should preserve relevant infor- mation [50, 60]. Previous work has formalized this quality for counterfactual generation contexts through a proxy called substitutability [68].
Quantification. Substiturability measures external classifier's performance real data when trained using only synthetic images. This metric has been used in other contexts outside of explainability and has been called Classification Accuracy Score (CAS) [62]. CAS is more broadly applicable than Fréchet Inception Distance [28] & Inception Score [67] that are only useful for GAN evaluation. Furthermore, CAS can reveal information that none of these inception scores successfully capture [62]. A higher substitutability score suggests that explanations have retained relevant information and are of high quality.
We follow Samangouei et al. [68] for formulation of this metric.
A.3.5 STABILITY
Motivation. Explanations should be coherent for similar inputs, quality known as stability [21, 50, 60].
Quantification. To quantify stability in counterfactual explanations, we augment the test set with additive random noise to each sample æ and produce S randomly augmented copies z; Then, generate counterfactual explanations I and E respectively. We calculate the mean-squared difference between counterfactual images ž and i, and the resulting Jensen Shannon distance (JSD) between the predicted probabilities f(z) and (T
A.3.6 IMPLEMENTATION DETAILS
The VAE-based baselines support continuous values for latent dimensions zμ. Also, we can directly sample latent code values and produce CT₃ by keeping 2j (j * k) constant and monotonically increasing 2か values. However, to calculate the evaluation metries comparably to EPE, EPF-mod, and DISSECT, we do the following: We encode each query sample using the probabilistic encoder. We set 2, = J & where μ4, is the mean of the fitted Gaussian distribution for 2j. For dimension k, we produce N + linearly spaced values between μР + 2 op. where Ml and σ, the mean and standard deviation of the prior normal distribution, in our case 0.0 and 1.0 respectively. Note that these different values for 2 map out a, E {0, N. ... 1} in EPE, EPE-mod and DISSECT models. After this step, calculating all the metrics related to Importance, Realism, and Distinctness identical across all the models.
20

---

Published as a conference paper at ICLR 2022
Table 5: Summary of hyper-parameter values. Discriminator optimization happens every D steps. Similarly, generator optimization happens once every G steps. Ap is specific to DISSECT, and K is specific EpE- and DISSECT, All the remaining parameters are shared across EPE, EPE-mod, and DISSECT. Note that samples used for evaluation are not included in the training process.
Preprocessing
Training
Evaluation
N
samples
per bin
max
ACAAN
Avec
steps
steps
batch
epochs
X,
samples
batch
epochs
hold-our
ralio
test
Shapes
5,000
100
32
300
10
10,000
32
10
0.25
SynthDerm
2
1,350
2
100
300
10,000
0.25
Colebb
10
5,000
100
32
300
10,000
32
10
0.25
A.4 EXPERIMENT SETUP HYPER-PARAMETER TUNING DETAILS
Experiments were conducted on an internal compute cluster at authors' institution. Training and evaluation of all models across the three datasets have approximately taken 1000 hours on a com- bination Nvidia GPUs including GTX TITAN X, GTX 1080 Ti, RTX 2080 Rev, and Quadro K5200.
We seeded model's parameters from [73] based on the reported values their accompanying open-sourced repository." We used the same parameters for 3D Shapes, except for the number of bins, N, used for ordinal regression transformation of the classifier's 's posterior probability. The largest number of bins that resulted in non-zero samples per bin. was selected. We kept all the parameters shared between EPE, EPE-mod, and DISSECT the same.
Given the experiments' design, fixed the number of dimensions K in DISSECT and EPE-mod to We experimented with few values for X,, 1, 10, 20, 50. Based on manual inspection after 30k training batches, Ay was selected. Factors considered for selection included inspecting the perceived quality of generated samples and the learning curves of (D),N(D), LaGAN LGy) Lree and (G,
For evaluation, we used hold-out set including 10K samples. For post hoc evaluation classifiers predicting Distinctness and Realism, 75% of the samples were used for training, and the results were reported on the remaining 25%. See Table 5 for the summary of the hyper-parane parameter values.
A.5
ADDITIONAL. QUALITATIVE RESULTS
A.5.1 CASE STUDY
Recall that considering 3D Shapes, we define image as "colored correctly" if the shape hue red or the floor hue is cyan. Given not "colored correctly" query, we recover a CT related to the shape color and another CT associated with the floor color-two different pathways leading to switching the classifier outcome for that sample. See Figure for additional qualitative examples where classification outcome is flipped from False to True.
However, these two ground-truth concepts do not directly apply to switching the classifier outcome from True to False in this scenario. For example, if image has red shape and cyan floor, both colors need to be changed to switch the classification outcome. As shown in Figure 8, we still observe that applying DISSECT such cases results in two discovered CTs that change different combinations of colors while EPE-mod converges to the same CT.
A.5.2
CASE STUDY III
Recall the biased Ce Celeba leb experiment where smiling correlates with "blond hair" and "bangs" at- tributes. Figure 9 shows additional qualitative samples, suggesting that DISSECT can recover and separate the aforementioned concepts, which other techniques fail to do.
https: //gi thub. wtwptt/nnnab//ppamaa available under MIT License.
21

---

Published as a conference paper at ICLR 2022
Query 1a0
EPE mod.
DISSECT
Query
EPE mod
DISSECT
Query Jan
mDD-mo
DISSECT
Query wilh
DISSECT
Query
EPE-mod
DISSECT
Query
EPPEmmd
DISSECT
290 100
Figure 7: Qualitative results on 3D Shapes when flipping classification outcome from "False" to "True." " We observe that EPE-mod converges to finding the same concept, despite having the ability to express multiple pathways to switch the classifier outcome. However, DISSECT can discover the two Distinct ground-tre -trth concepts: CT₁ flips the floor color to cyan, and CT2 converts the shape color to red.
Query
Query
DISSECT
Query
EPE--mo
DISSECT
Aili ciee
Jif 10---60
Shaper Red Floor: Not Cyan
Stape: Not Red Floor: Cyan.
Shape:
Floor: Cyan
Figure 8: Qualitative results on 3D Shapes when flipping classification outcome from "True" to "False." We observe that HPE-mod converges to finding the same concept, despite having the ability to express multiple pathways to switch the classifier outcome. However, DISSECT capable of discovering Distinct paths to do so. Left: When the input query has red shape, but the floor color not cyan, CT₁ flips the shape color to orange and CT2 flips it to violet. Middle: When the input query has cyan floor, but the shape color not red, CT₁ flips the floor color to lime, and CT2 converts it to magenta. Right: When the input query has red shape and cyan floor, CT changes the shape color dark orange and floor color to lime, and CT2 flips the shape color to violet and floor color to magenta.
A.5.3 LIMITATIONS
While we provide several qualitative examples, further confirmation from human-subject studies to validate that CTs exhibit semantically meaningful attributes could strengthen our findings.
The number of concepts to be discovered by DISSECT is hyper parameter that can be selected by the user. While Distinctness and Substitutability are designed to be globally evaluated across all concepts, Importance, Realism, and Stability scores can be calculated for each concept. We hypothesize that ranking the discovered concepts based on these individual scores can benefit the users. In particular, it can help them focus informative concepts. Future user-studies are required to investigate these hypotheses.
salient concepts and not be overwhelmed by less
We emphasize that proposed method does not guarantee finding all the biases of classifier, ensures semantic meaningfulness across all found concepts. One avenue for future explainability work is extending earlier theories [45, 71] that obtain disentanglement guarantees.
22

---

Published as a conference paper at ICLR 2022
Query Image
Query Image
AAAE
EPE
HMT -.
EPE-med
An wa
100 Cit
AP 141
15t
Aot-
Ath .430
DISSECT
Har KM
Tok
(7 wAMt
AP
Tox- -MM
Ao--00
Smiling
Bamps
Blond Hair
Clussifying:
Correlated Ground-
truth Conceples
Figure 9: Qualitative results on CelebA. A biased classifier has been trained to predict smile prob- ability, where the training dataset has been sub-sampled such that smiling co-occurs only with "bangs" and "blond hair" attributes. EPE does not support multiple CTs. We observe that EPE- mod converges to finding the same concept, despite having the ability to express several pathways to change fz) through CT₁ and CT2. However, DISSECT can discover Distinct routes: CT₁ mainly changes hair color to blond, and CT2 does not alter hair color but focuses more on hairstyle and tries to add bangs. Thus it identifies otherwise hidden biases.
Table 6: Ablation experiment. Different rows show results for CelebA, with different Ay values. All the other variables are the same across experiments: AGGAN = 1, Arec 100, As 1.
Realoz
Sanierasily
KL
ASSE
Dec
Bx
PPac
Aes
usam:
uzam
S
5
J5D
B301
1738
1130
68.1
903
975
110
80
316
515
1520
774
481
301
405
0765
<760
Disser
101
1390
1013
1039
10485
99.48
99083
2034
1336
0789
22394
15.40
19.10
5338
11.00
06.20
0737
-200
0109
0300
Insser
038
030
6090
60.0
110%
1600
07.800
039
10
1084
11.0
2328
0004
30114
20543
9990
2010
01000
4000
19300
3299
074.4
-200
1495
0738
A.6 ABLATION/SENSITIVITY EXPERIMENTS
EPE-mod can be viewed as the ablated version of DISSECT where X, = 0. For more detailed anal- yses of the influence of the disentanglement element, we conducted an experiment with different X while fixing the other hyperparameters of the model. As seen in Table 6, increasing X, up point improves Distinctness and retains performance on the other criteria of interest. Further increasing X, hurts the quality of generated images, and thus negatively impacts Realism and Substitutability.
A.7 INDIVIDUAL CONCEPT INFLUENCE
Note that metrics such as Distinctness and Substiutaaillity are not meaningful for each concept. However, Importance. Realism, and Stability can be calculated each concept separately, over all concepts. Calculating such metries for each concept can help us rank them with respect to that criterion. For example, Importance scores per concept can tell us which concept more important
23

---

Published as a conference paper at ICLR 2022
Table 7: Individual Importance scores per concept discovered by DISSECT. Discovered 3D Shapes concepts: 1) Red shape, 2) Cyan floor. Discovered SynthDerm concepts: 1) Color, 2) Diameter, 3) Asymmetry, 4) Border, Surgical markings and color. Discovered CelebA con- cepts: 1) Blond hair, 2) Bangs.
Importance
1 CF
CC
1 CF
1 R
1 P
KL
MSE
Acc
Prec
Rec
CT
0.823
0.843
1.421
0.084
92.32
100.0
98.64
3D Shapes
CT2
0.822
0.766
1.794
0.085
98.15
T00.0
96.3
CTi
0.908
0.747
0.383
0.021
97.615
91.886
89.671
CT2
0.895
0.746
0.423
0.023
97.215
90.191
88.294
SynthDerm
CT3
0.935
0.75
0.276
0.015
98.26
93.61
93.037
CT4
0.929
0.749
0.303
0.016
98.195
93.244
92.923
CT₂
0.919
0.748
0.349
0.018
97.935
92.605
91.507
CTi
0.845
0.882
0.184
0.046
99.08
99.817
98.34
CelebA
CT2
0.842
0.882
0.192
0.047
99.25
99.868
98.63
Table 8: Interaction between generation quality and explanation quality. Three different sizes have been considered for the underlying GAN architecture: xxsmall 1 1, small 1, base. For comparison,
GAN-only version of base also included, where all the other components are disabled, i.c. Ay = 0, Avec = 0, and Xy = O.
Importancce
Realism
Distinctness
RR
1P
KL
MSE
FID
Ace
PrPre
RRcc
Acc
(micro)
(macro)
(micro)
(macro)
GA8 SAN-only
7.367
0.0
11.526
0.414
110.964
100.0
100.0
33.3
0.0
0.0
small
0.822
0803
29915
0.085
$1.002
99.08
99.15
98.99
99.980
99.990
93.90
99.980
99.980
0.84
42.109
98.1
%6.1
in the decision making of the classifier. Table 7 summarizes the Importance scores per concept. As expected, given the symmetric design our experiments, discovered concepts exhibit similar Importance scores.
A.8 INTERACTION BETWEEN GENERATION QUALITY AND EXPLANATION QUALITY
In this section, we evaluate DISSECT with different sizes of underlying GAN architectures. As shown in Table 8, even with imperfect generation quality, DISSECT still discover relevant cepts with comparable performance in terms of Importance, Distinctnsss. This is despite significant impacts on the Realism scores. Perfect generation quality can further improve realism. However, reasonable generation quality is sufficient for successful explanation with DISSECT.
With the recent advances in generative modeling that have achieved great generation quality over majority of domains, it is timely to utilize them when available for generating realistic- explanations. It is worth mentioning that perfect generation alone is not sufficient for good ex- planations in terms of performance criteria other than Realism. For comparison, we have calculated performance scores of strong generative model, excluding the interpretbbllity-reaaed terms the objective function. See the results in Table 8 for the GAN-on row.
24


Reviews:
[{"review_id": "uTXzSnoWDJ1", "reviewer": "Reviewer_1keG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:\n\n- novelty: the approach taken in this paper is very different from other counterfactual explanation methods. This is the first paper I've read that creates several explanations which increasingly rely on a concept. I think this aspect is very interesting and could be relevant in practice. For example, as highlighted in the paper, it can be used to depict how a mole turning from benign to malignant may progress. \n- experiments/evaluation: The authors demonstrate the efficacy of their method over three different datasets (each well motivated for the problem at hand) and extensively evaluate the performances. Specifically, they introduce metrics for each of the desiderata  (\"importance\", \"realism\", \"distinctness\", \"stability\").  The proposed method generally performs better than the benchmarks (VAEs designed for disentanglement, CSVAE, EPE, and modifications thereof). \n\n**Weaknesses**: \n\n- several of the evaluation metrics are first introduced in this paper. It would be great to see some more justification on whether the evaluation metrics actually measure what they are trying to target. E.g., for realism, one could simulate data for which we know the \"ground truth\" and then evaluate the metric.\n- to a certain extent, the method is designed to target these metrics. As such, I would expect it to perform better. Having said that, I think this is only a small weakness. Evaluation protocols are not widely agreed upon in the field, and so there isn't a clear protocol to follow. \n\n**Clarity**: \n\nGenerally, the paper is quite well-written. The architecture is well-motivated (particularly the loss in section 3.1) and the experimental setup (both evaluation metrics and choice of datasets) is well-argued. \n\n**Questions**:\n- did you perform a sensitivity analysis for the loss function? While the motivation for the loss function is extensively described, I am curious whether all terms are necessary. \n\n**Minor comments** (did not affect the score in any way):\n- use the following symbol twice for the left quotation marks in latex: `\n- there are some minor typos/grammar/spacing errors. ", "summary_of_the_paper": "The authors introduce DISSECT: Disentangled Simultaneous Explanations via Concept Traversal. The method creates counterfactual explanations for which the concept increasingly influences the classifier's decision. Experimentally, the authors show that the proposed method generally performs better than existing methods on several aspects (\"importance\", \"realism\", \"distinctness\", \"stability\"). ", "summary_of_the_review": "Overall, I would recommend an acceptance. The field of interpretability is becoming increasingly more important. The authors introduce a novel method, called DISSECT, which brings a meaningful contribution to the aforementioned field.  The method is novel (bringing a fresh perspective) and performs well empirically. I did not give an 8 as I still had some reservations about some aspects of the paper (as highlighted in the section before).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QORTf2lbJzZ", "reviewer": "Reviewer_WoTB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of using VAEs to generate multiple counterfactuals is novel. Authors have evaluated their work on 3 image datasets and a number of useful evaluation metrics and baselines. The results are promising. \n\nCould you please comment on similarity/differences with this work: \nhttps://arxiv.org/pdf/1905.12698.pdf\n\n", "summary_of_the_paper": "In line with further work on counterfactual explanations, this work proposes to provide multiple counterfactual explanations in terms of VAE style disentangled latent variables. The idea of using VAEs to obtain multiple counterfactuals is novel and quite useful for multiple practical applications. ", "summary_of_the_review": "While there is work on diverse counterfactual explanations and different VAEs (DIP-VAE, Beta-VAE, etc), and work algorithmic recourse, this work seems to marry the two ideas of of providing multiple counterfactuals based on disentangles latent variables. This can be quite helpful in practise. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nv4v4q2JZ-", "reviewer": "Reviewer_47RT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an original paper that introduces a novel approach to model explanations. It builds on several advancements in the literature, and aims at producing a solution that is useful for the users that will actually run it. It is clearly written and easy to follow, and includes an extensive experimental analysis that compares the approach to many strong baselines and on several substantially different dataset.\n\nWhile I do think that this is overall a good paper, I do have some minor issues that I would like to see clarified. First, I am worried about the dependency on a strong generative model that can successfully disentangle concepts. It could be that this approach would fail on data with structures that do not allow such an easy generation process, such as natural language. To strengthen the findings in the paper, I would like to see some exploration of the relationship between generation quality and explanation quality.\n\nSecond, I’m not sure that the fact that this method does not depend on predefined user concepts is a good thing. It can certainly be advantageous if we want to explore hidden biases, but it could also be problematic in cases where we want to test a specific hypothesis regarding a concept we already know and care about. How would an approach like DISSECT deal with cases where we care about some specific concepts, but would also like to discover unknown biases?\n\nThirdly, I think that the paper might benefit from a deeper discussion of the importance of generating realistic counterfactuals from a causal perspective. The main motivation for producing counterfactuals for explanations is the estimation of causal effects, and connecting DISSECT to this literature can improve the paper in my opinion.\n\nFourthly, this paper seems fairly based on a specific paper, “Explanation by Progressive Exaggeration”. While the authors are clear about their use of this work, the similarity of the approaches is significant. In fact, the overall objective is identical except for the extra term that enforces disentanglement. Given that, the technical contribution of this paper is a bit thin. \n\nLastly, I have trouble understanding the qualitative result in Figure 5. From carefully observing it, I am not convinced that DISSECT does a substantially better job than EPE-mod in disentangling the confounding concepts. While I do find the quantitative evidence convincing, I would consider clarifying or replacing Figure 5.\n", "summary_of_the_paper": "This paper proposes a generative approach to model explanations, where explanations are composed of automatically generated counterfactual examples that differ by the observed magnitude of a given learned concept. Through a series of extensive experiments, the authors show that the proposed approach, DISSECT, can successfully disentangle useful and realistic concepts.\n\nThe main contributions of the paper is the novel approach that generates several counterfactual examples across a concept spectrum, and a new synthetic dermatology dataset that allows testing such methods where they matter most. \n", "summary_of_the_review": "While I do think that this is overall a good paper, I do have some minor issues that I would like to see clarified. First, I am worried about the dependency on a strong generative model that can successfully disentangle concepts. Second, I’m not sure that the fact that this method does not depend on predefined user concepts is a good thing. Thirdly, I think that the paper might benefit from a deeper discussion of the importance of generating realistic counterfactuals from a causal perspective. Lastly, I think that the technical contribution might be a bit too thin, given prior work that this paper builds on. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EP8wWfbKYtm", "reviewer": "Reviewer_jPGJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The task is interesting, the proposed DISSECT is based on Explanation by Progressive Exaggeration (EPE) while proposing disentanglement modules. It is good to provide the pipeline comparison between EPE-mod and DISSECT, because they may look similar, which shows limited novelty.\n\nClarity: Overall is clear, however, some sentences are too long to follow: e.g., 1st paragraph in the Method section. \n\nSignificance and experiments: \n\n1 Baselines method and evaluation setting are thorough, while it would be better to show ablation study and show the necessity of each component, especially the difference between EPE-mod and DISSECT.\n\n2 I am curious about the generalization to a novel dataset, how to define the meaning of each concept that influences the classification? \n\n3 For fidelity of the explanation, I wondered about the correctness of the concept. How to verify the correctness of the concepts? How to rank the importance of the concepts for an explanation?\n\n4 For How to guarantee the provided CTs can be easily understood by humans? How to avoid confusing users when more concepts emerge?\n\n", "summary_of_the_paper": "This paper proposed a generation-based explainability method, DISSECT, that generates Concept Traversals to provide a counterfactual explanation. The Concept traversals (CTs) are sequences of generated examples with increasing degrees of concepts' influence on the classifier's decision. The CTs are able to provide disentangled explanations for different concepts. ", "summary_of_the_review": "I am borderline for this paper in the current stage and I may increase my score if the author can solve the above concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a counterfactual explanation method, termed DISSECT, for image classification. While previous work is concerned with generating one single counterfactual, DISSECT aims to produce multiple counterfactuals, with each illustrating one possible way the class label could be altered. Intermediate images between the benign example and the counterfactuals are also generated to show how the decision boundary is crossed.
The reviewers find the idea novel, the presentation clear, and the empirical evaluation thorough.  However, there are concerns regarding whether the method will generalize to other domains because it relies on a strong generative model.  In addition, there is no human-subject study to show whether and how much the method really help an end-user.
Decision: Accept (Poster)