DISSECT: DISENTANGLED SIMULTANEOUS EXPLA-
Year: 2021

Abstract:
Explaining deep learning model inferences is a promising venue for scientific understanding, improving safety, uncovering hidden biases, evaluating fairness, and beyond, as argued by many scholars. One of the principal benefits of counterfactual explanations is allowing users to explore “what-if” scenarios through what does not and cannot exist in the data, a quality that many other forms of explanation such as heatmaps and influence functions are inherently incapable of doing. However, most previous work on generative explainability cannot disentangle important concepts effectively, produces unrealistic examples, or fails to retain relevant information. We propose a novel approach, DISSECT, that jointly trains a generator, a discriminator, and a concept disentangler to overcome such challenges using little supervision. DISSECT generates Concept Traversals (CTs), defined as a sequence of generated examples with increasing degrees of concepts that influence a classifier’s decision. By training a generative model from a classifier’s signal, DISSECT offers a way to discover a classifier’s inherent “notion” of distinct concepts automatically rather than rely on user-predefined concepts. We show that DISSECT produces CTs that (1) disentangle several concepts, (2) are influential to a classifier’s decision and are coupled to its reasoning due to joint training (3), are realistic, (4) preserve relevant information, and (5) are stable across similar inputs. We validate DISSECT on several challenging synthetic and realistic datasets where previous methods fall short of satisfying desirable criteria for interpretability and show that it performs consistently well. Finally, we present experiments showing applications of DISSECT for detecting potential biases of a classifier and identifying spurious artifacts that impact predictions.

heading: 1 INTRODUCTION
text: Explanation of the internal inferences of deep learning models remains a challenging problem that many scholars deem promising for improving safety, evaluating fairness, and beyond [13, 16, 23, 49]. Many efforts in explainability methods have been working towards providing solutions for this challenging problem. One way to categorize them is by the type of explanations, some post hoc techniques focusing on the importance of individual features, such as saliency maps [19, 47, 76, 79], some on importance of individual examples [33, 34, 39, 87], some on importance of highlevel concepts [35]. There has been active research into the shortcomings of explainability methods (e.g. [1, 30, 61, 70, 84]). Scholars have also proposed tests to determine when attention can be used as an explanation [84].
While these methods focus on information that already exists in the data, either by weighting features or concepts in training examples or by selecting important training examples, recent progress in generative models [12, 29, 36, 40, 45] has lead to another family of explainability methods that provide explanations by generating new examples or features [14, 32, 68, 73]. New examples or features can be used to generate counterfactuals [82] allowing users to ask: what if this sample were to be classified as the opposite class, and how would it differ? This way of explaining mirrors the way humans reason, justify decisions [3], and learn [4, 6, 83]. Additionally, users across visual, auditory, and sensor data domains found examples the most preferred means of explanations [31].
To illustrate the benefits of counterfactual explanations, consider a dermatology task where an explanation method is used to highlight why a certain sample is classified as benign/malignant (Fig. 1).
Explanations like heatmaps, saliency maps, or segmentation masks only provide partial information. Such methods might hint at what is influential within the sample, potentially focusing on the lesion area. However, they cannot show what kind of changes in color, texture, or inflammation could transform the input at hand from benign to malignant. Retrieval-based approaches that provide examples that show a concept are not enough for answering “what-if” questions either. A retrieval-based technique might show input samples of malignant skin lesions that have similarities to a benign lesion in patient A, but from a different patient B, potentially from another body part or even a different skin tone. Such examples do not show what this benign lesion in patient A would have to look like if it were classified as malignant instead. On the other hand, counterfactuals depict how to modify the input sample to change its class membership. A counterfactual explanation visualizes what a malignant tumor could look like in terms of potential color or texture, or inflammation changes on the skin. Better yet, multiple counterfactuals could highlight several different ways that changes in a skin lesion could reveal its malignancy. For a classifier that relies on surgical markings [86] as well as meaningful color/texture/size changes to make its decision, a single explanation might fail to reveal this flaw resulting from dependence on spurious features, but multiple distinct explanations shed light on this phenomenon. Multiple explanations are strongly preferred in several other domains, such as educational settings, knowledge discovery, and algorithmic recourse generation [17, 54].
In this work, we propose a generation-based explainability method called DISSECT that generates Concept Traversals (CTs)–sequences of generated examples with increasing degrees of concepts’ influence on a classifier’s decision. While current counterfactual generation techniques fail to satisfy the most consistently agreed-upon properties desired for an explainability method simultaneously [50, 60, 68, 73, 74], DISSECT aims to overcome this challenge. CTs are generated by jointly training a generator, a discriminator, and a CT disentangler to produce examples that (1) express one distinct factor [50] at a time; (2) are influential [73, 74] to a classifier’s decision and are coupled to the classifier’s reasoning, due to joint training; (3) are realistic [73]; (4) preserve relevant information [50, 60, 68]; and (5) are stable across similar inputs [21, 50, 60]. Compared to other approaches that require a human to identify concepts a priori and find samples representing them, e.g. [9, 35], DISSECT uncovers them automatically. We compare DISSECT with several baselines, some of which have been optimized for disentanglement, some used extensively for explanation, and some that fall in between. DISSECT is the only technique that performs well across all these dimensions. Other baselines either have a hard time with influence, lack fidelity, generate poor quality and unrealistic samples, or do not disentangle properly. We evaluate DISSECT using 3D Shapes [7], CelebA [44], and a new synthetic dataset inspired by the challenges faced in the dermatology domain. We show that DISSECT outperforms prior work in addressing all of these challenges. We also discuss this work’s applications to detect a classifier’s potential biases and identify spurious artifacts using simulated experiments.
This paper makes five main contributions: 1) presents a novel counterfactual explanation approach that manifests several desirable properties outperforming baselines; 2) demonstrates applications through experiments showcasing the effectiveness of this approach for detecting potential biases of a classifier; 3) presents a set of explainability baselines inspired by approaches used for generative disentanglement; 4) translates desired properties commonly referred to in the literature across forms of explanation into measurable quantities for bench-marking and evaluating counterfactual generation approaches; 5) releases a new synthetic dermatology dataset inspired by real-world challenges.

heading: 2 RELATED WORK
text: Our method relates to the active research area of post hoc explainability methods. One way to categorize them is by explanation form. While met with criticisms [1, 75], many feature-based explainability methods are shown to be useful [47, 79], which assign a weight to each input feature to indicate their importance in classification [10, 57]. Example-based methods are another popular category [33, 34, 39, 87] that instead assign importance weights to individual examples. More recently, concept-based methods have emerged that attribute weights to concepts, i.e., higher-level representations of features [22, 25, 35] such as “long hair”.
Our work leverages recent progress in generative modeling, where the explanation is presented through several conditional generations [52, 56]. Efforts for the “discovery” of concepts are also related to learning disentangled representations [12, 29, 35, 36], which has been shown to be challenging without additional supervision [45]. Recent findings suggest that weak supervision in the following forms could make the problem identifiable: how many factors of variation have changed [71], using labels for validation [46], or using sparse labels during training [46]. While some techniques like Conditional Subspace VAE (CSVAE) [38] began to look into conditional disentanglement by incorporating labels, their performance has not yet reached their unconditional counterparts. Our work uses a new form of weak supervision to improve upon existing methods: the posterior probability and gradient of the classifier-under-test.
Many explainability methods have emerged that use generative models to modify existing examples or generate new examples [14, 15, 26, 32, 48, 68, 73]. Most of these efforts use pre-trained generators and generate a new example by moving along the generator’s embedding. Some aim to generate samples that would flip the classifier’s decision [15, 32], while others aim to modify particular attribution of the image and observe the classifier’s decision change [14]. More recent work allows the classifier’s predicted probabilities or gradients to flow through the generator during training [68], and visualize the continuous progression of a key differential feature [73]. This progressive visualization has additional benefits, as Cai et al. [9] showed that pathologists could iterate more quickly using a tool that allowed them to progressively change the value of a concept, compared to purely using examples. Additionally, continuous exaggeration of a concept facilitated thinking more systematically about the specific features important for diagnosis.
Most of these approaches assume that there is only one path that crosses the decision boundary, and they generate examples along that path. Our work leverages both counterfactual generation techniques and ideas from the disentanglement literature to generate diverse sets of explanations via multiple distinct paths influential to the classifier’s decision, called Concept Traversals (CT). Each CT is made of a sequence of generated examples that express increasing degrees of a concept.

heading: 3 METHODS
text: Our key innovation is developing an approach to provide multiple potential counterfactuals. We show how to successfully incorporate the classifier’s signal while disentangling relevant concepts. We review existing generative models for building blocks of our approach, and follow Singla et al. [73] for DISSECT’s backbone. This choice is informed by the good conditional generation capability of [73] and being a strong interpretability baseline. We then introduce a novel component for unsupervised disentanglement of explanations. Multiple counterfactual explanations could reveal several different paths that flip the classification outcome and shed light on different biases of the classifier. However, a single explanation might fail to reveal such information. Disentangling distinct concepts without relying on predefined user labels reduces the burden on the user side. Additionally, it can surface concepts that the user might have missed otherwise. However, unsupervised disentanglement is particularly challenging [45]. To address this challenge, we design a disentangler network. The disentangler, along with the generator, indirectly enforces different concepts to be as distinct as possible. This section summarizes our final design choices. For more details about baselines, DISSECT, and ablations studies, see §A.1, §A.2, and §A.6, respectively. Notation. Without loss of generality, we consider a binary classifier f : X → Y = {−1, 1} such that f(x) = p(y = 1|x) where x ∼ PX . We want to find K latent concepts that contribute to the decision-making of f . Given x, α ∈ [0, 1], and k, we want to generate an image, x̄, by perturbing the latent concept k, such that the posterior probability f(x̄) = α. In addition, when conditioning on
x and a concept k, by changing α, we hope for a smooth change in the output image x̄. This smoothness resembles slightly changing the degree of a concept. Putting it together, we desire a generic generation function that generates x̄, defined as G(x, α, k; f) : X × [0, 1]× {0, 1, . . . ,K − 1} → X , where f(G(x, α, k; f)) ≈ α. The generation function G conditions on x and manipulates the concept k such that a monotonic change in the f(G(x, 0, k)), . . . , f(G(x, 1, k)) is achieved.

heading: 3.1 ENCODER-DECODER ARCHITECTURE
text: We realize the generic (conditional) generation process using an encoder-decoder framework. The input x, is encoded into an embedding E(x), before feeding into the generator G, which serves as a decoder. In addition to E(x), we have a conditional input c(α, k) for the kth latent concept we are going to manipulate with level α. α is the desired prediction of f we want to achieve in the generated sample by perturbing the kth latent concept. The generative process G can be implemented by G(E(x), c(α, k)), where we can manipulate c for interpretation via conditional generation.
There are many advances in generative models, which can be adopted to trainG(E(x), c(α, k)). For example, Variational Autoencoders (VAE) explicitly designed for disentanglement [11, 29, 38, 41]. However, VAE-based approaches often generate blurry images which requires customized architecture designs [63, 81]. In contrast, GANs [24] tend to generate high-quality images. Therefore, we consider an encoder-decoder design with GANs [5, 55, 58, 64, 66, 89] to have flexible controllability and high quality generation for explainability.
There are different design choices for realizing the conditioning code c. A straightforward option is multiplying a K-dimensional one-hot vector, which specifies the desired concept k, with the probability α. However, conditioning on continuous variables is usually challenging for conditional GANs compared with their discretized counterparts [52]. Therefore, we discretize α into [0, 1/N, . . . , 1], and parametrize c as a binary matrix [0, 1](N+1)×K for N + 1 discrete values and K concepts, following Singla et al. [73]. Therefore, c(α, k) denotes that we set the elements (m, k) to be 1, where m is an integer and m/N ≤ α < (m+ 1)/N . See § A.2 for more details. We include the following elements in our objective function following Singla et al. [73], and further modify them to support multiple concepts:
Interpreting classifiers. We align the generator outputs with the classifier being interpreted 1: Lf (G) = α log ( f ( G (x, c(α, k)) )) + ( 1− α ) log ( 1− f ( G (x, c(α, k)) )) . (1)
Reconstruction. We adopt `1 reconstruction loss in our encoder-decoder design [55, 89]. Given x ∼ PX and classifier f , the ground-truth α for all concepts is f(x):
Lrec(G) = 1
K K−1∑ k=0 ∥∥∥x−G(x, c (f(x), k) )∥∥∥ 1 . (2)
Cycle consistency. Define x̄α,k = G(x, c(α, k)). We add a cycle consistency loss [89]:
Lcyc(G) = 1
K K−1∑ k=0 Eα [∥∥∥x−G(x̄α,k, c (f(x), k) )∥∥∥ 1 ] . (3)
GAN loss. We use spectral normalization [53] with its hinge loss variant [52] to compare the distribution x ∼ PX and the generated data:2 LcGAN(D) = −Ex∼PX [ min ( 0,−1 +D(x) )] − Ex∼PXEα,k [ min ( 0,−1−D ( G (x, c(α, k)) ))] , (4)
LcGAN(G) = −Ex∼PXEα,k [ D ( G (x, c(α, k)) )] . (5)

heading: 3.2 ENFORCING DISENTANGLEMENT
text: Ideally, we want theK concepts to capture distinctively different qualities influential to the classifier being explained. Though G is capable of expressing K different concepts, the above formulation does not enforce distinctness among them. To address this challenge, we introduce a concept disen-
1Note that G is trained having access to f ’s signal. The LogLoss between the desired probability α and the predicted probability of the perturbed image is minimized. Thus, the model supports fidelity by design.
2One can use any GAN loss [2, 24, 27, 42, 53, 88]. Generated data comes from uniformly sampling k & α.
tangler R inspired by the disentanglement literature [8, 11, 12, 29, 38, 41, 43]. Most disentanglement measures quantify proxies of distinctness of changes across latent dimensions and similarity of changes within a latent dimension [45]. This is usually achieved through latent code traversal, keeping a particular dimension fixed, or keeping all but one dimension fixed. Thus, we design R to encourage distinctness across concepts and similarity within a concept.
The K concepts can be viewed as “knobs” that can modify a sample along a conceptual dimension. Without loss of generality, we assume modifying only one “knob” at a time. Given a pair of images (x, x′), R tries to predict which concept k, k ∈ {0, . . . ,K − 1}, has perturbed query x to produce x′. To formalize this, let x̄k,α = G(x, c(α, k)) and x̃k = G(x̄k,α, c(f(x), k)). The loss function is
Lr(G,R) = CE(ek, R(x, x̄k)) + CE(ek, R(x̄k, x̃k)), (6) where CE is the cross entropy and ek is a one-hot vector of size K for the kth concept. Note that while the kth “knob” can modify a sample query with α degree, α ∈ [0, 1/N, . . . , 1], R tries to solve a multi-class binary prediction problem. Thus, for any α, α 6= f(x), R(x, x̄k) is trained to predict a one-hot vector for the kth concept. Furthermore, note that no external labels are required to distinguish between concepts. Instead, by jointly optimizing R and G, we penalize the generator G if any concept knob results in similar-looking samples to other concept knobs. Encouraging noticeably distinct changes across concept knobs indirectly promotes disentanglement. In summary, the overall objective function of our method is:
min G,R max D
λcGANLcGAN(D,G) + λfLf (D,G) + λrecLrec(G) + λrecLcyc(G) + λrLr(G,R). (7)
We call our approach DISSECT as it disentangles simultaneous explanations via concept traversals.

heading: 4 EXPERIMENTS
text: 3D Shapes. We first use 3D Shapes [7] purely for validation and demonstration purposes due to the controllability of all its factors. It is a synthetic dataset available under Apache License 2.0 composed of 480K 3D shapes procedurally generated from 6 ground-truth factors of variation. These factors are floor hue, wall hue, object hue, scale, shape, and orientation.
SynthDerm. Second, we create a new dataset, SynthDerm (Fig. 2). Real-world characteristics of melanoma skin lesions in dermatology settings inspire the generation of this dataset [80]. These characteristics include whether the lesion is asymmetrical, its border is irregular or jagged, is unevenly colored, has a diameter more than 0.25 inches, or is evolving in size, shape, or color over time. These qualities are usually referred to as the ABCDE of melanoma [65]. We generate SynthDerm algorithmically by varying several factors: skin tone, lesion shape, lesion size, lesion location (vertical and horizontal), and whether there are surgical markings present. We randomly assign one of the following to the lesion shape: round, asymmetrical, with jagged borders, or multicolored (two different shades of colors overlaid with salt-and-pepper noise). For skin tone values, we simulate Fitzpatrick ratings [20]. Fitzpatrick scale is a commonly used approach to classify the skin by its reaction to sunlight exposure modulated by the density of melanin pigments in the skin. This rating has six values, where 1 represents skin with lowest melanin and 6 represents skin with highest melanin. For our synthetic generation, we consider six base skin tones that similarly resemble different amounts of melanin. We also add a small amount of random noise to the base color to add further variety. SynthDerm includes more than 2,600 64x64 images.
CelebA. We also include the CelebA dataset [44] that is available for non-commercial research purposes. This dataset includes “identities [of celebrities] collected from the Internet” [78], is realistic, and closely resembles real-world settings where the attributes are nuanced and not mutually
independent. CelebA includes more than 200K images with 40 annotated face attributes, such as smiling and bangs.

heading: 4.1 BASELINES
text: We consider several baselines. First, we modify a set of VAEs explicitly designed for disentanglement [8, 11, 29, 41] to incorporate the classifier’s signal during their training processes and encourage the generative model to learn latent dimensions that influence the classifier, i.e., learning Important CTs. While these approaches are not naturally designed to explain an external classifier, they have been designed to find distinct factors of variation. We refer to these baselines with a - mod postfix, e.g. β-VAE-mod. Second, we include CSVAE [38] that aims to solve unsupervised learning of features associated with a specific label using a low-dimensional latent subspace that can be independently manipulated. CSVAE, without any modification, is the closest alternative in the literature that tries to discover both Important and Distinct CTs. Third, we include Explanation by Progressive Exaggeration (EPE) [73], a GAN-based approach that learns to generate one series of counterfactual and realistic samples that change the prediction of f , given data and the classifier’s signal. EPE is the most similar baseline to DISSECT in terms of design choices; however, it is not designed to handle distinct CTs. We introduce EPE-mod by modifying EPE to allow learning multiple pathways by making the generator conditional on another variable: the CT dimension (see more details in §A.1).

heading: 4.2 EVALUATION METRICS
text: This section provides a brief summary of our evaluation metrics. We consider Importance, Realism, Distinctness, Substitutability, and Stability, which commonly appear as desirable qualities in explainability literature.3 See more in §A.3. Importance. Explanations should produce the desired outcome from the black-box classifier f . Previous work has referred to this quality using different terms, such as importance [22], compatibility with classifier [73], and classification model consistency [74]. While most previous methods have relied on visual inspection, we introduce a quantitative metric to measure the gradual increase of the target class’s posterior probability through a CT. Notably, we compute the correlation between α and f(I(x, α, k; f)) introduced in § 3. For brevity, we refer to f(I(x, α, k; f)) as f(x̄) in the remainder of the paper. We also report the mean-squared error and the Kullback–Leibler (KL) divergence between α and f(x̄). We also calculate the performance of f on the counterfactual explanations. Specifically, we replace the test set of real images with their generated counterfactual explanations and quantify the performance of the pre-trained black-box classifier f on the counterfactual test set. Better performance suggests that counterfactual samples are compatible with f and lie on the correct side of the classifier’s boundary. Note that Importance scores can be calculated per concept and provide a ranking across concepts, or they can be aggregated to summarize overal performance. See §A.7 for more information about concept ranking using individial concept Importance scores. Realism. We need the generated samples that form a CT to look realistic to enable users to identify concepts they represent. It means the counterfactual explanations should lie on the data manifold. This quality has been referred to as realism or data consistency [73]. We use Fréchet Inception Distance (FID) [28] as a measure of Realism. Additionally, inspired by [18], we train a post hoc classifier that predicts whether a sample is real or generated. Note that it is essential to do this step post hoc and independent from the training procedure because relying on the discriminator’s accuracy in an adversarial training framework can be misleading [18]. Distinctness. Another desirable quality for explanations is to represent inputs with non-overlapping concepts, often referred to as diversity [50]. Others have suggested similar properties such as coherency, meaning examples of a concept should be similar but different from examples of other concepts [22]. To quantify this in a counterfactual generation setup, we introduce a distinctness measure capturing the performance of a secondary classifier that distinguishes between CTs. We train a classifier post hoc that given a query image x and a generated image x′ and K number of CTs, predicts CTk that has perturbed x to produce x′, k ∈ {1, . . . ,K}. This classifier is agnostic to our model and only uses (x, x’) pair.
3Eq. 1 encourages Importance, Eq. 5 encourages Realism, Eq. 6 encourages Distinctness, Eq. 2 and Eq. 3 indirectly help with Stability. All the elements of Eq. 7 contribute to improving Substitutability.
Substitutability. The representation of a sample in terms of concepts should preserve relevant information [50, 60]. Previous work has formalized this quality for counterfactual generation contexts through a proxy called substitutability [68]. Substitutability measures an external classifier’s performance on real data when it is trained using only synthetic images.4 A higher substitutability score suggests that explanations have retained relevant information and are of high quality. Stability. Explanations should be coherent for similar inputs, a quality known as stability [21, 50, 60]. To quantify stability in counterfactual explanations, we augment the test set with additive random noise to each sample x and produce S randomly augmented copies x ′′
i . Then, we generate counterfactual explanations x̄ and x̄ ′′
i , respectively. We calculate the mean-squared difference between counterfactual images x̄ and x̄ ′′
i and the resulting Jensen Shannon distance (JSD) between the predicted probabilities f(x̄) and f(x̄ ′′
i ).

heading: 4.3 CASE STUDY I: VALIDATING THE QUALITIES OF CONCEPT TRAVERSALS
text: Considering 3D Shapes [7], we define an image as “colored correctly” if the shape hue is red or the floor hue is cyan. We train a classifier to detect whether a sample image is “colored correctly” or not. In this synthetic experiment, only these two independent factors contribute to the decision of this classifier. Given a not “colored correctly” query, we would like the algorithm to find a CT related to the shape color and another CT associated with the floor color–two different pathways that lead to switching the classifier outcome.5
Tab. 1 summarizes the quantitative results on 3D Shapes. Most VAE variants perform poorly in terms of Importance, CSVAE performs slightly better, and EPE, EPE-mod, and DISSECT perform best. Our results suggest that DISSECT performs similarly to EPE that has been geared explicitly toward exhibiting Importance and its extension, EPE-mod. Additionally, DISSECT still keeps Realism intact. Also, it notably improves the Distinctness of CTs compared to relevant baselines.
4This metric has been used in other contexts outside of explainability and has been called Classification Accuracy Score (CAS) [62]. CAS is more broadly applicable than Fréchet Inception Distance [28] & Inception Score [67] that are only useful for GAN evaluation.
5In this scenario, these two ground-truth concepts do not directly apply to switching the classifier outcome from True to False. For example, if an image has a red shape and a cyan floor, both of these colors need to be changed to switch the classification outcome. We still observe that DISSECT finds CTs that change different combinations of colors, but the baseline methods converge to the same CT. See §A.5.1 for more details.
Fig. 3 shows the examples illustrating the qualitative results for EPE, EPE-mod, and DISSECT. Our results reveal that EPE converges to finding only one of these concepts. Similarly, both CTs generated by EPE-mod converge to finding the same concept, despite being given the capability to explore two pathways to switch the classifier outcome. However, DISSECT finds the two distinct ground-truth concepts through its two generated CTs. This is further evidence for the necessity of the disentanglement loss to enforce the distinctness of discovered concepts. For brevity, three sample queries are visualized (See more in §A.5.1). Note that given the synthetic nature of this dataset, the progressive paths may not be as informative. Thus, we only visualize the far end of the spectrum, i.e., the example that flips the classification outcome all the way. See §4.5 for the progressive exaggeration of concepts in a more nuanced experiment.

heading: 4.4 CASE STUDY II: IDENTIFYING SPURIOUS ARTIFACTS
text: A “high performance” model could learn to make its decisions based on irrelevant features that only happen to correlate with the desired outcome, known as label leakage [85]. One of the applications of DISSECT is to uncover such spurious concepts and allow probing a black-box classifier. Motivated by real-world examples that revealed classifier dependency on surgical markings in identifying melanoma [86], we design this experiment. Given the synthetic nature of SynthDerm and how it has been designed based on real-world characteristics of melanoma [65, 80], each sample has a deterministic label of melanoma or benign. If the image is asymmetrical, has jagged borders, has different colors represented by salt-and-pepper noise, or has a large diameter (i.e., does not fit in a 40×40 square), the sample is melanoma. Otherwise, the image represents the benign class. Similar to in-situ dermatology images, melanoma samples have surgical markings more frequently than benign samples. We train a classifier to distinguish melanoma vs. benign lesion images.
Given a benign query, we would like to produce counterfactual explanations that depict how to modify the input sample to change its class membership. We want DISSECT to find CTs that disentangle meaningful characteristics of melanoma identification in terms of color, texture, or shape [65], and identify potential spurious artifacts that impact the classifier’s predictions.
Tab. 2 summarizes the quantitative results on SynthDerm. Our method performs consistently well across all the metrics, significantly boosting Distinctness and Substitutability scores and making meaningful improvements on Importance scores. Our approach has higher performance compared to EPE-mod and EPE baselines and substantially improves upon CSVAE. Our method’s high Distinctness and Substitutability scores show that DISSECT covers the landscape of potential concepts very well and retains the variety seen in real images strongly better than all the other baselines.
Fig. 4 illustrates a few examples to showcase DISSECT’s improvements over the strongest baseline, EPE-mod. EPE-mod converges to finding a single concept that only vaguely represents meaningful ground-truth concepts. However, DISSECT successfully finds concepts describing asymmetrical shapes, jagged borders, and uneven colors that align with ABCDE of melanoma [65]. DISSECT also identifies surgical markings as a spurious concept impacting the classifier’s decisions. Overall, the qualitative results show that DISSECT uncovers several critical blind spots of baselines.

heading: 4.5 CASE STUDY III: IDENTIFYING BIASES
text: Another potential use case of DISSECT is to identify biases that might need to be rectified. Since our approach does not depend on predefined user concepts, it may help discover unkwown biases. We design an experiment to test DISSECT in such a setting. We sub-sample CelebA to create a training dataset such that smiling correlates with “blond hair” and “bangs” attributes. In particular, positive samples either have blond hair or have bangs, and negative examples are all dark-haired and do not have bangs. We use this dataset to train a purposefully biased classifier. We employ DISSECT to generate two CTs. Fig. 5 shows the qualitative results, which depict that DISSECT automatically discovers the two biases, which other techniques fail to do. Tab. 3 summarizes the quantitative results that replicate our finding from Tab. 1 in § 4.3 and Tab. 2 in § 4.4 in a real-world dataset, confirming that DISSECT quantitatively outperforms all the other baselines in Distinctness without negatively impacting Importance or Realism.

heading: 5 CONCLUDING REMARKS
text: We present DISSECT that successfully finds multiple distinct concepts by generating a series of realistic-looking, counterfactually generated samples that gradually traverse a classifier’s decision boundary. We validate our approach via quantitative and qualitative experiments, showing significant improvement over prior methods. Mirroring natural human reasoning for explainability [3, 4, 6, 83], the new method provides additional checks-and-balances for practitioners to probe a trained model prior to deployment, especially in high-stakes tasks. DISSECT helps identify whether the model-under-test reflects practitioners’ domain knowledge and discover biases that weren’t previously known to users.
One avenue for future explainability work is extending earlier theories [45, 71] that obtain disentanglement guarantees. While DISSECT uses an unsupervised approach for concepts discovery, it is possible to modify the training procedure and the concept disentangler network to allow for concept supervision, which can be another avenue for future work. Furthermore, extending this technique to domains beyond visual data such as time series or text-based input can broaden this work’s impact. While we provide an extensive list of qualitative examples in the Appendix, further confirmation from human-subject studies to validate that CTs exhibit semantically meaningful attributes could strengthen our findings.

heading: 6 ETHICS STATEMENT
text: We emphasize that our proposed method does not guarantee finding all the biases of a classifier, nor ensures semantic meaningfulness across all found concepts. Additionally, it should not replace procedures for promoting transparency in model evaluation such as model cards [51]; instead, we recommend it be used in tandem. We also acknowledge the limitations for Fitzpatrick ratings used in SynthDerm dataset in capturing variability, especially for darker skin tones [59, 77], and would like to consider other ratings for future.

heading: 7 REPRODUCIBILITY STATEMENT
text: The code for all the models and metrics is publicly available at [Anon. Link]. Additionally, we released the new dataset, which is publicly available at [Anon. Link]. See §A.3.6 and §A.4 for additional details about evaluation metrics, experiment setup, and hyperparameter selection.

heading: A APPENDIX

heading: A.1 BASELINES

heading: A.1.1 BASELINE 1: MULTI-MODAL EXPLAINABILITY THROUGH VAE-BASED DISENTANGLEMENT
text: Disentanglement approaches have demonstrated practical success in learning representations that correspond to factors for variation in data [71], though some gaps between theory and practice remain [45]. However, the extent to which these techniques can aid post hoc explainability in conjunction with an external model is not well understood. Thus, we consider a set of baseline approaches based on VAEs explicitly designed for disentanglement: β-VAE [29], Annealed-VAE [8], β-TCVAE [11], and DIPVAE [41]. We extend each of them to incorporate the classifier’s signal during their training processes for a fair comparison with DISSECT. Intuitively speaking, this encourages the generative model to learn latent dimensions that could influence the classifier, i.e., learning Influential CTs. Note that it is necessary to also cover data generative factors that are independent of the external classifier. This means that, along with having a good quality reconstruction and high likelihood, we need to promote sparsity in sensitivity of the external classifier to the latent dimensions of the generative model.
More formally, consider a vanilla VAE that has an encoder eθ with parameters θ, a decoder dφ with parameters φ, and the M -dimensional latent code z with prior distribution p(z). Recall that x denotes the input sample. The objective of a VAE is to minimize the loss:
Lvanilla VAEθ,φ = −Ez∼eθ(z|x)[log dφ(x|z)] + KL(eθ(z|x)||p(z)).
We introduce an additional loss term for incorporating the black-box classifier’s signal: 1/K ∑K k=1 ∂f(x)/∂zk. We impose this only for the first K dimensions in the latent space
6, in other words, the number of desired CTs, K ≤ M . Minimizing this term provides CTks, k ∈ {1, 2, . . . ,K}, with negative ∂f(x)/∂zk, with a high |∂f(x)/∂zk|. The final loss is:
Lθ,φ = Lvanilla VAEθ,φ + λ ∗ ∑K k=1 ∂f(x)/∂zk
K ,
6Without loss of generality, the additional term can be applied to the first K dimensions, and there is no need to consider ( K M ) potential selections.
where λ is a hyper-parameter. We apply this modification to the four aforementioned VAE-based approaches and refer to them with a -mod postfix, e.g. β-VAE-mod.7
Development process. To promote discovering Important CTs, we introduced Laux =∑K d=1 ∂f(x)/∂zd K , which incorporated the directional derivative of f with respect to the latent dimensions of interest into the loss function of VAE. Despite experimentation with many variants of Laux, we observed two common themes. First, a monotonic increase of f(x̄) through traversing one latent dimension and keeping the rest static was hardly achieved. Second, while the purpose of Laux was to promote exerting Importance only in the first K dimensions of the latent space, ∂f(x)/∂zd for d ∈ {K+1,K+2, · · · ,M} were impacted similarly. Having strongly correlated dimensions is a failure in achieving the very goal of disentanglement approaches. Table 4 summarizes a subset of the variants of Laux studied.

heading: A.1.2 BASELINE 2: MULTI-MODAL EXPLAINABILITY THROUGH CONDITIONAL SUBSPACE
text: Another relevant area of work is conditional generation. In particular, Conditional subspace VAE (CSVAE) is a method aiming to solve unsupervised learning of features associated with a specific label using a low-dimensional latent subspace that can be independently manipulated [38]. CSVAE partitions the latent space into two parts: w learns representations correlated with the label, and z covers the remaining characteristics for data generation. An assumption of independence between z and w is made. To explicitly enforce independence in the learned model, we minimize the mutual information between Y and Z. CSVAE has proven successful in providing counterfactual scenarios to reverse unfavorable decisions of an algorithm, also known as algorithmic recourse [17]. To adjust CSVAE to explain the decision-making of an external classifier f , we treat the predictions of the classifier as the label of interest.
More formally, the generative model can be summarized as:
w|y ∼ N(µy, σ2y.I), y ∼ Bern(p), x|w, z ∼ N(dφµ(w, z), σ2 .I), z ∼ N(0, σ2z .I)
Conducting inference leads to the following objective function:
M1 = ED(x,y)[−Eqφ(z,w|x,y)[log pθ(x|w, z)]+KL(qφ(w|x, y)||pγ(w|y))+KL(qφ(z|x, y)||p(z))−log p(y)] M2 =Eqφ(z|x)D(x)[ ∫ Y qδ(y|z) log qδ(y|z) dy]
M3 =Eq(z|x)D(x,y)[qδ(y|z)] 7We build these baselines on top of the open-sourced implementations provided in https://github.
com/google-research/disentanglement_lib under Apache License 2.0.
min θ,φ,γ β1M1 + β2M2; max δ β3M3

heading: A.1.3 BASELINE 3: MULTI-MODAL EXPLAINABILITY THROUGH PROGRESSIVE EXAGGERATION
text: Explanation by Progressive Exaggeration (EPE) [73] is a recent successful generative approach that learns to generate one series of counterfactual and realistic samples that change the prediction of f , given data and the classifier’s signal. It is particularly relevant to our work as it explicitly optimizes Influence and Realism. EPE is a type of Generative Adversarial Network (GAN) [24] consisting of a discriminator (D) and a generator (G) that is based on Projection GAN [52]. It incorporates the amount of desired perturbation α on the outcome of f as:
LcGAN(D) = −Ex∼pdata [min(0,−1 +D(x, 0))]− Ex∼pdata [min(0,−1−D(G(x, α), α))] (8)
LcGAN(G) = −Ex∼pdata [D(G(x, α), α)] (9) A Kullback–Leibler divergence (KL) term in the objective function between the desired perturbation (α) and the achieved one (f(G(x, α))) promotes Importance [73]:
Lf (D,G) = r(D,G(x, α)) + KL(α|f(G(x, α))), (10)
where the first term is the likelihood ratio defined in projection GAN [24], and [73] uses an ordinalregression parameterization of it.
A reconstruction loss and a cycle loss promote self-consistency in the model, meaning that applying a reverse perturbation or no perturbation should reconstruct the query sample:
Lrec(G) = ||x−G(x, f(x))||1 (11)
Lcyc(G) = ||x−G(G(x, α), f(x))||1. (12) Thus, the overall objective function of EPE is the following:
min G max D
λcGANLcGAN(D,G) + λfLf (D,G) + λrecLrec(G) + λrecLcyc(G), (13)
where λcGAN, λf , and λrec are the hyper-parameters.
Note that EPE only finds one pathway to switch the classifier’s outcome. We argue that classifiers learned from challenging and realistic datasets will have complex reasoning pathways that could enhance model explainability if revealed. Decomposing this complexity is needed to make reasoning comprehensible for humans. We thus create a more powerful baseline, an EPE-variant, EPE-mod. EPE-mod learns multiple pathways by making the generator conditional on another variable: the CT dimension. More formally, EPE-mod updates G(·, ·) to G(·, ·, k) in Eq. equation 8-equation 12, while Eq. equation 13 remains unchanged. We compare DISSECT to both EPE and EPE-mod.

heading: A.2 DISSECT DETAILS
text: We build our proposed method on EPE-mod and further promote distinctness across CTs by adding a disentangler network, R. The disentangler is a classifier with K classes. Given a pair of (x, x′) images, R tries to predict which CTk;k∈{1,...,K} has perturbed query x to produce x′. Note that R can return close to 0 probability for all classes if x′ is just a reconstruction of x, indicating no tweaked dimensions. The disentangler also penalizes the generator if any CTs use similar pathways to cross the decision boundary. See the appendix for schematics of our method.
To formalize this, let:
x̂k = G(x, f(x), k)
x̄k = G(x, α, k)
x̃k = G(x̄k, f(x), k).
Note that x̂k and x̃k are reconstructions of x while x̄k is perturbed to change the classifier output from f(x) to α. Therefore, x, x̄k and x̃k form a cycle, and k represents CTk. R(., .) is the predicted probabilities of the perturbed concept given a pair of examples, which is a vector of size K, where
each element is a value in [0, 1]. We define the following cross entropy loss that is a function of both R and G:
Lr(G,R) = CE(ek, R(x, x̄k)) + CE(ek, R(x̄k, x̃k))
= −Ex∼pdata K∑ k=1 [eklogR(x, x̄k) + eklogR(x̄k, x̃k)] (14)
Here, ek refers to a one-hot vector of size K where the k-th element is one and the remaining elements are zero. This term enforces R to identify no change when receiving reconstructions of the same image as input and utilizes the cycle and promotes determining the correct dimension when a
non-zero change has happened, either increasing or decreasing the outcome of f . In summary, the overall objective function of our method is:
min G,R max D
[λcGANLcGAN(D,G) + λfLf (D,G) + λrecLrec(G) + λrecLcyc(G)
+ λrLr(G,R)] (15)
For this adversarial min-max optimization, we use the Adam optimizer [37]. See Figure 6 for a for a visualization of DISSECT .We open-source our implementation at [Anon. link] under MIT license, which builds on top of the open-source implementation of EPE [73] 8.

heading: A.3 EVALUATION METRICS DETAILS
text: In this section, we provide more details about the evaluation metrics. First, we provide evidence supporting these criteria as desirable qualities for explainability. Then, we summarize how these qualities have been measured in prior work. Finally, we explain when we can use prior formulation of these metrics as is, and when we need to modify them to make them applicable to the counterfactual explanation case. We justify how these formulations capture the desired criteria.
A.3.1 IMPORTANCE
Motivation. Explanations should produce the desired outcome from the black-box classifier f . Previous work has referred to this quality using different terms, such as importance [22], compatibility with classifier [73], and classification model consistency [74].
Quantification. Most previous methods have relied on visual inspection. Singla et al. [74] have plotted expected outcome from the classifier, against the actual classifier prediction of the generated explanations. Ghorbani et al. [22] have plotted prediction accuracy vs. adding or removal of discovered concepts.
We introduce a quantitative metric to measure the gradual increase of the target class’s posterior probability through a CT. Notably, we compute the correlation between α and f(I(x, α, k; f)) introduced in § 3. For brevity, we refer to f(I(x, α, k; f)) as f(x̄) in the remainder of the paper. We also report the mean-squared error and the Kullback–Leibler (KL) divergence between α and f(x̄). This is one way to summarize information in the plots used by Singla et al. [74] for Importance evaluation.
We also calculate the performance of f on the counterfactual explanations. Specifically, we replace the test set of real images with their generated counterfactual explanations and quantify the performance of the pre-trained black-box classifier f on the counterfactual test set. Better performance suggests that counterfactual samples are compatible with f and lie on the correct side of the classifier’s boundary. This is one way to summarize the information in the plots used by Ghorbani et al. [22] for Importance evaluation.

heading: A.3.2 REALISM
text: Motivation. We need the generated samples that form a CT to look realistic to enable users to identify concepts they represent. It means the counterfactual explanations should lie on the data manifold. This quality has been referred to as realism or data consistency [73].
Quantification. A variety of inception scores have perviously been used to quantify visual quality of generated images, such as Fréchet Inception Distance [28] and Inception Score [67].
Similar to prior work, we use FID [28]. However, since FID is based on an Inception V3 network trained on ImageNet, it could be a better fit for more realistic data that has similar distribution to ImageNet. This is not the case, especially for 3d Shapes and SynthDerm in our work. Therefore, we include an additional metric inspired by [18]. We train a post hoc classifier that predicts whether a sample is real or generated. We report performance of this classifier as a proxy for Realism.
8https://github.com/batmanlab/Explanation_by_Progressive_Exaggeration available under MIT License.

heading: A.3.3 DISTINCTNESS
text: Motivation. Another desirable quality for explanations is to represent inputs with non-overlapping concepts, often referred to as diversity [50]. Others have suggested similar properties such as coherency, meaning examples of a concept should be similar but different from examples of other concepts [22].
Quantification. To quantify this in a counterfactual generation setup, we introduce a distinctness measure capturing the performance of a secondary classifier that distinguishes between CTs. We train a classifier post hoc that given a query image x and a generated image x′ and K number of CTs, predicts CTk that has perturbed x to produce x′, k ∈ {1, . . . ,K}. This

heading: A.3.4 SUBSTITUTABILITY
text: Motivation. The representation of a sample in terms of concepts should preserve relevant information [50, 60]. Previous work has formalized this quality for counterfactual generation contexts through a proxy called substitutability [68].
Quantification. Substitutability measures an external classifier’s performance on real data when it is trained using only synthetic images. This metric has been used in other contexts outside of explainability and has been called Classification Accuracy Score (CAS) [62]. CAS is more broadly applicable than Fréchet Inception Distance [28] & Inception Score [67] that are only useful for GAN evaluation. Furthermore, CAS can reveal information that none of these inception scores successfully capture [62]. A higher substitutability score suggests that explanations have retained relevant information and are of high quality.
We follow Samangouei et al. [68] for formulation of this metric.

heading: A.3.5 STABILITY
text: Motivation. Explanations should be coherent for similar inputs, a quality known as stability [21, 50, 60].
Quantification. To quantify stability in counterfactual explanations, we augment the test set with additive random noise to each sample x and produce S randomly augmented copies x ′′
i . Then, we generate counterfactual explanations x̄ and x̄ ′′
i , respectively. We calculate the mean-squared difference between counterfactual images x̄ and x̄ ′′
i and the resulting Jensen Shannon distance (JSD) between the predicted probabilities f(x̄) and f(x̄ ′′
i ).
A.3.6 IMPLEMENTATION DETAILS
The VAE-based baselines support continuous values for latent dimensions zk. Also, we can directly sample latent code values and produce CTk by keeping zj (j 6= k) constant and monotonically increasing zk values. However, to calculate the evaluation metrics comparably to EPE, EPE-mod, and DISSECT, we do the following: We encode each query sample using the probabilistic encoder. We set zj = µj , j 6= k where µj is the mean of the fitted Gaussian distribution for zj . For dimension k, we produce N + 1 linearly spaced values between µp ± 2 ∗ σp, where µp and σp are the mean and standard deviation of the prior normal distribution, in our case 0.0 and 1.0 respectively. Note that these different values for zk map out to α, α ∈ {0, 1N , · · · , 1} in EPE, EPE-mod, and DISSECT models. After this step, calculating all the metrics related to Importance, Realism, and Distinctness is identical across all the models.

heading: A.4 EXPERIMENT SETUP AND HYPER-PARAMETER TUNING DETAILS
text: Experiments were conducted on an internal compute cluster at ¡Anonymous Institution. Training and evaluation of all models across the three datasets have approximately taken 1000 hours on a combination of Nvidia GPUs including GTX TITAN X, GTX 1080 Ti, RTX 2080 Rev, and Quadro K5200.
We seeded the model’s parameters from [73] based on the reported values in their accompanying open-sourced repository.9 We used the same parameters for 3D Shapes, except for the number of bins, N , used for ordinal regression transformation of the classifier’s posterior probability. The largest number of bins that resulted in non-zero samples per bin, 3, was selected. We kept all the parameters shared between EPE, EPE-mod, and DISSECT the same.
Given the experiments’ design, we fixed the number of dimensions K in DISSECT and EPE-mod to 2. We experimented with a few values for λr, 1, 10, 20, 50. Based on manual inspection after 30k training batches, λr was selected. Factors considered for selection included inspecting the perceived quality of generated samples and the learning curves of LcGAN(D), LcGAN(G), Lcyc(G), Lrec(G), and Lr(G,R). For evaluation, we used a hold-out set including 10K samples. For post hoc evaluation classifiers predicting Distinctness and Realism, 75% of the samples were used for training, and the results were reported on the remaining 25%. See Table 5 for the summary of the hyper-parameter values.

heading: A.5 ADDITIONAL QUALITATIVE RESULTS

heading: A.5.1 CASE STUDY I
text: Recall that considering 3D Shapes, we define an image as “colored correctly” if the shape hue is red or the floor hue is cyan. Given a not “colored correctly” query, we recover a CT related to
9https://github.com/batmanlab/Explanation_by_Progressive_Exaggeration available under MIT License.
the shape color and another CT associated with the floor color–two different pathways leading to switching the classifier outcome for that sample. See Figure 7 for additional qualitative examples where classification outcome is flipped from False to True.
However, these two ground-truth concepts do not directly apply to switching the classifier outcome from True to False in this scenario. For example, if an image has a red shape and a cyan floor, both colors need to be changed to switch the classification outcome. As shown in Figure 8, we still observe that applying DISSECT to such cases results in two discovered CTs that change different combinations of colors while EPE-mod converges to the same CT.

heading: A.5.2 CASE STUDY III
text: Recall the biased CelebA experiment where smiling correlates with “blond hair” and “bangs” attributes. Figure 9 shows additional qualitative samples, suggesting that DISSECT can recover and separate the aforementioned concepts, which other techniques fail to do.

heading: A.5.3 LIMITATIONS
text: While we provide several qualitative examples, further confirmation from human-subject studies to validate that CTs exhibit semantically meaningful attributes could strengthen our findings.
The number of concepts to be discovered by DISSECT is a hyper parameter that can be selected by the user. While Distinctness and Substitutability are designed to be globally evaluated across all concepts, Importance, Realism, and Stability scores can be calculated for each concept. We hypothesize that ranking the discovered concepts based on these individual scores can benefit the users. In particular, it can help them focus on more salient concepts and not be overwhelmed by less informative concepts. Future user-studies are required to investigate these hypotheses.
We emphasize that our proposed method does not guarantee finding all the biases of a classifier, nor ensures semantic meaningfulness across all found concepts. One avenue for future explainability work is extending earlier theories [45, 71] that obtain disentanglement guarantees.

heading: A.6 ABLATION/SENSITIVITY EXPERIMENTS
text: EPE-mod can be viewed as the ablated version of DISSECT where λr = 0. For a more detailed analyses of the influence of the disentanglement element, we conducted an experiment with different λr while fixing the other hyperparameters of the model. As seen in Table 6, increasing λr up to a point improves Distinctness and retains performance on the other criteria of interest. Further increasing λr hurts the quality of generated images, and thus negatively impacts Realism and Substitutability.
A.7 INDIVIDUAL CONCEPT INFLUENCE
Note that metrics such as Distinctness and Substitutability are not meaningful for each concept. However, Importance, Realism, and Stability can be calculated on each concept separately, or over all concepts. Calculating such metrics for each concept can help us rank them with respect to that criterion. For example, Importance scores per concept can tell us which concept is more important in the decision making of the classifier. Table 7 summarizes the Importance scores per concept. As expected, given the symmetric design of our experiments, all discovered concepts exhibit similar Importance scores.
A.8 INTERACTION BETWEEN GENERATION QUALITY AND EXPLANATION QUALITY
In this section, we evaluate DISSECT with different sizes of underlying GAN architectures. As shown in Table 8, even with imperfect generation quality, DISSECT can still discover relevant con-
cepts with comparable performance in terms of Importance, Distinctness. This is despite significant impacts on the Realism scores. Perfect generation quality can further improve realism. However, a reasonable generation quality is sufficient for successful explanation with DISSECT.
With the recent advances in generative modeling that have achieved great generation quality over a majority of domains, it is timely to utilize them when available for generating realistic-looking explanations. It is worth mentioning that a perfect generation alone is not sufficient for good explanations in terms of performance criteria other than Realism. For comparison, we have calculated performance scores of a strong generative model, excluding the interpretability-related terms in the objective function. See the results in 8 for the GAN-only row.

Review summaries:
This paper proposed a generation-based explainability method, DISSECT, that generates Concept Traversals to provide a counterfactual explanation. The Concept traversals (CTs) are sequences of generated examples with increasing degrees of concepts' influence on the classifier's decision. The CTs are able to provide disentangled explanations for different concepts. 

The authors introduce DISSECT: Disentangled Simultaneous Explanations via Concept Traversal. The method creates counterfactual explanations for which the concept increasingly influences the classifier's decision. Experimentally, the authors show that the proposed method generally performs better than existing methods on several aspects ("importance", "realism", "distinctness", "stability"). 

This paper proposes a generative approach to model explanations, where explanations are composed of automatically generated counterfactual examples that differ by the observed magnitude of a given learned concept. Through a series of extensive experiments, the authors show that the proposed approach, DISSECT, can successfully disentangle useful and realistic concepts.  The main contributions of the paper is the novel approach that generates several counterfactual examples across a concept spectrum, and a new synthetic dermatology dataset that allows testing such methods where they matter most.