Beyond Trivial Counterfactual Generations with Diverse Valuable Explanations

1 INTRODUCTION . Consider a face authentication system for unlocking a device . In case of non-authentications ( possible false-negative predictions ) , this system could provide generic advices to its user such as “ face the camera ” or “ remove any face occlusions ” . However , these may not explain the reason for the possible malfunction . To provide more insights regarding its decisions , the system could instead provide information specific to the captured image ( its input data ) . It might list the input features that most contributed to its decision ( e.g. , a region of the input image ) , but this feature could be “ face ” , which is trivial and does not suggest an alternative action to its user . Further , it provides little useful information about the model . Instead , non-trivial explanations may be key for better understanding and diagnosing the system— including the data it was trained on— and improving its reliability . Such explanations might improve systems across a wide variety of domains including in medical imaging [ 58 ] , automated driving systems [ 48 ] , and quality control in manufacturing [ 22 ] . The explainability literature aims to understand the decisions made by a machine learning ( ML ) model such as the aformentionned face authentication system . Counterfactual explanation methods [ 11 , 13 , 4 ] can help discover the limitations of a ML model by uncovering data and model biases . The counterfactual explanation methods provide perturbed versions of the input data that emphasize features that contributed most to the ML model ’ s output . For example , if an authentication system is not recognizing a user wearing sunglasses then the system could generate an alternative image of the user ’ s face without sunglasses that would be correctly recognized . This is different from other types of explainability methods such as feature importance methods [ 50 , 51 , 4 ] and boundary approximation methods [ 47 , 37 ] . The former highlight salient regions of the input but do not indicate how the ML model could achieve a different prediction . The second family of methods produce explanations that are limited to linear approximations of the ML model . Unfortunately , these linear approximations are often inaccurate . In contrast , counterfactual methods suggest changes in the input that would lead to a change in the corresponding output , providing information not only about where the change should be but also what the change should be . Counterfactual explanations should be actionable , i.e. , a user should be able to act on it . An actionable explanation would suggest feasible changes like removing sunglasses instead of unrealistic ones like adding more eyes to the user ’ s face . Counterfactual explanations that are valid , proximal , and sparse are more likely to be actionable [ 49 , 38 ] . That is , a counterfactual explanation that changes the outcome of the ML model ( valid ) by changing the minimal number of input features ( sparse ) , while remaining close to the input ( proximal ) . Generating a set of diverse explanations increases the likelihood of finding an actionable explanation [ 49 , 38 ] . A set of counterfactuals is diverse if each one proposes to change a different set of attributes . Intuitively , each of these explanations shed light on a different action that user can take to change the ML model ’ s outcome . Current counterfactual generation methods like xGEM [ 26 ] generates a single explanation that is far from the input . Thus , it fails to be proximal , sparse , and diverse . Progressive Exaggeration ( PE ) [ 53 ] provides higher-quality explanations more proximal than xGEM , but it still fails to provide a diverse set of explanations . In addition , the image generator of PE is trained on the same data as the ML model in order to detect biases thereby limiting their applicability . Moreover , like the previous methods in the literature , these two methods tend to produce trivial explanations . For instance , an explanation that suggests to increase the ‘ smile ’ attribute of a ‘ smile ’ classifier for an already-smiling face is trivial and it does not explain why a misclassification occurred . In this work , we focus on diverse valuable explanations ( DiVE ) , that is , valid , proximal , sparse , and non-trivial . We propose Diverse Valuable Explanations ( DiVE ) , an explainability method that can interpret a ML model by identifying sets of valuable attributes that have the most effect on the ML model ’ s output . DiVE produces multiple counterfactual explanations which are enforced to be valuable , and diverse resulting in more actionable explanations than the previous literature . Our method first learns a generative model of the data using a β-TCVAE [ 5 ] to obtain a disentangled latent representation which leads to more proximal and sparse explanations . In addition , the VAE is not required to be trained on the same dataset as the ML model to be explained . DiVE then learns a latent perturbation using constraints to enforce diversity , sparsity , and proximity . In order to generate nontrivial explanations , DiVE leverages the Fisher information matrix of its latent space to focus its search on the less influential factors of variation of the ML model . This mechanism enables the discovery of spurious correlations learned by the ML model . We provide experiments to assess whether our explanations are more valuable and diverse than current state-of-the-art . First , we assess their validity on the CelebA dataset [ 33 ] and provide quantitative and qualitative results on a bias detection benchmark [ 53 ] . Second , we show that the generated explanations are more proximal in terms of Fréchet Inception Distance ( FID ) [ 19 ] , which is a measure of similarity between two datasets of images commonly used to evaluate the generation quality of GAN . In addition , we evaluate the latent space closeness and face verification accuracy , as reported by Singla et al . [ 53 ] . Third , we assess the sparsity of the generated counterfactuals by computing the average change in facial attributes . Fourth , we show that DiVE is more successful at finding more non-trivial explanations than previous methods and baselines . In the supplementary material we provide additional results on the out-of-distribution performance of DiVE . We summarize the contributions of this work as follows : 1 ) We propose DiVE , an explainability method that can interpret a ML model by identifying the attributes that have the most effect on its output . 2 ) DiVE achieves state of the art in terms of the validity , proximity , and sparsity of its explanations , detecting biases on the datasets , and producing multiple explanations for an image . 3 ) We identify the importance of finding non-trivial explanations and we propose a new benchmark to evaluate how valuable the explanations are . 4 ) We propose to leverage the Fisher information matrix of the latent space for finding spurious features that produce non-trivial explanations . 2 RELATED WORK . Explainable artificial intelligence ( XAI ) is a suite of techniques developed to make either the construction or interpretation of model decisions more accessible and meaningful . Broadly speaking , there are two branches of work in XAI , ad-hoc and post-hoc . Ad-hoc methods focus on making mod- els interpretable , by imbuing model components or parameters with interpretations that are rooted in the data themselves [ 45 , 39 , 25 ] . Unfortunately , most successful machine learning methods , including deep learning ones , are uninterpretable [ 6 , 32 , 18 , 24 ] . Post-hoc methods aim to explain the decisions of non interpretable models . These methods can be categorized as non-generative and generative . Non-generative methods use information from a ML model to identify the features most responsible for an outcome for a given input . Approaches like [ 47 , 37 , 41 ] interpret ML model decisions by using derived information to fit a locally interpretable model . Others use the gradient of the ML model parameters to perform feature attribution [ 59 , 60 , 52 , 54 , 50 , 1 , 51 ] , sometimes by employing a reference distribution for the features [ 51 , 11 ] . This has the advantage of identifying alternative feature values that when substituted for the observed values would result in a different mode outcome . These methods are limited to small contiguous regions of features with high influence on the target model outcome . In so doing , they can struggle to provide plausible changes of the input that are actionable by an user in order to correct a certain output or bias of the model . Generative methods such as [ 7 , 5 , 4 ] propose plausible modifications of the input that change the model decision . However the generated perturbations are usually found in pixel space and thus are bound to masking small regions of the image without necessarily having a semantic meaning . Closest to our work are generative counterfactual explanation methods [ 26 , 9 , 15 , 53 ] which synthesize perturbed versions of observed data that result in a corresponding change of the model prediction . While these methods provide valid and proximal explanations for a model outcome , they fail to provide a diverse set of non-trivial explanations . Mothilal et al . [ 38 ] addressed the diversity problem by introducing a diversity constraint between a set of randomly initialized counterfactuals ( DICE ) . However , DICE shares the same problems as [ 7 , 4 ] since perturbations are directly performed on the observed feature space , and does not take into account trivial explanations . In this work we propose DiVE , a counterfactual explanation method that generates a diverse set of valid , proximal , sparse , and non-trivial explanations . Appendix A provides a more exhaustive review of the related work . 3 PROPOSED METHOD . We propose DiVE , an explainability method that can interpret a ML model by identifying the latent attributes that have the most effect on its output . Summarized in Figure 1 , DiVE uses an encoder , a decoder , and a fixed weight ML model . The ML model could be any function for which we have access to its gradients . In this work , we focus on a binary image classifier in order to produce visual explanations . DiVE consists of two main steps . First , the encoder and the decoder are trained in an unsupervised manner to approximate the data distribution on which the ML model was trained . Unlike PE [ 53 ] , our encoder-decoder model does not need to train on the same dataset that the ML model was trained on . Second , we optimize a set of vectors i to perturb the latent representation z generated by the trained encoder . The details of the optimization procedure are provided in Algorithm 1 in the Appendix . We use the following 3 main losses for this optimization : a counterfactual loss LCF that attempts to fool the ML model , an proximity loss Lprox that constrains the explanations with respect to the number of changing attributes , and a diversity loss Ldiv that enforces the explainer to generate diverse explanations with only one confounding factor for each of them . Finally , we propose several strategies to mask subsets of dimensions in the latent space to prevent the explainer from producing trivial explanations . Next we explain the methodology in more detail .

Summary: The authors propose interpreting the decision of a black-box (BB) image classifier using diverse counterfactual explanations. The proposed model consists of a pre-trained β-TCVAE, which learns to extract a disentangled latent representation for the input image. To generate explanations for a given image, the model optimizes to find n latent perturbations. Each decoded output from β-TCVAE is similar to the original image and produces a desired outcome from the BB classifier. To ensure the diversity among the n latent perturbations, the model minimizes the pairwise similarity loss between the latent perturbations. The model further performs spectral clustering to partition the latent space into different attributes. Thus, at inference time, for the same input image, multiple counterfactual images can be generated as explanations by changing different dimensions of the latent space. The experiments demonstrate the realistic quality of the explanations and their ability to discover bias in the BB classifier.