Getting a CLUE: A  Method for Explaining Uncertainty Estimates

1 INTRODUCTION . There is growing interest in probabilistic machine learning models , which aim to provide reliable estimates of uncertainty about their predictions ( MacKay , 1992 ) . These estimates are helpful in highstakes applications such as predicting loan defaults or recidivism , or in work towards autonomous vehicles . Well-calibrated uncertainty can be as important as making accurate predictions , leading to increased robustness of automated decision-making systems and helping prevent systems from behaving erratically for out-of-distribution ( OOD ) test points . In practice , predictive uncertainty conveys skepticism about a model ’ s output . However , its utility need not stop there : we posit predictive uncertainty could be rendered more useful and actionable if it were expressed in terms of model inputs , answering the question : “ Which input patterns lead my prediction to be uncertain ? ” Understanding which input features are responsible for predictive uncertainty can help practitioners learn in which regions the training data is sparse . For example , when training a loan default predictor , a data scientist ( i.e. , practitioner ) can identify sub-groups ( by age , gender , race , etc . ) under-represented in the training data . Collecting more data from these groups , and thus further constraining their model ’ s parameters , could lead to accurate predictions for a broader range of clients . In a clinical scenario , a doctor ( i.e. , domain expert ) can use an automated decision-making system to assess whether a patient should receive a treatment . In the case of high uncertainty , the system would suggest that the doctor should not rely on its output . If uncertainty were explained in terms of which features the model finds anomalous , the doctor could appropriately direct their attention . While explaining predictions from deep models has become a burgeoning field ( Montavon et al. , 2018 ; Bhatt et al. , 2020b ) , there has been relatively little research on explaining what leads to neural networks ’ predictive uncertainty . In this work , we introduce Counterfactual Latent Uncertainty Explanations ( CLUE ) , to our knowledge , the first approach to shed light on the subset of input space features that are responsible for uncertainty in probabilistic models . Specifically , we focus on explaining Bayesian Neural Networks ( BNNs ) . We refer to the explanations given by our method as CLUEs . CLUEs try to answer the question : “ What is the smallest change that could be made to an input , while keeping it in distribution , so that our model becomes certain in its decision for said input ? ” CLUEs can be generated for tabular and image data on both classification and regression tasks . An application of CLUE is to improve transparency in the real-world deployment of a probabilistic model , such as a BNN , by complementing existing approaches to model interpretability ( Ribeiro et al. , 2016 ; Sundararajan et al. , 2017 ; Chang et al. , 2019 ) . When the BNN is confident in its prediction , practitioners can generate an explanation via earlier feature importance techniques . When the BNN is uncertain , its prediction may well be wrong . This potentially wrong prediction could be the result of factors not related to the actual patterns present in the input data , e.g . parameter initialization , randomness in mini-batch construction , etc . An explanation of an uncertain prediction will be disproportionately affected by these factors . Indeed , recent work on feature attribution touches on the unreliability of saliency maps when test points are OOD ( Adebayo et al. , 2020 ) . Therefore , when the BNN is uncertain , it makes sense to provide an explanation of why the BNN is uncertain ( i.e. , CLUE ) instead of an explanation of the BNN ’ s prediction . This is illustrated in Figure 1 . Our code is at : github.com/cambridge-mlg/CLUE . We highlight the following contributions : • We introduce CLUE , an approach that finds counterfactual explanations of uncertainty in input space , by searching in the latent space of a deep generative model ( DGM ) . We put forth an algorithm for generating CLUEs and show how CLUEs are best displayed . • We propose a computationally grounded approach for evaluating counterfactual explanations of uncertainty . It leverages a separate conditional DGM as a synthetic data generator , allowing us to quantify how well explanations reflect the true generative process of the data . • We evaluate CLUE quantitatively through comparison to baseline approaches under the above framework and through ablative analysis . We also perform a user study , showing that CLUEs allow practitioners to predict on which new inputs a BNN will be uncertain . 2 PRELIMINARIES . 2.1 UNCERTAINTY IN BNNS . Given a dataset D= { x ( n ) , y ( n ) } Nn=1 , a prior on our model ’ s weights p ( w ) , and a likelihood function p ( D|w ) =∏Nn=1 p ( y ( n ) |x ( n ) , w ) , the posterior distribution over the predictor ’ s parameters p ( w|D ) ∝ p ( D|w ) p ( w ) encodes our uncertainty about what value w should take . Through marginalization , this parameter uncertainty is translated into predictive uncertainty , yielding reliable error bounds and preventing overfitting : p ( y∗|x∗ , D ) = ∫ p ( y∗|x∗ , w ) p ( w|D ) dw . ( 1 ) For BNNs , both the posterior over parameters and predictive distribution ( 1 ) are intractable . Fortunately , there is a rich literature concerning approximations to these objects ( MacKay , 1992 ; Hernández-Lobato & Adams , 2015 ; Gal , 2016 ) . In this work , we use scale-adapted Stochastic Gradient Hamiltonian Monte Carlo ( SG-HMC ) ( Springenberg et al. , 2016 ) . For regression , we use heteroscedastic Gaussian likelihood functions , quantifying uncertainty using their standard deviation , σ ( y|x ) . For classification , we take the entropy H ( y|x ) of categorical distributions as uncertainty . Details are given in Appendix B. Hereafter , we useH to refer to any uncertainty metric , be it σ or H . Predictive uncertainty can be separated into two components , as shown in Figure 2 . Each conveys different information to practitioners ( Depeweg , 2019 ) . Irreducible or aleatoric uncertainty is caused by inherent noise in the generative process of the data , usually manifesting as class overlap . Model or epistemic uncertainty represents our lack of knowledge about w. Stemming from a model being under-specified by the data , epistemic uncertainty arises when we query points off the training manifold . Capturing model uncertainty is the main advantage of BNNs over regular NNs . It enables the former to be used for uncertainty aware tasks , such as OOD detection ( Daxberger & HernándezLobato , 2019 ) , continual learning ( Nguyen et al. , 2018 ) , active learning ( Depeweg et al. , 2018 ) , and Bayesian optimization ( Springenberg et al. , 2016 ) . 2.2 UNCERTAINTY SENSITIVITY ANALYSIS . To the best of our knowledge , the only existing method for interpreting uncertainty estimates is Uncertainty Sensitivity Analysis ( Depeweg et al. , 2017 ) . This method quantifies the global importance of an input dimension to a chosen metric of uncertainty H using a sum of linear approximations centered at each test point : Ii = 1 |Dtest| |Dtest|∑ n=1 ∣∣∣∣∂H ( yn|xn ) ∂xn , i ∣∣∣∣ . ( 2 ) As discussed by Rudin ( 2019 ) , linear explanations of non-linear models , such as BNNs , can be misleading . Even generalized linear models , which are often considered to be “ inherently interpretable , ” like logistic regression , produce non-linear uncertainty estimates in input space . This can be seen in Figure 2 . Furthermore , high-dimensional input spaces limit the actionability of these explanations , as∇xH will likely not point in the direction of the data manifold . In Figure 3 and Appendix D , we show how this can result in sensitivity analysis generating meaningless explanations . Our method , CLUE , leverages the latent space of a DGM to avoid working with high-dimensional input spaces and to ensure explanations are in-distribution . CLUE does not rely on crude linear approximations . The counterfactual nature of CLUE guarantees explanations have tangible meaning . 2.3 COUNTERFACTUAL EXPLANATIONS . The term “ counterfactual ” captures notions of what would have happened if something had been different . Two meanings have been used by ML subcommunities . 1 ) Those in causal inference make causal assumptions about interdependencies among variables and use these assumptions to incorporate consequential adjustments when particular variables are set to new values ( Kusner et al. , 2017 ; Pearl , 2019 ) . 2 ) In contrast , the interpretability community recently used “ counterfactual explanations ” to explore how input variables must be modified to change a model ’ s output without making explicit causal assumptions ( Wachter et al. , 2018 ) . As such , counterfactual explanations can be seen as a case of contrastive explanations ( Dhurandhar et al. , 2018 ; Byrne , 2019 ) . In this work , we use “ counterfactual ” in a sense similar to 2 ) : we seek to make small changes to an input in order to reduce the uncertainty assigned to it by our model , without explicit causal assumptions . Multiple counterfactual explanations can exist for any given input , as the functions we are interested in explaining are often non-injective ( Russell , 2019 ) . We are concerned with counterfactual input configurations that are close to the original input x0 according to some pairwise distance metric d ( · , · ) . Given a desired outcome c different from the original one y0 produced by predictor pI , counterfactual explanations xc are usually generated by solving an optimization problem that resembles : xc = arg maxx ( pI ( y=c|x ) − d ( x , x0 ) ) s.t . y0 6=c . ( 3 ) Naively optimizing ( 3 ) in high-dimensional input spaces may result in the creation of adversarial inputs which are not actionable ( Goodfellow et al. , 2015 ) . Telling a person that they would have been approved for a loan had their age been −10 is of very little use . To right this , recent works define linear constraints on explanations ( Ustun et al. , 2019 ; Sharma et al. , 2020 ) . An alternative more amenable to high dimensional data is to leverage DGMs ( which we dub auxiliary DGMs ) to ensure explanations are in-distribution ( Dhurandhar et al. , 2018 ; Joshi et al. , 2018 ; Chang et al. , 2019 ; Booth et al. , 2020 ; Tripp et al. , 2020 ) . CLUE avoids the above issues by searching for counterfactuals in the lower-dimensional latent space of an auxiliary DGM . This choice is well suited for uncertainty , as the DGM constrains CLUE ’ s search space to the data manifold . When faced with an OOD input , CLUE returns the nearest in-distribution analog , as shown in Figure 3 . 3 PROPOSED METHOD . Without loss of generality , we useH to refer to any differentiable estimate of uncertainty ( σ or H ) . We introduce an auxiliary latent variable DGM : pθ ( x ) = ∫ pθ ( x|z ) p ( z ) dz . In the rest of this paper , we will use the decoder from a variational autoencoder ( VAE ) . Its encoder is denoted as qφ ( z|x ) . We write these models ’ predictive means as Epθ ( x|z ) [ x ] =µθ ( x|z ) and Eqφ ( z|x ) [ z ] =µφ ( z|x ) respectively . CLUE aims to find points in latent space which generate inputs similar to an original observation x0 but are assigned low uncertainty . This is achieved by minimizing ( 4 ) . CLUEs are then decoded as ( 5 ) . L ( z ) = H ( y|µθ ( x|z ) ) + d ( µθ ( x|z ) , x0 ) , ( 4 ) xCLUE = µθ ( x|zCLUE ) where zCLUE = arg minzL ( z ) . ( 5 ) The pairwise distance metric takes the form d ( x , x0 ) =λxdx ( x , x0 ) + λydy ( f ( x ) , f ( x0 ) ) such that we can enforce similarity between uncertain points and CLUEs in both input and prediction space . The hyperparameters ( λx , λy ) control the trade-off between producing low uncertainty CLUEs and CLUEs which are close to the original inputs . In this work , we take dx ( x , x0 ) = ‖x − x0‖1 to encourage sparse explanations . For regression , dy ( f ( x ) , f ( x0 ) ) is mean squared error . For classification , we use cross-entropy . Note that the best choice for d ( · , · ) will be task-specific . The CLUE algorithm and a diagram of our procedure are provided in Algorithm 1 and Figure 4 , respectively . The hyperparameter λx is selected by cross validation for each dataset such that both terms in ( 4 ) are of similar magnitude . We set λy to 0 for our main experiments , but explore different values in Appendix H.1 . We minimize ( 4 ) with Adam by differentiating through both our BNN and VAE decoder . To facilitate optimization , the initial value of z is chosen to be z0=µφ ( z|x0 ) . Optimization runs for a minimum of three iterations and a maximum of 35 iterations , with a learning rate of 0.1 . If the decrease in L ( z ) is smaller than L ( z0 ) /100 for three consecutive iterations , we apply early stopping . CLUE can be applied to batches of inputs simultaneously , allowing us to leverage GPU-accelerated matrix computation . Our implementation is detailed in full in Appendix B. Algorithm 1 : CLUE Inputs : original datapoint x0 , distance function d ( · , · ) , Uncertainty estimatorH , DGM decoder µθ ( · ) , DGM encoder µφ ( · ) 1 Set initial value of z = µφ ( z|x0 ) ; 2 while loss L is not converged do 3 Decode : x = µθ ( x|z ) ; 4 Use predictor to obtainH ( y|x ) ; 5 L = H ( y|x ) + d ( x , x0 ) ; 6 Update z with∇z L ; 7 end 8 Decode explanation : xCLUE = µθ ( x|z ) ; Output : Uncertainty counterfactual xCLUE As noted by Wachter et al . ( 2018 ) , individual counterfactuals may not shed light on all important features . Fortunately , we can exploit the non-convexity of CLUE ’ s objective to address this . We initialize CLUE with z0 =µφ ( z|x0 ) + , where =N ( z ; 0 , σ0I ) , and perform Algorithm 1 multiple times to obtain different CLUEs . We find σ0 = 0.15 to give a good trade-off between optimization speed and CLUE diversity . Appendix C shows examples of different CLUEs obtained for the same inputs . We want to ensure noise from auxiliary DGM reconstruction does not affect CLUE visualization . For tabular data , we use the change in percentile of each input feature with respect to the training distribution as a measure of importance . We only highlight continuous variables for which CLUEs are separated by 15 percentile points or more from their original inputs . All changes to discrete variables are highlighted . For images , we report changes in pixel values by applying a sign-preserving quadratic function to the difference between CLUEs and original samples : ∆CLUE=|∆x|·∆x with ∆x=xCLUE−x0 . This is showcased in Figure 5 and in Appendix G. It is common for approaches to generating saliency maps to employ constraints that encourage the contiguity of highlighted pixels ( Chang et al. , 2019 ; Dabkowski & Gal , 2017 ) . We do not employ such constraints , but we note they might prove useful when applying CLUE to natural images .

Summary: The authors consider the problem of post-hoc explainability for decisions rendered by machine learning models.  They focus on addressing uncertain model predictions, producing counterfactual data that is both likely under a generative model of the data, as well as more certain in the classification task.  They present both experimental evidence, as well as a user study geared towards practitioners, that show the benefits of counterfactual explanations targeting uncertainty.