Getting a CLUE: A  Method for Explaining Uncertainty Estimates

1 INTRODUCTION . There is growing interest in probabilistic machine learning models , which aim to provide reliable estimates of uncertainty about their predictions ( MacKay , 1992 ) . These estimates are helpful in highstakes applications such as predicting loan defaults or recidivism , or in work towards autonomous vehicles . Well-calibrated uncertainty can be as important as making accurate predictions , leading to increased robustness of automated decision-making systems and helping prevent systems from behaving erratically for out-of-distribution ( OOD ) test points . In practice , predictive uncertainty conveys skepticism about a model ’ s output . However , its utility need not stop there : we posit predictive uncertainty could be rendered more useful and actionable if it were expressed in terms of model inputs , answering the question : “ Which input patterns lead my prediction to be uncertain ? ” Understanding which input features are responsible for predictive uncertainty can help practitioners learn in which regions the training data is sparse . For example , when training a loan default predictor , a data scientist ( i.e. , practitioner ) can identify sub-groups ( by age , gender , race , etc . ) under-represented in the training data . Collecting more data from these groups , and thus further constraining their model ’ s parameters , could lead to accurate predictions for a broader range of clients . In a clinical scenario , a doctor ( i.e. , domain expert ) can use an automated decision-making system to assess whether a patient should receive a treatment . In the case of high uncertainty , the system would suggest that the doctor should not rely on its output . If uncertainty were explained in terms of which features the model finds anomalous , the doctor could appropriately direct their attention . While explaining predictions from deep models has become a burgeoning field ( Montavon et al. , 2018 ; Bhatt et al. , 2020b ) , there has been relatively little research on explaining what leads to neural networks ’ predictive uncertainty . In this work , we introduce Counterfactual Latent Uncertainty Explanations ( CLUE ) , to our knowledge , the first approach to shed light on the subset of input space features that are responsible for uncertainty in probabilistic models . Specifically , we focus on explaining Bayesian Neural Networks ( BNNs ) . We refer to the explanations given by our method as CLUEs . CLUEs try to answer the question : “ What is the smallest change that could be made to an input , while keeping it in distribution , so that our model becomes certain in its decision for said input ? ” CLUEs can be generated for tabular and image data on both classification and regression tasks . An application of CLUE is to improve transparency in the real-world deployment of a probabilistic model , such as a BNN , by complementing existing approaches to model interpretability ( Ribeiro et al. , 2016 ; Sundararajan et al. , 2017 ; Chang et al. , 2019 ) . When the BNN is confident in its prediction , practitioners can generate an explanation via earlier feature importance techniques . When the BNN is uncertain , its prediction may well be wrong . This potentially wrong prediction could be the result of factors not related to the actual patterns present in the input data , e.g . parameter initialization , randomness in mini-batch construction , etc . An explanation of an uncertain prediction will be disproportionately affected by these factors . Indeed , recent work on feature attribution touches on the unreliability of saliency maps when test points are OOD ( Adebayo et al. , 2020 ) . Therefore , when the BNN is uncertain , it makes sense to provide an explanation of why the BNN is uncertain ( i.e. , CLUE ) instead of an explanation of the BNN ’ s prediction . This is illustrated in Figure 1 . Our code is at : github.com/cambridge-mlg/CLUE . We highlight the following contributions : • We introduce CLUE , an approach that finds counterfactual explanations of uncertainty in input space , by searching in the latent space of a deep generative model ( DGM ) . We put forth an algorithm for generating CLUEs and show how CLUEs are best displayed . • We propose a computationally grounded approach for evaluating counterfactual explanations of uncertainty . It leverages a separate conditional DGM as a synthetic data generator , allowing us to quantify how well explanations reflect the true generative process of the data . • We evaluate CLUE quantitatively through comparison to baseline approaches under the above framework and through ablative analysis . We also perform a user study , showing that CLUEs allow practitioners to predict on which new inputs a BNN will be uncertain . 2 PRELIMINARIES . 2.1 UNCERTAINTY IN BNNS . Given a dataset D= { x ( n ) , y ( n ) } Nn=1 , a prior on our model ’ s weights p ( w ) , and a likelihood function p ( D|w ) =∏Nn=1 p ( y ( n ) |x ( n ) , w ) , the posterior distribution over the predictor ’ s parameters p ( w|D ) ∝ p ( D|w ) p ( w ) encodes our uncertainty about what value w should take . Through marginalization , this parameter uncertainty is translated into predictive uncertainty , yielding reliable error bounds and preventing overfitting : p ( y∗|x∗ , D ) = ∫ p ( y∗|x∗ , w ) p ( w|D ) dw . ( 1 ) For BNNs , both the posterior over parameters and predictive distribution ( 1 ) are intractable . Fortunately , there is a rich literature concerning approximations to these objects ( MacKay , 1992 ; Hernández-Lobato & Adams , 2015 ; Gal , 2016 ) . In this work , we use scale-adapted Stochastic Gradient Hamiltonian Monte Carlo ( SG-HMC ) ( Springenberg et al. , 2016 ) . For regression , we use heteroscedastic Gaussian likelihood functions , quantifying uncertainty using their standard deviation , σ ( y|x ) . For classification , we take the entropy H ( y|x ) of categorical distributions as uncertainty . Details are given in Appendix B. Hereafter , we useH to refer to any uncertainty metric , be it σ or H . Predictive uncertainty can be separated into two components , as shown in Figure 2 . Each conveys different information to practitioners ( Depeweg , 2019 ) . Irreducible or aleatoric uncertainty is caused by inherent noise in the generative process of the data , usually manifesting as class overlap . Model or epistemic uncertainty represents our lack of knowledge about w. Stemming from a model being under-specified by the data , epistemic uncertainty arises when we query points off the training manifold . Capturing model uncertainty is the main advantage of BNNs over regular NNs . It enables the former to be used for uncertainty aware tasks , such as OOD detection ( Daxberger & HernándezLobato , 2019 ) , continual learning ( Nguyen et al. , 2018 ) , active learning ( Depeweg et al. , 2018 ) , and Bayesian optimization ( Springenberg et al. , 2016 ) . 2.2 UNCERTAINTY SENSITIVITY ANALYSIS . To the best of our knowledge , the only existing method for interpreting uncertainty estimates is Uncertainty Sensitivity Analysis ( Depeweg et al. , 2017 ) . This method quantifies the global importance of an input dimension to a chosen metric of uncertainty H using a sum of linear approximations centered at each test point : Ii = 1 |Dtest| |Dtest|∑ n=1 ∣∣∣∣∂H ( yn|xn ) ∂xn , i ∣∣∣∣ . ( 2 ) As discussed by Rudin ( 2019 ) , linear explanations of non-linear models , such as BNNs , can be misleading . Even generalized linear models , which are often considered to be “ inherently interpretable , ” like logistic regression , produce non-linear uncertainty estimates in input space . This can be seen in Figure 2 . Furthermore , high-dimensional input spaces limit the actionability of these explanations , as∇xH will likely not point in the direction of the data manifold . In Figure 3 and Appendix D , we show how this can result in sensitivity analysis generating meaningless explanations . Our method , CLUE , leverages the latent space of a DGM to avoid working with high-dimensional input spaces and to ensure explanations are in-distribution . CLUE does not rely on crude linear approximations . The counterfactual nature of CLUE guarantees explanations have tangible meaning . 2.3 COUNTERFACTUAL EXPLANATIONS . The term “ counterfactual ” captures notions of what would have happened if something had been different . Two meanings have been used by ML subcommunities . 1 ) Those in causal inference make causal assumptions about interdependencies among variables and use these assumptions to incorporate consequential adjustments when particular variables are set to new values ( Kusner et al. , 2017 ; Pearl , 2019 ) . 2 ) In contrast , the interpretability community recently used “ counterfactual explanations ” to explore how input variables must be modified to change a model ’ s output without making explicit causal assumptions ( Wachter et al. , 2018 ) . As such , counterfactual explanations can be seen as a case of contrastive explanations ( Dhurandhar et al. , 2018 ; Byrne , 2019 ) . In this work , we use “ counterfactual ” in a sense similar to 2 ) : we seek to make small changes to an input in order to reduce the uncertainty assigned to it by our model , without explicit causal assumptions . Multiple counterfactual explanations can exist for any given input , as the functions we are interested in explaining are often non-injective ( Russell , 2019 ) . We are concerned with counterfactual input configurations that are close to the original input x0 according to some pairwise distance metric d ( · , · ) . Given a desired outcome c different from the original one y0 produced by predictor pI , counterfactual explanations xc are usually generated by solving an optimization problem that resembles : xc = arg maxx ( pI ( y=c|x ) − d ( x , x0 ) ) s.t . y0 6=c . ( 3 ) Naively optimizing ( 3 ) in high-dimensional input spaces may result in the creation of adversarial inputs which are not actionable ( Goodfellow et al. , 2015 ) . Telling a person that they would have been approved for a loan had their age been −10 is of very little use . To right this , recent works define linear constraints on explanations ( Ustun et al. , 2019 ; Sharma et al. , 2020 ) . An alternative more amenable to high dimensional data is to leverage DGMs ( which we dub auxiliary DGMs ) to ensure explanations are in-distribution ( Dhurandhar et al. , 2018 ; Joshi et al. , 2018 ; Chang et al. , 2019 ; Booth et al. , 2020 ; Tripp et al. , 2020 ) . CLUE avoids the above issues by searching for counterfactuals in the lower-dimensional latent space of an auxiliary DGM . This choice is well suited for uncertainty , as the DGM constrains CLUE ’ s search space to the data manifold . When faced with an OOD input , CLUE returns the nearest in-distribution analog , as shown in Figure 3 . 3 PROPOSED METHOD . Without loss of generality , we useH to refer to any differentiable estimate of uncertainty ( σ or H ) . We introduce an auxiliary latent variable DGM : pθ ( x ) = ∫ pθ ( x|z ) p ( z ) dz . In the rest of this paper , we will use the decoder from a variational autoencoder ( VAE ) . Its encoder is denoted as qφ ( z|x ) . We write these models ’ predictive means as Epθ ( x|z ) [ x ] =µθ ( x|z ) and Eqφ ( z|x ) [ z ] =µφ ( z|x ) respectively . CLUE aims to find points in latent space which generate inputs similar to an original observation x0 but are assigned low uncertainty . This is achieved by minimizing ( 4 ) . CLUEs are then decoded as ( 5 ) . L ( z ) = H ( y|µθ ( x|z ) ) + d ( µθ ( x|z ) , x0 ) , ( 4 ) xCLUE = µθ ( x|zCLUE ) where zCLUE = arg minzL ( z ) . ( 5 ) The pairwise distance metric takes the form d ( x , x0 ) =λxdx ( x , x0 ) + λydy ( f ( x ) , f ( x0 ) ) such that we can enforce similarity between uncertain points and CLUEs in both input and prediction space . The hyperparameters ( λx , λy ) control the trade-off between producing low uncertainty CLUEs and CLUEs which are close to the original inputs . In this work , we take dx ( x , x0 ) = ‖x − x0‖1 to encourage sparse explanations . For regression , dy ( f ( x ) , f ( x0 ) ) is mean squared error . For classification , we use cross-entropy . Note that the best choice for d ( · , · ) will be task-specific . The CLUE algorithm and a diagram of our procedure are provided in Algorithm 1 and Figure 4 , respectively . The hyperparameter λx is selected by cross validation for each dataset such that both terms in ( 4 ) are of similar magnitude . We set λy to 0 for our main experiments , but explore different values in Appendix H.1 . We minimize ( 4 ) with Adam by differentiating through both our BNN and VAE decoder . To facilitate optimization , the initial value of z is chosen to be z0=µφ ( z|x0 ) . Optimization runs for a minimum of three iterations and a maximum of 35 iterations , with a learning rate of 0.1 . If the decrease in L ( z ) is smaller than L ( z0 ) /100 for three consecutive iterations , we apply early stopping . CLUE can be applied to batches of inputs simultaneously , allowing us to leverage GPU-accelerated matrix computation . Our implementation is detailed in full in Appendix B. Algorithm 1 : CLUE Inputs : original datapoint x0 , distance function d ( · , · ) , Uncertainty estimatorH , DGM decoder µθ ( · ) , DGM encoder µφ ( · ) 1 Set initial value of z = µφ ( z|x0 ) ; 2 while loss L is not converged do 3 Decode : x = µθ ( x|z ) ; 4 Use predictor to obtainH ( y|x ) ; 5 L = H ( y|x ) + d ( x , x0 ) ; 6 Update z with∇z L ; 7 end 8 Decode explanation : xCLUE = µθ ( x|z ) ; Output : Uncertainty counterfactual xCLUE As noted by Wachter et al . ( 2018 ) , individual counterfactuals may not shed light on all important features . Fortunately , we can exploit the non-convexity of CLUE ’ s objective to address this . We initialize CLUE with z0 =µφ ( z|x0 ) + , where =N ( z ; 0 , σ0I ) , and perform Algorithm 1 multiple times to obtain different CLUEs . We find σ0 = 0.15 to give a good trade-off between optimization speed and CLUE diversity . Appendix C shows examples of different CLUEs obtained for the same inputs . We want to ensure noise from auxiliary DGM reconstruction does not affect CLUE visualization . For tabular data , we use the change in percentile of each input feature with respect to the training distribution as a measure of importance . We only highlight continuous variables for which CLUEs are separated by 15 percentile points or more from their original inputs . All changes to discrete variables are highlighted . For images , we report changes in pixel values by applying a sign-preserving quadratic function to the difference between CLUEs and original samples : ∆CLUE=|∆x|·∆x with ∆x=xCLUE−x0 . This is showcased in Figure 5 and in Appendix G. It is common for approaches to generating saliency maps to employ constraints that encourage the contiguity of highlighted pixels ( Chang et al. , 2019 ; Dabkowski & Gal , 2017 ) . We do not employ such constraints , but we note they might prove useful when applying CLUE to natural images .

Summary: This paper introduces CLUE -- a method to explain uncertainty estimates.  The method utilizes a VAE trained on the original data set to search effectively for low confidence instances.  The method utilizes a gradient based search through the latent space of the VAE.  The authors assess their approach on a variety of tabular data sets and MNIST.  They evaluate along change in uncertainty of the counterfactual as well as human evaluation.  They generally find improvements using their method over baselines.  Additionally, their method works much better in human evaluations.