Getting a CLUE: A  Method for Explaining Uncertainty Estimates
Authors: Javier Antoran
Umang Bhatt
Tameem Adel
Adrian Weller
José Miguel Hernández-Lobato
Venue: ICLR 2021 Oral (2021)
Keywords: interpretability
uncertainty
explainability

Abstract: Both uncertainty estimation and interpretability are important factors for trustworthy machine learning systems. However, there is little work at the intersection of these two areas. We address this gap by proposing a novel method for interpreting uncertainty estimates from differentiable probabilistic models, like Bayesian Neural Networks (BNNs). Our method, Counterfactual Latent Uncertainty Explanations (CLUE), indicates how to change an input, while keeping it on the data manifold, such that a BNN becomes more confident about the input's prediction. We validate CLUE through 1) a novel framework for evaluating counterfactual explanations of uncertainty, 2) a series of ablation experiments, and 3) a user study. Our experiments show that CLUE outperforms baselines and enables practitioners to better understand which input patterns are responsible for predictive uncertainty.

Published as a conference paper at ICLR 2021
GETTING A CLUE: A METHOD FOR EXPLAINING UNCERTAINTY ESTIMATES
Javier Antorán
Umang Bhatt
Tameem Adel
University of Cambridge ja 6660 ac..uk
University of Cambridge usb20 cam. cam.ac.c....oooccccmmmmmmmm ac.uk
University of Cambridge
University of Liverpool tah4 7 cam. ac uk
Adrian Weller University of Cambridge The Alan Turing Institute 6w6650 cam ac. uk
José Miguel Hernández-Lobato University of Cambridge The Alan Turing Institute
jmh233 cam ac uk
ABSTRACT
Both uncertainty estimation and interpretability are important factors for trustwor- thy machine learning systems. However, there is little work at the intersection these two areas. We address this gap by proposing a novel method for interpret- ing uncertainty estimates from differentiable probabilistic models, like Bayesian Neural Networks (BNNs). Our method, Counterfactual Latent Uncertainty Expla- nations (CLUE), indicates how to change an input, while keeping on the data manifold, such that BNN becomes more confident about the input's prediction. We validate CLUE through 1) a novel framework for evaluating counterfactual explanations of uncertainty, 2) a series of ablation experiments, and user study. Our experiments show that CLUE outperforms baselines and enables practitioners to better understand which input patterns are responsible for predictive uncertainty.
INTRODUCTION
There is growing interest in probabilistic machine learning models, which aim to provide reliable estimates of uncertainty about their predictions (MacKay) 1992). These estimates are helpful in high- stakes applications such as predicting loan defaults or recidivism, or in work towards autonomous vehicles. Well-calibrated uncertainty can be as important as making accurate predictions, leading to increased robustness of automated decision-making systems and helping prevent systems from behaving erratically for out-of out-off-dstrrbbl distrribbution (OOD) test points. In practice, predictive uncertainty conveys skepticism about model's output. However, its utility need not stop there: we posit predictive uncertainty could be rendered more useful and actionable if it were expressed in terms of model inputs, answering the question: "Which inpul patterns lead my prediction to be uncertain?"
Understanding which input features are responsible for predictive uncertainty can help practitioners learn in which regions the training data sparse. For example, when training loan default predictor, data scientist (i.e., practitioner) can identify sub-groups (by age, gender, race, etc.) under-represented in the training data. Collecting more data from these groups, and thus further constraining their model's 'ss parameters, could lead to accurate predictions for a broader range of clients. In a clinical scenario, a doctor (i.e.. domain expert) can use an automated whether a patient should receive treatment. In the case of high uncertainty, the system would suggest that the doctor should not rely on output. If uncertainty were explained in terms of which features the model finds anomalous, the doctor could appropriately direct their attention.
docision-maaiinn system to assess
While explaining predictions from deep models has become a burgeoning field (Montavon et al. 2018 Bhatt et al. 2020b), there has been relatively little research on explaining what leads to neural networks' predictive uncertainty. In this work, we introduce Counterfactual Latent Uncertainty Explanations (CLUE), to knowledge, the first approach shed light the subset of input space features that are responsible for uncertainty in probabilistic models. Specifically, we focus on explaining Bayesian Neural Networks (BNNs). We refer to the explanations given by method as

---

Published as a conference paper at ICLR 2021
Probabilistic Model
Uncertainty Quantification
Explanation
Interpretability Mothod
LIME (hibero
Integrated Grad (Sue- dararaua et. 20121
C Input x
Yes
FIDO [Chong et al., 2019]
Certain
Prediction?
6
No
CLUE
A
Figure 1: Workflow for automated decision making with transparency. Our probabilistic classifier produces distribution over outputs. In cases of high uncertainty, CLUE allows to identify features which are responsible for class ambiguity in the input (denoted by A and highlighted in dark blue).
Otherwise, resort to existing feature importance approaches to explain certain decisions.
CLUEs. CLUEs try to answer the question: "What is the smallest change that could be made to input, while keeping it in distribution, so that our model becomes certain its decision for said imput? CLUES can be generated for tabular and image data both classification and regression tasks.
An application of CLUE is to improve transparency in the real-world deployment of a probabilistic model, such BNN, by complementing existing approaches model interpretability (Ribeiro et al. 2016 Sundararajan et al. 2017: Chang et al. 2019). When the BNN is confident in its prediction, practitioners can generate an explanation
earlier feature importance techniques. When the BNN is uncertain, its prediction may well be wrong. This potentially wrong prediction could be the result of factors not related to the actual patterns present in the input data, e.g. parameter initialization, randomness in mini-batch construction, etc. An explanation of uncertain prediction will disproportionately affected by these factors. Indeed, recent work on feature attribution touches on the unreliability of saliency maps when test points are OOD (Adebayo et al. 2020). Therefore, when the BNN is uncertain, it makes sense to provide an explanation of why the BNN is uncertain (i.c., CLUB) instead of explanation of the BNN's prediction. This is illustrated in Figure Our code is at: g github. com wwm.criiiiooo sembbridge-miee CLUE We highlight the following contributions:
We introduce CLUE, an approach that finds counterfactual explanations of uncertainty in input space, by searching in the latent space of deep generative model (DGM). We put forth an algorithm for generating CLUES and show how CLUES are best displayed.
We propose computationally grounded approach for evaluating counterfactual explanations of uncertainty. It leverages a separate conditional DGM as a synthetic data generator, allowing us to quantify how well explanations reflect the true generative process of the data,
We evaluate CLUE quantitatively through comparison to baseline approaches under the above framework and through ablative analysis. We also perform a user study, showing that CLUES allow practitioners predict on which new inputs BNN will be uncertain.
PRELIMINARIES
2.1 UNCERTAINTY BNNs
Given a dataset D = (x(), (x)) N Jn=1. a prior on our model's weights p(w), and a likelihood func- tion p(D w) 11 ((y(")/>ef w). the posterior distribution over the predictor's parameters p(w(D) & pDD (D/)[)]j)))) hwwpppp (p(w)) w) encodes our uncertainty about what value w should take. Through marginal- ization, this parameter uncertainty is translated into predictive uncertainty, yielding reliable error bounds and preventing overfitting:
ply'Ix- / /
p(y"|x", w)p(w/D)) dw.
For BNNs, both the posterior over parameters and predictive distribution are intractable. For- tunately, there is a rich literature concerning approximations these objects (MacKay, 1992] Hernández Lobato Adams 2015; Gal 2016). In this work, we use scale-adapted Stochastic Gradient Hamiltonian Monte Carlo (SG-HMC (Springenherg et al. 2016). For regression,
2

---

Published as a conference paper at ICLR 2021
10
10
0.7
5
(siev)
-5
-10
0.0
10
10 100
10
Figure 2: Left: Training points and predictive distribution for variational Bayesian Logistic Regression on the Moons dataset. Center: Aleatoric entropy H. matches regions of class non-separability. Right: Epistemic entropy He grows away from the data. Both uncertainties are detailed in Appendix B.2
heteroscedastic Gaussian likelihood functions, quantifying uncertainty using their standard deviation, (y]x). For classification, we take the entropy H (yy|x) (x) of categorical distributions as uncertainty. Details are given in Appendix B Hereafter, we use H. refer any uncertainty metric, be a or H.
Predictive uncertainty can be separated into two components, as shown in Figure 2 Each conveys different information to practitioners (Depeweg 2019). Irreducible aleatoric uncertainty is caused by inherent noise in the generative process of the data, usually manifesting as class overlap. Model or epistemic uncertainty represents our lack of knowledge about w. Stemming from model being under-specified by the data, epistemic uncertainty arises when we query points off the training manifold. Capturing model uncertainty the main advantage of BNNs over regular NNs. It enables the former to be used for uncertainty aware tasks, such as OOD detection (Daxberger & Hernández. Lobato 2019), continual learning (Nguyen et al. 2018), active learning (Depeweg et al. 2018), and Bayesian optimization (Springenberg al. 2016).
2.2 UNCERTAINTY SENSITIVITY ANALYSIS
To the best of our knowledge, the only existing method for interpreting uncertainty estimates is Uncertainty Sensitivity Analysis (Depeweg al. 2017). This method quantifies the global importance of an input dimension to chosen metric of uncertainty H using sum of linear approximations centered at each test point:
Dea
Σ
cest
izn,t
As discussed by Rudin (2019), linear explanations of non-linear models, such as BNNs, can be mis- leading. Even generalized linear models, which are often considered to be "inherently interpretable." like logistic regression, produce non-linear uncertainty estimates in input space. This be seen in Figure 2] Furthermore, nigh-dimensional input spaces limit the actionability of these explanations, and Appendix [
V H will likely not point in the direction of the data manifold. In Figure X show how this can result in sensitivity analysis generating meaningless explanations.
Our method, CLUE, leverages the latent space of DGM avoid working with high-dimensional input spaces and to ensure explanations are in-distribution. CLUE does not rely on crude linear approximations. The counterfactual nature of CLUE guarantees explanations have tangible meaning.
Sensitivity
CLUE
z
1,(0(x,)
2g
H(y 1x.) = 1.77
(i,]) ) 0.12
Hly (xg)
-Pp (CC(
Hly Necug) = 0.19
Figure 3: Left: Taking a step in the direction of maximum sensitivity leads to a seemingly noisy input configuration for which H is small. Right: Minimizing CLUB's uncertainty-basee objective in terms of a DGM's latent variable z produces a plausible digit with a corrected lower portion.
3

---

Published as a conference paper at ICLR 2021
2.3 COUNTERFACTUAL EXPLANATIONS
The term "counterfactual" captures notions of what would have happened if something had been different. Two meanings have been used by ML subcommunities. 1) Those in causal inference make causal assumptions about interdependencies among variables and use these assumptions to incorporate consequential adjustments when particular variables are set to new values (Kusner et al.) 2017, Pearl 2019). 2) In contrast, the interpretability community recently used "counterfactual explanations" to explore how input variables must be modified to change a model's ss output withour making explicit causal assumptions (Wachter et al.] 2018). As such, counterfactual explanations can be seen as a case of contrastive explanations (Dhurandhar et al., 2015 8 Byme 2019). In this work, we use "counterfactual" in a sense similar to 2): we seek to make small changes to an input in order to reduce the uncertainty assigned to it by our model, without explicit causal assumptions.
Multiple counterfactual explanations can exist for any given input, as the functions are interested in explaining are often non-ing non-injective (Russell 2019). We are concerned with counterfactual input configurations that are close the original input Xo according some pairwise distance metric d(,; .)- Given desired outcome e different from the original one Yu produced by predictor pr, counterfactual explanations x, are usually generated by solving an optimization problem that resembles:
x arg max (pı(y=c(x)- xo)) s.t. yozc.
Naively optimizing in high- sigh-dimensional input spaces may result in the creation of adversarial inputs which are not actionable (Goodfellow et al. 2015). Telling a person that they would have been approved for a loan had their age been - 10 is of very little use. To right this, recent works define linear constraints on explanations (Ustun et al., amenable to high dimensional data to leverage DGMs (which
[Sharma et al., 2020). An alternative more dub auxiliary DGMs) to ensure explanations are in-distribution (Dhurandhar et al. 2018 Joshi et al.) 2018: Chang et al. 2019 Booth et al., 2020 Tripp et al., 2020). CLUE avoids the above issues by searching for counterfactuals in the diwer-dimensional latent space of an auxiliary DGM. This choice is well suited for uncertainty, as the DGM constrains CLUE's search space the data manifold. When faced with an OOD input, CLUE
2019
returns the nearest in-distribution analog, as shown in Figure 3
3 PROPOSED METHOD
Without loss of generality, we use H. to refer to any differentiable estimate of uncertainty (a or H).
We introduce an auxiliary latent variable DGM: Px)) = Po (x(x)z)p(z)) dz. In the rest of this paper, we will use the decoder from a variational autoencoder (VAF). Its encoder is denoted as (x(x))) We
write these models' predictive means as (xpo ( and (x, [1]=
respectively.
CLUE aims to find points in latent space which generate inputs similar to an original observation X but are assigned low uncertainty. This is achieved by minimizing (4). CLUES are then decoded as (5)
((z) H(yluo +
(4)
XCLUE = no (x where ZCLUE = arg C(n,
(5)
The pairwise distance metric takes the form d(x, xn) = dz (x, xo) + (x)) (xx)) such that we can enforce similarity between uncertain points and CLUES in both input and prediction space. The hyperparameters (x) control the trade-off between producing low uncertainty CLUES and CLUES which are close to the original inputs. In this work, we take (x, xu) = =||x xolla to encourage sparse explanations. For regression, (fxx), f(xo)) )) is mean squared error. For classification, we use cross-entropy. Note that the best choice for d(-,-) will be task-specific.
The CLUE algorithm and a diagram of our procedure are provided in Algorithm I and Figure respectively. The hyperparameter X₂ is selected by cross validation for each dataset such that both terms in (4) are of similar magnitude. We set Xy to 0 for our main experiments, but explore different values in Appendix H.I We minimize (4) with Adam by differentiating through both our BNN and VAE decoder. To facilitate optimization, the initial value of is chosen to be Z0= Xo). Optimization runs for minimum of three iterations and a maximum of 35 iterations, with learning rate of 0.1. If the decrease in L(z) is smaller than ((xo)/100 for three consecutive iterations, we apply early stopping. CLUE can be applied to batches of inputs simultancously, allowing us to leverage GPU-accelerated matrix computation. Our implementation is detailed in full in Appendix B
4

---

Published as a conference paper at ICLR 2021
Latent space
Algorithm 1: CLUE
z!r x"1
Inputs:
original datapoint Xo. distance
function d(;;,) :). Uncertainty
Decoder pylxte"
1. Gemerata Counterfactual
estimator H. DGM decoder pe(-),
3. Calculate V,A(H)
DGM encoder (.(())
Set initial value of 2 = Ho Xo):
while loss not converged do
High-dimecsioonll Input space
Decode: x (x)xx
Use predictor to obtain H(y x)
L= H( + xo): Update z with V L;
BNN plyie" "y
Prediction
end
Decode explanation: XCLUE Me (x z):
Figure 4: Latent codes are decoded into inputs for which BNN generates uncertainty estimates; their gradients are backpropagated to latent space.
Output: Uncertainty counterfactual XCLUE
As noted by Wachter et al., (2018), individual counterfactuals may not shed light on all important fea- tures. Fortunately, we can exploit the non non-convexity of CLUE's objective to address this. We initialize CLUE with zu = Ho xo) + e, where c (2; 0, IJI, and perform Algorithm] I multiple times to obtain different CLUEs. We find de = 0.15 to give good trade-off between optimization speed and CLUE diversity. Appendix [C examples of different CLUEs obtained for the same inputs.
We want to ensure noise from auxiliary DGM re- construction does not affect CL.UE visualization. For tabular data, we use the change in percentile of each input feature with respect to the training distribution as measure of importance. We only highlight continuous variables for which CLUES are separated by 15 percentile points more from their original inputs. All changes to discrete variables are highlighted. For images, we report changes in pixel values by applying Sugn-preserving quadratic function to the dif- ference between CLUEs and original samples:
Oriirol
Original
CLUE
LSAT
41.0
7
7
UGPA
3.7
3.0
Race
Asian
Race
White
X
Q
Sex
Famale
Sex
Female
(a) MNIST
(b) LSAT
Figure 5: Example image and tabular CLUEs.
ACLUE= Ax| Ax with Ax=XcLUE -xp. This showcased in Figure and in Appendix G is common for approaches to generating saliency maps to employ constraints that encourage the contiguity of highlighted pixels (Chang et al. 2019 Dabkowski & Gal 2017). We do not employ such constraints, but note they might prove useful when applying CLUE to natural images.
4 A FRAMEWORK FOR EVALUATING COUNTERFACTUAL EXPLANATIONS OF UNCERTAINTY COMPUTATIONALLY
Evaluating explanations quantitatively (without resorting to expensive user studies) is a difficult but important task (Doshi-Velez Velez & Kim 2017 Weller 2019). We put forth computational framework to evaluate counterfactual explanations of uncertainty. In the spirit of [Bhatt et al. (2020a). we desire counterfactuals that are 1) informative: they should highlight features which affect our BNN's uncertainty, and 2) relevant: counterfactuals should lie close to the original inputs and represent plausible parameter settings, lying close the data manifold. Recall, from Figure 3 that inputs for which our BNN is certain can be constructed by applying adversarial perturbations to uncertain ones. Alas, evaluating these criteria requires access the generative process of the data.
To evaluate the above requirements, we introduce an additional DGM that will act as "ground truth" data generating process (g.t. DGM). Specifically, we use a variational autoencoder with arbitrary conditioning (Ivanov et al. 2019) (g.t. VAEAC). It jointly models inputs and targets Pat (x,(x, y). Measuring counterfactuals log density under this model log (x.) allows us to evaluate they are in in-distribution. The g.t. VAEAC also allows us query the conditional distribution over targets given inputs, (x) From this distribution, we can compute an input's 's true uncertainty Hgr. as given by the generative process of the data. This allows evaluate if counterfactuals address the true
5

---

Published as a conference paper at ICLR 2021
g.t. VAEAC on real data
VAE on artificial data
Generate counterfactuals for artificial data
Evaluate counterfactuals with g.t. VAEAC
err g 4
log log (8)
Figure 6: Pipeline for computational evaluation of counterfactual explanations of uncertainty, The VAEAC which we treat data generating process is colored in green. Colored in orange is the auxiliary DGM used by the approach being evaluated. For approaches that do not use an auxiliary DGM, like Uncertainty Sensitivity Analysis, the orange element will not be present.
sources of uncertainty in the data, as opposed to exploiting adversarial vulnerabilities in the BNN. The evaluation procedure, shown in Figure 6 is as follows:
1. Train a g.t. VAEAC on a real dataset to obtain (x. y) as well as conditionals (y)(y) x).
2. Sample artificial data (x, y) ~ (x, y). Use them to train a BNN and an auxiliary DGM.
3. Sample more artificial data. Generate counterfactual explanations Xg for uncertain samples.
4. Use the g.t. VAEAC obtain the conditional distribution targets given counterfactual inputs Pe (y) xx) and Hgt. Evaluate counterfactuals are on-manifold through log Pa (x.).
Given an uncertain artificially generated test point Xg ~ Pat and corresponding counterfactual explanation X., quantify informativeness the amount of uncertainty that has been explained away. The variance (or entropy) of (yr(y|x) reflects the ground truth alcatoric uncertainty associated with x. Hence, for aleatoric uncertainty, we quantify informativeness A pp [?g. (y (y|xo) Hyr (8.)]. Epistemic uncertainty only depends our BNN. It cannot be directly computed from Pet (y(x). However, its reduction can be measured implicitly through the reduction in the BNN's prediction eπror with respect to the labels outputted by the g.t. VAEAC: Aerrge gt Ep,, [erT err (x.)). Here erry
dyy (p(y) x)) arg max,, Pg (g((y/x)) (y/x)). Approaches that exploit adversarial weaknesses in the BNN will not transfer to the g.t. VAEAC, failing to reduce uncertainty error. We assess the relevance of counterfactuals through their likelihood under the g.t. VAEAC log (g. and through their & distance to the original inputs | = || xell 1. We refer Appendix for a detailed discussion on g.t. VAEACs and their use for comparing counterfactual generations.
5 EXPERIMENTS
We validate CLUE on LSAT academic performance regression (Wightman et al. 1998). UCI Wine quality regression, UCI Credit classification (Dua Graff] (017). a feature variant of COMPAS recidivism classification (Angwin et al.).. and MNIST image classification (LeCun & Cortes 2010). For each, we select roughly the 20% most uncertain test points as those for which we reject our BNNs' decisions. We only generate CLUEs for "rejected" points. Rejection thresholds, architectures, and hyperparameters are in Appendix Experiments with non-Bayesian NNs are in Appendix H.I
As baseline, we introduce localized version of Uncertainty Sensitivity Analysis (Depeweg et al.) 2017p. It produces counterfactuals by taking single step, of size 7,, in the direction of the gradient of input's 's uncertainty estimates x. Xx H Xo). Averaging [X0- xel across test set, we recover (2). As a second baseline, we adapt FIDO (Chang et al. 2019). 1 a counterfactual feature importance method, minimize uncertainty. We dub this approach U-FIDO. This method places binary mask over the set of input variables xy. The mask is modeled by a product of Bernoulli random variables: (b) II (bun Pw). The of masked inputs xB is substituted by its expectation under auxiliary conditional generative model p(xg x)),)). We use VAEAC. U-FIDO finds the masking parameters which minimize (6)
L(p) = P. (b)))x((b)) + >bo[b|[1),
(6)
(bb ④ Xo + (1 b) @ Ex(xx [xB]
(7)
Counterfactuals are generated by 777. where ② is the Hadamard product. We compare CLUE to feature importance methods (Ribeiro et al. 2016 Lundberg & Lee 2017) in Appendix

---

Published as a conference paper at ICLR 2021
5.1 COMPUTATIONAL EVALUATION
We compare CLUE, Localized Sensitivity, and U-FIDO using the evaluation framework put forth in Section 4 We would like counterfactuals explain away much uncertainty possible while staying as close to the original inputs as possible. We manage this informativeness (large AHg) to relevance (small || | trade-off with the hyperparameters 77, Ag. and A for Local Sensitivity. CLUE, and U-FIDO, respectively. We perform logarithmic grid search over hyperparameters and plot Pareto-like curves. Our two metrics of interest take minimum values of but their maximum is dataset and method dependent. For Sensitivity, || | grows linearly with v). For CLUE and U-FIDO, these metrics saturate for large and small values of Aa (or Aτ). As a result, the values obtained by these methods do not overlap. As shown in Figure 7 uncertainty (AHg) than U-FIDO, and U-FIDO always obtains smaller values of ||Ax|1 than CLUE.
CLUE is able to explain away more
Table 1:
Agg vs || measure obtained by all methods all datasets under consideration. Lower better. The dimensionality of
.0-
each dataset is listed next to their names. and a indicate results for epistemic errgy) and aleatoric (AHg) uncertainty respectively.
Sans.
CLUC
Kni pl
Method
LSAT (4)
COMPAS (7)
Wine (11)
Credit (23)
MNIST (784)
a
a
Sensitivity
0.70
0.67
0.71
0.13
0.69
0.03
0.63
0.50
0.66
0.68
CLUE
0.52
0.64
0.71
0.18
0.01
0.14
0.52
0.29
0.26
0.27
||
U-FIDO
0.36
0.51
0.71
0,31
0.22
0.02
0.45 0.63
0.38
0.50
Figure 7: MNIST knee-points.
To construct single performance metric, we scale all measurements by the maximum values obtained between U-FIDO CLUE, e.g. (v2. max(AHg F.roO, Agg CLUE)) linearly mapping them to [0, 1/v].. We then negate Agg. making its optimum value O. We consider each method's best performing hyperparameter configuration, as determined by its curve's 's point nearest the origin, or knee-point. The euclidean distance from each method's 's knee-point to the origin acts as a metric of relative performance. The best value is 0 and the worst is 1. Knee-point distances, computed across three runs, are shown for both uncertainty types in Table !
Local Sensitivity performs poorly on all datasets except COMPAS. We attribute this to the implicit low dimensionality of COMPAS: only two features are necessary to accurately predict targets (Dressel & Farid, 2018). U-FIDO's s masking mechanism allows for counterfactuals that leave features unchanged. It performs well in low dimensional problems but suffers from variance as dimensionality grows. We conjecture that optimization in latent (instead of input) space makes CLUE robust to data complexity.
We perform analogous experiment where relevance is quantified as proximity to the data manifold: Pgt = min (0, Pg - P (xn)). Here, Po (Xo) refers to the log- log-likelihood of the artificial data for which counterfactuals are generated. Results are in Table 2 where CLUE performs best in 8/10 tasks. Generating counterfactuals with a VAE ensures that CLUEs are relevant. In Appendix we perform analogous informativeness relevance experiment on real data. We obtain similar lar results to our computational evaluation framework, validating its reliability.
Table 2: A Pg vs |
|| measure obtained by all methods on all datasets under consideration. Lower is better. e and a indicate epistemic (Aerrg) and aleatoric (AH.)) uncertainty respectively.
Method
LSAT (4)
COMPAS (7)
Wine (11)
Credit (23)
MNIST (784)
a
e
Sensitivity
0.697
0.672
0.707
0.122
0.691
0.001
0.623
0.454
0.698
CLUE
0.419
0.070
0.707
0.044
0
0.128
0
0.009
0,273
0.146
U-FIDO
0
0.707
0.303
0.224
0.233
0.628
0,450
0.516
5.2
USER STUDY
Human- based evaluation is a key step in validating the utility of tools for ML explainability (Hoffman et al. 2018). We want to assess the extent to which CLUEs help machine learning practitioners identify sources of uncertainty in ML models compared to using simple linear approximations (Local Sensitivity) or human intuition. To do this, we propose a
task (Doshi- Velez & Kim 2017). focusing appropriate local test evaluate CLUES. We show practitioners one
7

---

Published as a conference paper at ICLR 2021
Certain Pilot Procedure
Main Survey.
Certain
Point
Test Polut
Lincertain
1) Participaut
LAST
410
4) Participants identify
USP
Entire Test Set
the certuinty of the test
puint rundom from test
Vade
point given the two
points
3) certinin
Uncertain Context Point
contert point based on being
LSMT
41.0
Tea Set wie certain points
2) Participant A pairs the selected point with unceriain point
UGRA
CLUE
Semsiiivity
Rondom
Sec
Faria
Human CLUE
Figure 8: Experimental workflow for our tabular data user study.
Certain
Age
Less than 25
Age
Less than
Age
Less than 25
Race
Caucasian
Atrican-American
Race
Hispenic
Sex
Male
Sex
Male
Sex
Male
Current Charge
Misdemeanour
Current Charge
Miademeenour
Current Charge
Misdemnanour
Reoffended Before
Yes
Reoffended Before
Reoffended Before
No
Prior Convictions
Prior Convictions
Prior Convictions
0
Days Served
Days
Days Served
Figure 9: Example question shown main survey participants for the COMPAS dataset: Given the uncertain example on the left and the certain example in the middle, will the model be certain on the test example on the right? The red text highlights the features that differ between context points.
datapoint below our "rejection" threshold and one datapoint above. The former is labeled as "certain" and the latter as "uncertain"; we refer to these context points. The certain context point serves as a local counterfactual explanation for the uncertain context point. Using both context points for reference, practitioners threshold (i.e., will our BNN's uncertainty be high or low for the new point). Our survey compares the utility of the certain context points generated by CLUE relative those from baselines.
asked predict whether
test point will be above below
In our survey, we compare four different methods, varying how we select certain context points. We either 1) select certain point at random from the test set as control, generate a counterfactual certain point with 2) Local Sensitivity or with 3) CLUE, or 4) display human selected certain point (Human CLUE). To generate Human CLUE, we ask participants (who will not take the main survey) to pair uncertain context points with similar certain points. We select the points used in our main survey with pilot procedure similar to Grgic-Hlaea prevents us from injecting biases into point selection and ensures context points are relevant to test points. In our procedure, participant shown a pool of randomly selected certain and uncertain points. We ask this participant select points from this pool: these will be test points. We then ask the participant to map each selected test point to a similar uncertain point without replacement. In this way, obtain uncertain context points that are relevant to test points.
al. (2018). This procedure, shown in Figure 8
We use the LSAT and COMPAS datasets in our user study. Ten different participants take each variant of the main survey: participants ML graduate students, who in industry. The main survey consists of 18 questions, 9 per dataset. An example question is shown in Figure The average participant accuracy by variant is: CLUE (82.22) (62.22%) Human CLUE (62.222 (22.22%) Random (61.67%), and Local Sensitivity (52.78%). We measure the statistical significance of CLUE's superiority with unpaired Wilcoxon signed-rank tests (Demšar) 2006) of CLUE each baseline. We obtain the following p-values: Human-CLUE (2.34e-5) 55). Random (1.47e-5), and
proxies for practitioners
Sensitivity (2.60e-9) Additional analysis is included in Appendix H.3
'Our between- geoveen-groupppooo experimental design could potentially result in dependent observations, which violates the tests' assumptions. However, the p-values obtained are extremely low, providing confidence in rejecting the null hypothesis of equal performance among approaches.
8

---

Published as a conference paper at ICLR 2021
2g P/ (*) xg)
20
130
1.80
80
0.8
70
1.38
Myy
100
0.6
Mly M(XXXCUU
|Ax
H(y(sg) ||Ax|11
x0)
0.4
50
0.52
0.2
30
0.10
0.0
-1.50
-0.75
2
20 30
40
80
Hly (CLUE) - H(y|x))
Figure 10: Left: CLUEs are similarly informative under encoder-based and encoder-free initializa- tions. The colorbar indicates the original samples' uncertainty. Its horizontal blue line denotes our rejection threshold. Right: Auxiliary DGMs with more capacity result in more relevant CLUES.
We find that linear explanations (Local Sensitivity) of a non-linear function (BNN) mislead prac- titioners and perform worse than random. While Human CLUE explanations are real datapoints, CLUE generates explanations from a VAE. We conjecture that CLUE's increased flcxibility produces relevant explanations in broader range of cases. In our tabular data user study, only show one pair of context points per test point. We find that otherwise the survey is difficult for practitioners to smaller
follow, due non-expertise in college admissions criminal justice. Using MNIST, scale study, wherein we show participants larger sets of context points. Results are in Appendix
5.3 ANALYSIS OF CLUE'S AUXILIARY DEEP GENERATIVE MODEL
We study CLUE'S reliance on its auxiliary DGM. Further ablative analysis is found Appendix H
Initialization Strategy: We compare Algorithm I encoder- based initialization
(e (2)x)) with Zn 0. As shown in Figure TO, for high dimensional datasets, like MNIST, initializing z with the encoder's 's mean leads to CLUES that require smaller changes in input space to explain away similar amounts of uncertainty (i.e., more relevant). In Appendix [H.T] similar behavior is observed for Credit, second highest dimensional dataset. On other datasets, both approaches yield indistinguishable results. CLUES could plausibly generated with diffcrentiable DGMs that lack an encoding mechanism, such GANs. However, appropriate initialization strategy should be employed.
Capacity of CLUE's DGM: Figure 10 shows how auto-encoding uncertain MNIST samples with low-capacity VAEs significantly reduces these points' predictive entropy. CLUEs generated with these VAEs highlight features that the VAES are unable to reproduce but are not reflective of our BNN's uncertainty. This results in large values of ||Ax|]:: although the counterfactual examples indeed more certain than the original samples, they contain unnecessary changes. As our auxiliary DGMs' capacity increases, the amount of uncertainty preserved when auto-encoding inputs increases as well. |||]| decreases while the predictive entropy of our CLUEs stays the same. More expressive DGMs allow generating sparser, more relevant, CLUES. Fortunately, even in scenarios where our predictor's 's training dataset is limited, we can train powerful DGMs by leveraging unlabeled data.
6 CONCLUSION
With the widespread adoption of data data-driven decision making has come a need for the development of ML tools that can be trusted by their users. This has spawned subfield of ML dedicated to interpreting deep learning systems' predictions. A transparent deep learning method should also inform stakeholders when it does not know the correct prediction (Bhatt et al., (022). turn, this creates need for being able to interpret why deep learning methods uncertain.
We address this issue with Counterfactual Latent Uncertainty Explanations (CLUE), a method that reveals which input features can be changed to reduce the uncertainty of probabilistic model, like a BNN. We then turn to assessing the utility, to stakeholders, of counterfactual explanations of predictive uncertainty. We put forth framework for computational evaluation of these types of explanations. Quantitatively, CLUE outperforms simple baselines. Finally, we perform a user study. It finds that users better able predict their models' behavior after being exposed CLUES.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
JA acknowledges support from Microsoft Research through its PhD Scholarship Program. UB acknowledges support from DeepMind and the Leverhulme Trust via the Leverhulme Centre for the Future of Intelligence (CFI) and from the Mozilla Foundation. AW acknowledges support from Turing AI Fellowship under grant EP/V025379/1, The Alan Turing Institute under EPSRC grant EP/S510129/1 & TU/B/000074, and the Leverhulme Trust via CFI.
REFERENCES
Julius Adebayo, Michael Muelly, llaria Liccardi, and Been Kim. Debugging tests for model explana- tions. Advances in Neural Information Processing Systems, 33, 2020.
Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias.
there's software used across the country predict future criminals. and it's bi-
ased against blacks.
URL
J. Antoran and A. Miguel. Disentangling and learning robust representations with natural clustering. In 2019 18th IEEE International Conference On Machine Leurning And Applications (ICMLA), pp. 694-699, 2019.
Yoshua Bengio. Estimating or propagating gradients through stochastic neurons. abs/1305. 290222209298 2013. URL httt:://arxiv..rrrbbb//////
CoRR,
Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature- based model explanations. Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp- 3016-3022, 7 2020a. [RL.https
//doi.org/1..222 249.//30.2249///0 iaaiiijcaiia/ 2020/417
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. Explainable machine learning in deployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, pp. 648-657, New York, NY, USA, 2020b. Association for Computing Machinery, doi: 00.1145/35110553375644
ISBN 9781450369380
3351095 3375624
Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. Uncertainty as a form of transparency: Measuring, communicating, and using uncertainty. arXiv preprint ar arXiv:2011.07586 2021.
Serena Booth, Yilun Zhou, Ankit Shah, and Julie Shah. Bayes-probe: Distribution-guided sampling for prediction level sets. arXiv preprint 2rXiv:2002.10248 2020.
Ruth MJ Byrne. Counterfactuals in explainable artificial intelligence (xai): evidence from human reasoning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 6276-6282, AAAI Press, 2019.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations, 2019.
Guy W. Cole and Sinead A. Williamson. Avoiding resentment via monotonic fairesss. ArXiv, abs/1909.01 010001151, 2019.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 6970-6979, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 110660964.
10

---

Published as a conference paper at ICLR 2021
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Confer- ence on Learning Representations, 2019. URL SRRLtttts///ppeerrii...eee het.://wwwwwweeeeeeeeeeeeeeeeeee www.://ww.eeefooumi..... B1e0X3C9t0
Erik Daxberger and José Miguel Hernández-Lobato. Bayesian variational autoencoders for unsuper- vised out-of-distribution detection, 2019.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Jan):1-30, 2006.
Stefan Depeweg. Modeling Epistemic and Aleatoric Uncertainty with Bayesian Neural Networks and Latent Variables. Phl) thesis, Technical University of Munich, 2019.
Stefan Depeweg, José Miguel Hernáneez-Lobato, Steffen Udluft, and Thomas A. Runkler. Sensitivity analysis for predictive uncertainty. In ESANN, 2017.
Stefan Depeweg, Jose-Miguel Hernandez-Lobate Finale Doshi-Velez, and Steffen Udluft. De- composition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learn- ing. Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Con- ference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
1184-1193, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. URL http:
//proceedings.nin mlr press http://g.......pppee/ee/////eaaa
depewegl8a.
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, pp. 592-603, 2018.
Finale Doshi-Velez and Been Kim. Towards A Rigorous Science of Interpretable Machine Learning. arXiv e-prints, art. arXiv:17022 100.10002.600, Feb 2017.
Julia Dressel and Hany Farid. The accuracy, fairness, and limits of predicting recidivism. Science advances, 401))ea055800eaaao 55805580, Jan 2018. ISSN 2375-2548. doi: 10. (0.116//ciia
URL httph:ttt////tt
ncbi. nlm. nih. gov gov/pubbmed/2937622
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://c/ccciveee Ics. edi.md
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. arxiv.org/aai wrt.v/orggbbbs////..
6572
Nina Hlaca, Elissa M Redmiles, Krishna P Gummadi, and Adrian Weller. Human perceptions of fairness algorithmic decision making: case study of criminal risk prediction. In Proceedings of the 2018 World Wide Web Conference, pp. 903-912, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
José Miguel Hernández-Loban and Ryan P. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, (CML'15, pp. 1861-1861 JMLR.org, acm. org wwwgg/iiaatinnn.o cfm wtw.::n.............00....... 3.0.1....00001111.3....331 3045316
2015. URL/attp:
Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman. Metrics for explainable ai: Challenges and prospects. arXiv preprint arXiv:1812.04600. 2018.
Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary con- ditioning. International Conference on Learning Representations, 2019. URL https //openrevien net. htt.eeeene..ooommmmosstttttttto

---

Published as a conference paper at ICLR 2021
Shalmali Joshi, Oluwasanmi Koyejo, Been Kim, and Joydeep Ghosh. xGEMs: Generating examplars to explain black-box models. arXiv preprint arXiv:1806.08880 00000008867.222 2018.
Matt Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp.
Curran Associates, Inc., 2017. URL
http::/page
nips cC/ hwtpe/appr//aee 6995
fairness paf
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL htllhttt://anm
lecun. www./ooo//oiiseeeeeeee
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In Proceedings of the Eighth Internationl Conference on Learning Representations (ICLR 2020), April 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017. http: /ppprs/pppes..a//aa/ cctpppscccppprrrre/p 7062 a- unified
to iotereereeiiinnnnnnnggnmnnn mote-moooelll prapeppeeiiiinooeeeppeeeiiiioooo predictioooinnnssee .
David C. MacKay, practical Bayesian framework for backpropagation networks. Neural Comput., 4(3):448 472, May 1992. ISSN 0899-7667. doi: 10. 1ttp///ecco.1922..3444444 URL
httpp://at doi. oor.xx/ooioor//....... 11.00.162200 1162 we11///ncc... 1992. .488
Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. Digital Signal Processing, 73:1 - 15, 2018. ISSN 1051-2004. doi: https://coo.orrg10.001////// 2017. 10.011. URL / /www/ sc wenceediiiiicccccooooooooo rect com/
sc seiianee/articce////////11111212
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL https //openreview net forum? Bd-
Judea Pearl. The seven tools of causal inference, with reflections machine learning. 62(3), 2019. ISSN 0001-0782. doi: 10. 10.1145/324136 URL http:::t:///oo 1145/3241111
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22Nd ACM SIGKID International Conference
on Knowledge Discovery and Data Mining, KDD 116, pp. 114, New York, NY, USA, 2016
ACM. ISBN 978-1-45503-4232-22 doi: 10. 10.114559999772.99977.. URL http: /oi. acm.
oIc/10..114/2999677222 299.//22229999222.77779
Mihaela Rosca, Balaji Lakshminarayanan, and Shakir Mohamed. Distribution Matching in Variational Inference. arXiv e-prints, art. arXiv:1802.0668 Feb 2018.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1:206-215, May 2019.
Chris Russell. Efficient search for diverse coherent explanations. In Proceedings of the Conference Fairness, Accountabiliry, and Transparency, FAT* 19, pp. 20-28, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 978 1450361 255. doi: 10.1145/287550.32875699 7 3287569)
Shubham Sharma, Jette Henderson, and Joydeep Ghosh. Certifai: A common framework to provide explanations and analyse the fairness and robustness of black-bon box models. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, AIES 220, pp. 166-172, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978 1781450371 3375812. 22. htl.https:
doi: 10. 10.1145/3375677. http:::://oooooooo/.111555//5/51 htt..o//.....1/33555222222000 33758811
12

---

Published as a conference paper at ICLR 2021
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimiza- tion with robust bayesian neural networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4134 4142. Curran Associates, Inc., 2016. URL /ttt:://paper..... nips wwwp.ccpppppp..r.etttcpppppppppp 6117- 1117-bayesian-pp mi www. htt.://tttttttta..oooooo-oo--aaa
pdf
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In
Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML pp.
JMLR.org, 2017. http::/tpp//... www.aaa.......attaiaoe. ott.o//.rrrr.c.aar...o.ooooo....
3306024
Austin Tripp. Erik Daxberger. and José Miguel Hermandez-Lobato. Sample-effieient op- timization in the latent space of deep generative models via weighted retraining. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11259-11272. Curran Asso- neurips. cc cc/paperr2020/iiee. ciates, Inc., 2020. URL httpttts///ooceeeeiiiiii Paper. pdf 8 D1132226cad455331177330008222822
Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse linear classification. In Proceedings of the Conference Fairness, Accountability, and Transparency, FAT '19, pp. 10- 19, New York, NY, USA, 2019. ACM. ISBN 978-1-450336615-55 doi: 10.11455/287556.3225566 URL http: /tctp://doi.. acm. org/ 10. 1 328 328773
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law & 'Technology, 31)), 2018.
Adrian Weller. Transparency: Motivations and challenges. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 23-40. Springer, 2019.
L.F. Wightman, H. Ramsey, and Law School Admission Council. LSAC national longitudinal bar passage study. LSAC research report series, Law School Admission Council, 1998. URL https://books google co wttp//oookssiddd////Aa.aa
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization, 2020.
Muhammad Bilal Zafar, Isabel Valera, Manuel Rodriguez, Krishna Gummadi, and Adrian Weller. From parity to preference-basee notions of fairnsss in classification. In Advances in Neural Information Processing Systems, pp. 229-239, 2017.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
This appendix formatted as follows.
1. We discuss the datasets used in Appendix
2. Implementation details for our experiments are provided in Appendix
3. We provide examples of the multiplicity of CLUEs in Appendix C
4. We discuss the application of uncertainty sensitivity analysis in high dimensional spaces in Appendix [
5. We visualize CLUE's optimization in the latent space Appendix
6. We compare CLUE to existing feature importance techniques in Appendix F
7. We provide additional examples of CLUEs and U-FIDO counterfactuals in Appendix G
8. We provide additional experimental results in Appendix H
9. We note additional details of our computational evaluation framework for counterfactual explanations of uncertainty in Appendix
10. We include more details on the setup of our user studies in Appendix J
14

---

Published as a conference paper at ICLR 2021
A DATASETS
We employ 5 datasets in our experiments, 4 tabular and one composed of images. All of them are publicly available. Their details are given in Table
Table Summary datasets used in our experiments. (*) We use 7 feature version of COMPAS, however, other versions exist.
Name
Targets
Input Type
N. Inputs
N. Train
N. Test
LSAT COMPAS
Continuous
Continuous & Categorical
17432
4358
Binary
Continuous & Categorical
5554
618
Wine (red)
Continuous
Continuous
11
1438
160
Credit
Binary
Continuous & Categorical
27000
3000
MNIST
Categorical
Image (greyscale)
28x28
60000
10000
We use the LSAT loading script from Cole Williamson (2019)'s github page. The raw data can be downloaded
921900523//1
and (https:
For the COMPAS criminal recidivism prediction dataset Zafar et al. (217)ss loading and pre-processing seript. It can be found at (https:// com/ mbi http://www.....aaaaaaaaaaaaaaaaa
modified version of
sestreatnenttprooubllioto httpbbiiccacoop__da_t_t_e datta_deooooaaaeeeeaeeeeaaaaaaaa hatpt//ooodd_opps__aa_ae dar__aaa.pppp poyp We add an additional feature: "days served" which we compute as the difference, mea- sured in days, between the "c "c_jail_inp in" and "c_jail _ot"" variables. The raw data is found at
htttps://githab.com/rre httpppbbblaccccpapas.aaaaaaaaaaa anal //t.aaayy/yo/oo/////mtttttetetee
compas www.ss-creesssse tworsssstwoyyarrs.....
The red wine quality prediction dataset can be obtained from and is described detail (https)
//archive.ics. wci. http::e.....cccccccnaaaaaaaaaaaa
The default of credit card clients dataset, which we refer to as "Credit" in this work, can be ob- tained from and is described in detail at (https: //tac///rrccaaa/a icss..... uci ocdu/m//datasess Note that this dataset different from the also com-
fafuutt+er +o fotaultoofcroeeeee #creditt cetttccartcceenei monly used German credit dataset.
The MNIST handwritten digit image dataset can be obtained from latt:://yang lecun.ccm
exdb/ / mnist/).
15

---

Published as a conference paper at ICLR 2021
B IMPLEMENTATION DETAILS
B.1 INFERENCE IN BNNS
We choose Monte Carlo (MC) based inference approach for our BNNs due to these not being limited to localized approximations of the posterior. Specifically, we make use of scale adapted SG-HMC (Springenberg et al. 2016), an approach to stochastic gradient Hamiltonian Monte Carlo with automatic hyperparameter discovery. This technique estimates the mass matrix and the noise introduced by stochasticity the gradients using exponentially decaying moving average filters during the chain's s burn-in phase. We use a fixed step size of 0.01 and batch sizes of 512. We set diagonal mean Gaussian prior p(w) = N(w: 0, o I) over each layer of weights. We place per-layer conjugate Gamma hyperprior over oe with parameters update d for each layer using Gibbs sampling.
10. We periodically
On MNIST, burn in chain for 25 epochs, using the first 15 to estimate SG-HMC parameters. We re-sample momentum parameters every 10 steps and perform a Gibbs sweep over the prior variances every 45 steps. We save parameter settings every epochs until total of 300 sets of weights are stored. This makes for total of 625 epochs.
For tabular datasets, perform burn-in of 400 epochs, using the first 120 estimate SG-HMC parameters. We save weight configurations every 20 epochs until a total of 100 sets weights are saved. This makes for total of 2500 epochs. Momentum is re-sampled every 10 epochs and the prior over weights is re-sampled every 50 epochs. We use a batch size of 512 for all datasets.
B.2 COMPUTING UNCERTAINTY ESTIMATES
In this work, we consider NNs which parametrize two types of distributions over target variables: the categorical for classification problems and the Gaussian for regression. For classification, networks output probability vector with elements (x, w), corresponding to classes { The likelihood function is p(y|x, /x, w) = Cat(y: fx, w)). Given a posterior distribution over weights ((w(D), we use marginalizatton [ to translate uncertainty in w into uncertainty in predictions. Unfortunately, this operation is intractable for BNNs. We resort to approximating the predictive posterior with M MC samples:
D) E Bp "x""""""w)) w)]
M f(x'
w w>p(w(D).
m==0
The resulting predictive distribution categorical. We quantify its uncertainty using entropy:
K H(y"|x" Eiv'
= [x",
=xxx"" DD)
This quantity contains aleatoric and epistemic components (Has He). The former is estimated as:
M
H. = E (Hl)" =
Σ M
Hxy"lx".w) w pDw
m
The epistemic component be obtained the difference between the total and aleatoric entropies. This quantity is also known as the mutual information between y" and w:
H T w D) H xx D) Ep (y [x*, w)).
For regression, we employ heteroscedastic likelihood functions. Their mean and variance are parametrized by our NN: p(y*|x |x', w) = A'y; fa(x", w), fx*(x", w)). Marginalizing over w with MC induces Gaussian mixture distribution over outputs. Its mean is obtained as:
1 Ha
fu w): PDw(D).
M m=0
16

---

Published as a conference paper at ICLR 2021
There is no closed-fome expression for the entropy of this distribution. Instead, use the variance of the GMM as an uncertainty metric. also decomposes into aleatoric and epistemic components (o oī):
(y"lx",D) Bp(wiD) lo (y" x" w) ((wiD))uuy w)l
These are also estimated with MC:
x D)
Σ ply x², w)2
Σ uly x² w))
1 A1
Σ 2² x w): w p(w()
Here, of reflects model uncertainty our lack of knowledge about while of tells us about the irreducible uncertainty noise in our training data.
In Figure 11 we show the fit obtained with a BNN with scale adapted SG-HMC on the toy moons dataset. We would like to highlight 2 key differences with respect the logistic regression example shown in Figure _ Neural networks are very flexible models. They are capable of perfectly fitting non- linear manifolds, such moons. In consequence. when these models present aleatoric uncertainty is most often due to the inputs not containing enough information to predict the targets. As little such noise exists in our particular instantiation of moons, estimates of aleatoric entropy close to 0. Despite their flexibility, selecting a NN involves adopting some inductive biases (Wilson & Izmailov 2020). Additionally, unlike logistic regression, the weight space posterior of a BNN is very difficult to characterize. Both of these things are reflected in the BNN predictive posterior's 's epistemic uncertainty only growing in the vertical axis, instead in all directions.
5
10
10
0.7
H
55
10
10
0.0
10
10
Figure 11: Left: Training points and BNN predictive distribution obtained on the moons dataset with Aleatoric entropy Ha expressed by the model matches regions of class overlap. Right: Epistemic entropy H. grows move away from the data.
SG-HMC. Center:
B.3 ARCHITECTURES AND OTHER NETWORK HYPERPARAMETERS
For all datasets, BNNs fully connected networks with residual connections. Auxiliary VAEs and VAEACs used for tabular data use fully connected encoders and decoders with residual connections and batch normalization at every layer. For MNIST, we employ convolutional bottleneck residual blocks (He et al., (2016) for both encoders and decoders. We use the same architecture for the VAEACs used as ground truth generative models in the computationally grounded evaluation framework put forth in Section 4 Note that the ground truth VAEAC models have slightly larger input spaces due to them modeling inputs and targets jointly. All architectural hyperparameters are provided in Table
In order improve the artificial sample quality of our "ground truth" VAEACs, we leverage two-stage VAE configuration (Dai & Wipf, 22019). For all datasets, the lower level VAEs use the standard tabular data VAE architecture described above, with 2 hidden layers. We use 300 hidden units for MNIST and 150 for other datasets. Additional details provided in Appendix[]
of two-stage VAEs are
We train all generative models with the RAdam optimizer (Liu al. 2020 with learning rate of le 4 for tabular data and 3e 4 for MNIST. We found RAdam to yield marginally better results than Adam.
17

---

Published as a conference paper at ICLR 2021
Table Network architecture hyperparameters used in all experiments. Depth refers to number of hidden layers or residual blocks. Latent dimension values marked with a star (*) refer to the second level VAES for "ground truth" " VAEACs.
Datasel
BNN Depth
BNN Width
VAE VAEAC Depth
VAB Width
VAEAC Width
VAE VAEAC Latent Dim
LSAT
200
3
300
350
4 (*4)
COMPAS
300
350
4 (+4)
Wine
200
300
350
6 (*6)
Credit
200
3
300
350
8 (*8)
MNIST
1200
20 (*8)
We convert categorical inputs to our BNNs into one-hot vectors. When building DGMs, we model continuous inputs with diagonal, unit variance (heteroscedastic) Gaussian distributions. This choice makes these models weigh all input dimensions equally, a desirable trait for explanation generation. We place categorical distributions over discrete inputs, expressing them one-hot vectors. For the LSAT, COMPAS, and Credit datasets, where there are both continuous and discrete features, data likelihood values are obtained the product of Gaussian likelihoods and categorical likelihoods. During the CLUE optimization procedure, we approximate gradients through one-hot hot vectors with the softmax function's 's gradients. This is known the softmax thgggghtttouuhhhhh estimator (Bengio) 2013). It is biased but works well in practice. For MNIST, we model pixels as the probabilities of product Bernoulli distributions. We feed these probabilities directly into our BNNs and DGMs.
We normalize all continuously distributed features such that they have O mean and unit variance. This facilitates model training and also ensures that all features are weighed equally under CLUE's pairwise distance metric in 4 For MNIST, this normalization is applied to whole images instead of individual pixels. Categorical variables are not normalized. Changing categorical variable implies changing two bits in the corresponding one-hot vector. This creates the same l regularization penalty as shifting a continuously distributed variable two standard deviations.
B.4 CLUE HYPERPARAMETERS
As mentioned in Section 5 in our experiments we only apply CLUE to points that present uncertainty above rejection threshold. The rejection thresholds used for each dataset are displayed Table [ The same table contains the values of X used all experiments. practice we define X ^ where d is the input space dimensionality of a dataset. This makes the strength of CLUE's pairwise input space distance metric agnostic to dimensionality. We choose significantly larger value of X for MNIST due to there being large number of pixels that are always black.
Table 5: Values of CLUE's input space similarity weight X₂ and uncertainty rejection thresholds used for all experiments. Next to each dataset's 's name is the the type of uncertainty quantified: standard deviation (r) or entropy (H). We report Ag upscaled by each dataset's 's input dimensionality d.
Dataset
LSAT (o)
COMPAS (H)
Wine (g)
Credit (H)
MNIST (H)
Ae
1.5
2.5
25
H threshold
1
0.2
2
0.5
0.5
C MULTIPLICITY OF CLUES
We exploit the non-convexity of CLUE's objective to generate diverse CLUEs. We initialize CLUE with Zg = (z xx) + e, where e = (2; ; 0, In. and perform Algorithm I multiple times to obtain different CLUEs. We choose do 0.15.. In Figure 12 showcase different CLUEs for the same original MNIST inputs. Different counterfactuals represent digits of different classes. Despite this, all explanations resemble the original datapoints being explained. Being exposed to this multiplicity could potentially inform practitioners about similarities an original input to multiple classes that lead their model to be uncertain.
18

---

Published as a conference paper at ICLR 2021
Different initializations lead to CLUEs that explain away different amounts of uncertainty. In few rare cases CLUE fails: the algorithm does not produce a feature configuration which has significantly lower uncertainty than the original input. This is the case for the third CLUE in the bottom 2 rows of We attribute this to a disadvantageous initialization of z.
Figure T2
Original
CLUE o
CLUE
CLUE
CLUE 3
CLUE
2
e
2
2
r
#=1.75, c=8
==0.30, c=8
#=0.32, c=8
#=051, c=8
==0.19 c=8
== c=8
7
/
/
#=1.70, c=1
#=0.04,
#=0.08,
H=0.03,
==0.07,
H=0.07,
3
3
3 ==068 c=3
==1600 c=9
#=0.4,, c=0
#=0.07, c=0
#=021, c=0
# =0 #0994 94, c=3
8
3
3
8 H=0.86,
8 ==0.14
8 H=0.31,
#=155
H=0.35,
#=0.12,
1
1
1
1
2
2 H=0.21, c=2
#=155 c=2
H=0.06, c=7
#=0.08, c=7
H=0.65, c=7
#=0.M, c=2
2
9 H=0.60,
#=152,
#=0.05,
#=0.14,
H=0.03,
==0.07, e=0
3
3
3 H=0.12, c=3
H=1,47, e=3
#=1.30, c=0
#=0.11, c=3
H=0.44 c=
#=0.01, e=0
2
2
#=1.44, o=2
H=0.66 o=
H-0.13, c=2
R=010,
#=0.15 e=2
R=0.99, o=
9
G
G ==0 c=6
#=143, c=6
H=0.2 c=6
#=0.06, c=0
==0.00 c=6
#=0.05, c=0
8
8
B
8
8
3
#=1.15, c=8
H=
#=017,
H=0
#=097,
H=0.11
#=1.38, a=
#=0.22, o=
#=1.08. c=
H=0000 c=
# #=1.55 cm
H=021 c=
Figure 12: We generate 5 possible CLUEs for 11 MNIST digits score above the uncertainty rejection threshold. Below each digit or counterfactual is the predictive entropy it is assigned H and the class of maximum probability c.
19

---

Published as a conference paper at ICLR 2021
show multiple CLUEs for single individual from the COMPAS dataset. In this case, uncertainty can be reduced by changing the individual's ' prior convictions and charge degree, or by changing their sex and age range. Making both sets of changes simultaneously also reduces uncertainty.
In Figure 13
Original
GLUE
GLUE
CLUE
Oruater then
22-45
Age
Arcan-Aerelisa
AlicanArrerrcr
Fonale
Pure
Midenwaner
Change
Folory
Current Charge
Fakeey
Recffended Setore
Restlended
Confiction
Days Served
Doys Served
Days Served
Figure 13: The leftmost entry is an uncertain COMPAS test sample. right are four candidate CLUEs. The first three successfully reduce uncertainty past our rejection threshold, while the rightmost does not.
D SENSITIVITY ANALYSIS IN HIGH DIMENSIONAL SPACES
In dighndimessional input spaces, V will often not point in the direction of the data manifold. This can result in meaningless explanations. In Figure [4] we show an example where step in the x
direction of V H leads seemingly noisy input configuration for which the predictive entropy is low. An "adversarial examples for uncertainty" is generated. Aggregating these steps for every point in the test leads to an uncertainty sensitivity analysis explanation that resembles white noise.
0.0300
Y
Y
0.0225
0.0150
0.0075
H-1.77
H-0.12
Figure 14: Left: A digit from the MNIST test set with large predictive entropy. Center: The same digit after step taken the direction of H. Non-zen weight is assigned to pixels that are always zero valued. Right: Uncertainty sensitivity analysis for the entire MNIST test set.
E VISUALIZING OPTIMIZATION IN LATENT SPACE
Figure shows a dimensional latent space trajectory from Zo to ZCLUE for a test point from the COMPAS dataset. In practice, we use larger latent spaces ensure CLUEs are relevant.
2.5
0.6
0.40
0.0
0.4 M
0.35
0.2
0.30
2.3
2.5
0.0
2.5
N. step
Figure 15: Left: CLUE latent trajectory for test point from the Credit dataset two-dimensional latent space. The blue dot marks the start of the trajectory and the orange one marks the end. Uncer- tainty levels are displayed in greyscale. Right: Changes in aleatoric entropy for inputs regenerated from latent codes along the trajectory.
20

---

Published as a conference paper at ICLR 2021
F COMPARING CLUE TO FEATURE IMPORTANCE ESTIMATORS
Among machine learning practitioners, two of the most popular approaches for determining feature importance from back-box models are LIME and SHAP SHAP Bhatt et al. (2020b). LIME locally approx- imates the back-box model of interest around a specific test point with surrogate linear model (Ribeiro et al. 2016). This surrogate is trained on points sampled from nearby the input interest. The surrogate model's weights for each class be interpreted as each feature's contribution towards the prediction of said class. Kernel SHAP extends lime by introducing a kernel such that resulting explanations have desirable properties (Lundberg & Lee 2017). For SHAP, reference input is chosen. allows importance to be only assigned where the inputs are different from the reference. For MNIST, the reference is an entirely black image. Note that alternative versions of SHAP exist that incorporate information about internal NN dynamics into their explanations. However, they produce very noisy explanations when applied to our BNNs. We conjecture that this high variance might be induced by disagreement among the multiple weight configurations from our BNNs.
Original
LIME
LIME B
LIME
SHAP
SHAP
SHAP
class:
dlass:
class:
class:
class:
class: 2
{
{
class:
class:
class:
class:
class:
class:
2
2
2
2
9
class:
dass:
class:
class:
class:
class:
3
3
3
3
=
class:
class:
closs: 8
class:
class:
closs: 8
4
4
4
#=4
class:
class:
class:
class:
class:
closs: 6
S
5
class:
class:
class
class:
class:
class: 3
6
class:
class:
class:
class:
class:
class:
7
7
7
7
class:
class:
class:
class:
class:
8
8
8
8
class:
dlass:
class:
class:
class:
class:
class:
class:
class:
class:
class:
class:
Figure 16: High confidence MNIST test examples together with LIME and SHAP explanations for the top 3 predicted classes. The model being investigated is a BNN with architecture described in Appendix B The highest probability class denoted by ÿ.
21

---

Published as a conference paper at ICLR 2021
Figure 16 shows examples of LIME and Kernel SHAP being applied BNN for high confidence MNIST test digits. We use the default LIME hyperparameters for MNIST: the "quickshift" segmenta- tion algorithm with kernel size 1, maximum distance 5 and ratio of 0.2. We plot the top 10 segments with weight greater than 0.01. We draw 1000 samples with both methods.
Using the same configuration, generate LIME and SHAP explanations for some MNIST digits which our BNN assigns predictive entropy above our rejection threshold. The results are displayed in
Figure 17
Original
CLUE
SHAP
SHAP
8
8
8
V
closs:
cluss:
claes: 5
class:
class: 4
class: 5
9
9
9
class:
class:
class:
class:
class:
class: 3
17
7
17
3
==7
V ==7
class:
class:
class:
class:
class:
class: 9
6
8=3
5=6
class:
class:
class:
class:
class:
class:
class:
class:
C
CA
==0
3
7
3
3
S
class:
class:
class:
3
class:
class:
class:
class:
class:
class: 8
3
8
=
closs:
class:
class:
closs:
class:
class: 5
g
8 =
=
class: 9
class:
class: 3
class:
class: 8
class: 3
Figure 17: Ten MNIST test digits for which our BNN's predictive entropy is above the rejection threshold. single CLUE example is provided for each one. For each digit, the top scoring class denoted by LIME and SHAP explanations are provided for the three most likely classes.
positive CLUE attribution means that the addition of that feature will make our model more certain. A positive feature importance attribution means the presence of that feature serves as evidence towards a predicted class. negative CLUE attribution means that the the absence of that feature will make the model more certain. A negative feature importance attribution means the absence of that feature would serve as evidence for particular prediction. While CLUE and feature importance techniques solve similar problems and both provide saliency maps, CLUE highlights regions that need to be added or removed to make the input certain a predictive model. some cases, we see that feature importance negative attribution aligns with CLUE negative attribution, suggesting the
22

---

Published as a conference paper at ICLR 2021
features which negatively contribute the model's predicted probability the features that need to be removed to increase the models' certainty. CLUE's ability to suggest the addition unobserved features (positive CLUE attribution) is unique.
The feature importance methods under consideration are difficult to retrofit for uncertainty. They are unable to add features; they limited explaining the contribution of existing features. This may suffice if our input contains all the information needed to make a prediction for a certain class but otherwise results in noisy, potentially meaningless, explanations.
Generative-model based methods methods are counterfactual because they do not assign importance the observed features but rather propose alternative features based the data manifold Chang et al.) (2019). This is the case for FIDO and CLUE. Generative modeling allows for increased flexibility, which is required when dealing with uncertain inputs. Quantitatively contrasting feature importance and uncertainty explanations under existing evaluation criteria Bhatt et al. (2020a) is an interesting direction for future work.
Methods like LIME and SHAP require choice of class to produce explanations. This complicates their use in scenarios where our model uncertain and multiple classes have similarly high predictive probability. On the other hand CLUES are class agnostic.
G ADDITIONAL CLUE AND U-FIDO EXAMPLES
We provide additional examples of CLUES generated for high uncertainty MNIST digits in Figure 18 U-FIDO counterfactuals generated for the same inputs are shown Figure T9 Both methods often attribute importance to the same features. However, in almost all cases, CLUE is able to reduce the original input's uncertainty significantly more than U-FIDO. The latter method suggests smaller changes. We attribute this to U-FIDO's input masking mechanism being less flexible than CLUE's latent space generation mechanism.
Original
CLUE
ACLUE
Original
CLUE
ACLUE
Original
CLUE
ACLUE
8
9 #=1.66
3
3
7
#1.337
#00.36
H=0.20
#=1.34
#=0.17
9
9
4
#=1.48 48 H-0.43
#=1.99
H=0.37
H=1.32
H=0.28
17
7
5
3
1.40
H--55
f1 0007
5
3
8
1140 H-1.00
1-37
0.24
#--118 H.12.1 0.12
7 #=1.77 #=012
7
C
98 #=L14 H=0.36
#=134 #=0.66
Figure 18: CLUES generated for MNIST digits for which BNN's predictive entropy is above the rejection threshold. The BNNs predictive entropy for both original inputs and CLUEs is shown under the corresponding images.
23

---

Published as a conference paper at ICLR 2021
Original
U-FIDO AU-FIDO
Original
U-FIDO AU-FIDO
Original
U-FIDO AU-FIDO
8
8
g
3
3
#=1.37 -00668
H=166
H=101
H=0.53
9
9
4
#=1.48
#00774
#=1.59
H=1.48
H=1332 H=035
1
17
5
5
#=1.40 I1 -090.99
#=151 H=
#=116 #=0077
33
33
#1144 H=0.2
#=1.37 H=1.14
#=116 H=051
7
Qa --13 1790.95
98
-137 0.65
1=114 11-0
Figure 19: U-FIDO counterfactuals generated for MNIST digits for which our BNN's predictive entropy is above the rejection threshold. The BNNs predictive entropy for both original inputs and counterfactuals is shown under the corresponding images.
24

---

Published as a conference paper at ICLR 2021
H ADDITIONAL EXPERIMENTAL RESULTS
H.1 ABLATION EXPERIMENTS
In this subsection, we modify some of CLUE's components individually and observe the effects on the procedure's 's results.
Initialization Strategy: Figure 20 compares Algorithm Is encoder-based initialization Zu Ho (z(x)) with Zu 0 on all datasets under consideration. For the LSAT, COMPAS and Wine datasets, both approaches produce indistinguishable results. On Credit, our second highest dimensional dataset, using an encoder-based initialization allows for CLUES to stay slightly closer to original inputs in terms of & distance.
The difference between both approaches is largest MNIST. We conjecture that this might be due to the higher dimensional nature of the latent space used with this dataset making optimization more difficult. By initializing z as the VAE encoder's mean, our optimizer starts near a local minima of d(x,xx) and potentially of ((z). When Algorithm|| I is applied, the magnitude of V H might not be large enough escape this basin of attraction. Thus, CLUE tends to leave most input features unchanged, only addressing those with most potential to reduce uncertainty. This is also desirable behavior for low uncertainty inputs; the closest low uncertainty sample is the input itself.
COMPAS
Credit
2g 100 (4|Mo)
zo=0
0.45
Ri: [4g (Xt (R(x))
Zo 0
0.80
10.0
30
0.34
0.62
K)
||Ax|
6.0
0.22
||x
20
0.45
2.5
0.11
0.28
0.00
0.10
0.2 0.0
0.2
0.3
0.3
Hy (XCLUE) (x((xx))
H(y|xcLuE)
H( xx)
Wine
LSAT
250 = p4g (2) (x0)
Zg O
Zg (x/
2g=0
2.20
1.10
10
1.78
1.03
(Dx) 6)
[Ax]]]
1.35
0.95
0.92
0.88
0.50
0.80
0.7
0.0
-1.4
0.7 0.0
< <yy(NCLuE) ((y)xx))
a (MCLUE) a (xx)
MNIST
1 (Xo)
zo=0
1.80
150
1.38
||x/|
100
0.95
50
0.52
0.10
0.00 -1.0
-0.75
H(y|xcwe)) Hy()))))
Figure 20: Initialization strategy experiment results for all datasets under consideration. Colorbars' horizontal blue line denotes each dataset's 's rejection threshold.
25

---

Published as a conference paper at ICLR 2021
To capture predictive model's reasoning, CLUE's DGM must be flexible enough to preserve atypical features in the inputs. As shown in Figure 21 reconstructions from low-capacity VAEs do not preserve the predictive uncertainty of original ippus. The CLUEs generated from these DGMs either leave the inputs unchanged or present large values of Áx while barely reducing H: these degenerate CLUES simply emphasize regions of large reconstruction As our DGM's capacity increases, so does the amount of uncertainty preserved in the auto-encoding operation. The amount of predictive uncertainty explained by CLUEs, which is given by the difference between the autoencoded input uncertainty (orange bars) and CLUE uncertainty (blue bars), increases. We see clear relationship between dataset dimensionality and size of latent space needed for CLUE to be effective.
Capacity of CLUE's DGM:
COMPAS
Credit
5.2
0.6
7.5
5.0
4.8
0.15
4.6
0.10
4.4
0.2
5.5
0.05
4.2
5.0
0.00
0.0
VAE latent dimension
VAE latent dimension
Wine
LSAT
3.0
1.05
2.5
1.6
2.0
1.00
1.5
1.5
1.4
1.0
0.95
1.3
0.0
1.2
2
3
8
12
16
2
4
5
8
12
16
VAE latent dimensson
VAE latent dimension
MNIST
Hy (x(=n))
0.6
(y (CCLUE
||Ax||ı
0.2
30
0.0
30
VAE latent dimension
Figure 21: Amount of uncertainty explained away and & distance between original inputs and CLUES for every dataset under consideration and different capacity VAEs.
26

---

Published as a conference paper at ICLR 2021
Output Space Regularization Parameter Ay: In Figure 22
show how increasing Xy reduces the proportion of samples for which the predicted class differs between original inputs and CLUEs. Interestingly, LSAT, Wine and COMPAS, small, but non-zero, value of Ay results in more uncertainty being explained away by CLUE. However, strongly enforcing similarity of predictions generally comes the cost of smaller amounts of uncertainty being explained away.
COMPAS predictions stay the same for all values of Ay. Class predictions only depend on this dataset's input features (Age and Previous Convictions) (Dressel & Farid 2018). We find that the remaining features can increase or reduce confidence in the prediction given by the two key features, but never change it. CLUEs only change non key features, reinforcing the current classification. On MNIST, we find that, for certain values of Xy, classifying CLUEs results in a lower error rate than classifying original inputs. This shown in Figure 23 We did not observe this effect for other datasets.
Cradit
COMPAS
0.04
pred. change
0.085
(y (ccuu))
prediction change
0.14
pred. change
A/f
0.13
0.12
0.02
0.080
0.12
0.11
0.075
0.10
-0.02
0.10
-0.04
0.070
0.08
0.08
0.0
0.5
1.0
1.5
2.0
Ag
Xy
Wine
LSAT
[0x (3nTX (A
prediction change
0.4
0.24
pred. change
0.7
0.041
pred. change
0.042
0.6
pradition
0.20
0.040
0.18
0.5
0.2
0.16
0.038
0.036
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
Xy
Ag
Figure 22: CLUE AH vs prediction change for all datasets under consideration. Prediction change refers to the proportion of CLUEs classified differently than their corresponding original inputs. All values shown are averages across all testset points above the uncertainty rejection threshold.
Applying CLUE to non-Bayesian NNs: These models are unable to capture model uncertainty. We train deterministic NNs on every dataset under consideration using the architectures described in Appendix [B.] We generate counterfactuals for their noise uncertainty. As shown in Figure [4] CLUE effective explaining away noise uncertainty for regular NNs. More uncertain inputs are subject to larger changes in terms of l distance.
27

---

Published as a conference paper at ICLR 2021
0.20
Hy XcLUE) (y (x))
0.6
proportion
0.15
pred. change
0.10
CLUE pred.
0.4
original pred.
0.3
0.05
0.2
3
Xy
x
Figure 23: Left: Prediction change refers to the proportion of CLUEs classified differently than their corresponding original inputs. Setting value of xy of around 0.7 results in class predictions for CLUES being closer to the true labels than the original class predictions. Right: Reduction in predictive entropy achieved by CLUE. All values shown are averages across all testset points above the uncertainty rejection threshold.
COMPAS
Credit
0.67
0.70
10
30
0.53
0.53
| 1
1 xx0)
20
0.38
0.36
0.24
10
0.20
0
0.03
-0.50
-0.25
0.00
- -0.50
-0.25
0.00
H (yy|xxLUEE (y|xx))
H (y XcCUE) Hy((x))
Wine
LSAT
1.03
1.05
1.00
0.85
||
4
0.67
0.94
x()
0.49
0.89
0
0.84
-0.2
0.0
-0.2
0.0
0.2
o(y ((y|XCLUE) - or (y (()x))/))
( XCLUE) - oy)(xo)
MNIST
0.70
150
0.53
1
100
RI x0)
0.35
50
0.18
0.00
-0.5
0.0
0.5
H(y) xCLUE) H Hy)xx) (xx)
Figure 24: Amount of noise uncertainty explained away E shift in input space for all datasets under consideration when applying CLUE to regular NNs. The colorbar indicates the original samples' predictive uncertainty.
28

---

Published as a conference paper at ICLR 2021
H.2 VERIFYING RESULTS FROM OUR COMPUTATIONAL EVALUATION FRAMEWORK
Our computational evaluation framework relies generating artificial data. There is reasonable concern that the characteristics of this data may not reflect that of real-world data, biasing our results. As explained in Appendix we careful to powerful g.t. DGM models that generate high quality artificial data. Be that as it may, we validate the results from our computational evaluation framework by performing an analogous informativeness vs relevance experiment on real data.
As we do not have access to the generative process of the real data, we can not exactly quantify the uncertainty of inputs how in-distribution they are. We instead resort to quantifying informativeness as the amount of our BNN's predictive uncertainty explained away AH. = H HH(y(x)). We measure relevance as the L2 distance of each counterfactual to La nearest neighbor within the train-set dv N 2x,,, D).
We report mean values of AH, 2 D) and dx across all uncertain test points. A test point is deemed to be uncertain according to the criteria outlined in Appendix [B.4 The hyperparameters employed for CLUE also match those provided in Appendix [.44, The step size used with local sensitivity analysis and U-FIDO's Λ parameter are found via grid search with methodology analogous to the one described in Section 5.1
AH
Table 6: Quantities of informativeness (AH, higher is better), relevance (dNN-2(Xe: D), lower is better) and their ratio x2 Dj. higher is better) obtained on real data from the LSAT, COMPAS and Wine datasets. The numbers in parenthesis indicate dataset dimensionality.
AH
Method
LSAT (4)
COMPAS (7)
Wine (11)
dxN-a(Xe, Xo)
ATl
4ft/dwx-2
(X,, Xn)
AHl
d Xg)
AH
Sensitivity CLUE
0.482
0.003
0,0192
7.975
0.265
0.033
3.317
0.481
0.154
0,080
0.092
1.664
0.067
0.014
0.737
1.274
1.409
1.188
U-FIDO
0.085
0.077
0.969
0.084
0.022
0.627
1.223
1.307
1.241
Table 7: Quantities of informativeness (AH, higher is better), relevance (dNN-2(X,, D), lower is better) and their ratio (NNN D) higher is better) obtained on real data from the Credit and MNIST datasets. The numbers in parenthesis indicate dataset dimensionality.
A'
Method
Credit (23)
MNIST (784)
dyx--2(Xe: Xo)
AH
dNN-2(Xe: Xa)
AH
AR//ux
Sensitivity CLUE
0.770
0.224
0.121
6.903
0.601
0.087
1.025
0.147
0.293
4.374
0.628
0.153
U-FIDO
1.863
0.017
0.052
4.887
0.409
0.088
As shown in Table and Table 7 CLUE outperforms U-FIDO in terms of d on all datasets AH except wine, where both approaches are very similar. The same is true for the ratio ZNx-;" Like in the artificial data experiments from Section 5.T the difference between both methods is most stark for high dimensional datasets (MNIST and Credit). Here, CLUE is able to explain away more uncertainty while providing counterfactuals that are similarly close to the training data. Sensitivity able to greatly reduce uncertainty in high dimensions. However, this comes the cost going off the data manifold. In low dimensions, there are less possible directions in which steps can be taken, rendering the direct gradient based approach less powerful.
We note that the similarity of these results to the ones obtained in the analogous experiments from Section 5.1 suggest the unbiasedness of our computational evaluation framework.
H.3 ADDITIONAL ANALYSIS OF USER STUDY
While the main text showed the mean accuracy of CLUE over all tabular questions, we also consider the breakdown of accuracy by dataset and by test point certainty in Table CLUE outperforms all baselines on both datasets. We find that sensitivity does significantly worse in higher dimensions (on COMPAS), lending further credence the intuition described in Appendix D
29

---

Published as a conference paper at ICLR 2021
When splitting by the certainty of test points, immediately notice that accuracy for uncertain test points is quite high for all methods. This similarity is expected since certain context points are the only factor that varies between each method's survey. Survey participants seemed not the certain context points to identify uncertain test points. This is probably due to pilot procedure, wherein Participant A carefully paired test points with relevant uncertain context points. Indeed, the random baseline, which controls for the possibility that our task can be solved without access to relevant counterfactual, performs best on uncertain test points. However, we note large difference between methods' results when identifying certain test points. CLUE's accuracy almost doubles the second best method's (Human CLUE). When generating Human CLUEs Participant had knowledge of the uncertain context point, but not the test point (just like other methods). For this reason, we expect to see dissimilarity methods' performance on certain test points. CLUE's ability to bring about most relevant contrast is one possible explanation for why it does so much better than baselines for certain context points.
Table 8: Accuracy (%) of participants on the Tabular main survey broken down by dataset and by certainty of test points.
Combined
LSAT
COMPAS
Certain Test
Uncertain Test
CLUE Human CLUE
82.22
83.33
81.11
71.00
96.25
62.22
61.11
63.33
38.00
92.50
Random
61.67
62.22
61.11
31.00
100
Local Sensitivity
52.78
56.67
48.89
20.90
92.50
I ADDITIONAL DETAILS ON THE GENERATIVE MODEL USED IN THE PROPOSED COMPUTATIONALLY GROUNDED EVALUATION FRAMEWORK
The framework described in Figure uses a conditional DGM, specifically VAEAC (Ivanov et al. 2019), to both generate artificial data and to evaluate explanations for said data. VAEs are known for generating blurry or overly smoothed data, For our evaluation framework to work well, we require the ground truth DGM to generate sharp data, with atypical characteristic that would to lead to predictor being uncertain. We can ensure that this is the case by using large latent dimensionality. However, this brings forth another well-known issue with VAEs: distribution mismatch (Dai & Wipi) Antoran Antorrn Miguel| 2019 Rosca et al., 2018). The region of latent space where the encoder places probability mass, also known as the aggregate posterior,
(z( = go (2(xx))p(x) dx
does not match the prior p(z).
1.75
1.25
4
6
T
0.75
-5
0.25
Figure 25: Predictive entropy estimates for artificial MNIST digits generated from a 2-dimensional VAE latent space. The MNIST test set digits have been projected onto the latent space and displayed with a different color per class.
30

---

Published as a conference paper at ICLR 2021
To visualize this phenomenon, train BNN and VAE on MNIST. We sample points from the VAE's latent space and evaluate their uncertainty with the BNN. As shown in Figure [2 clusters of same-class digits form in latent space. The aggregate posterior presents low density in the spaces between clusters. Digits generated from these areas are of low-quality, causing our BNN to be uncertain. The outer regions of latent space, where the isotropic Gaussian prior has low density, also generate uncertain digits.
Recently, Dai & Wipf (0119) have proposed the two-level VAE a solution distribution mismatch. After training a standard VAE, a second VAE is trained on samples from the first VAE's latent space. As illustrated in Figure 26 the aggregate posterior over the inner latent variables, which we denote by (u), more closely resembles the prior. The joint distribution over inputs and latent variables factorizes as: p(x, z, u) (((z))p(z, u)p(u). We refer the reader to (Dai Wipf 2019) for detailed analysis. Figure 27 shows that, while generating digits from samples of p(z) results in
large amount of low-quality or OOD reconstructions, samples from p(u) map to clean digits. The
two-stage mechanism restores the VAE's pivotal ancestral sampling capability, ensuring that our experiments with artificial data will representative of methods performance on real data.
9, (x (M]2)
9, (0))9g_(1)) (e)
p(x).
2,
U
9, xz
9 (u z)
Figure 26: In its first stage, the two-level VAE maps input samples to approximate posteriors in the outer latent space. The aggregate posterior over this latent space need not resemble the isotropic Gaussian prior. The second VAE maps samples from the outer latent space to approximate posteriors in the inner latent space. The aggregate posterior over the inner latent space more closely matches the prior.
9
3
7
3
9
3
9
7
3
B
8
S
8
2
7
7
7
0
4
6
2
7
3
9
9
9
8
6
5 5 4 47 109
8 z 1
7
9
9
2
8
3
8
4
/
0
7
8
9
$
4
/
3
3
2
7
6
8
0
&
6
7
2
9
6
8
3
5
3
9
6
1
9
8
6
9
0
6
4
8
7
6
3
4
7
8
7
0
9
0
2
1
f
4
2
4
9
67647
4
3
4
0
1
3
9
0
4
2
2
4
7 9
Figure 27: Left: Digits generated from the inner latent space of a VAEAC trained on MNIST with two-level mechanism. Right: Digits generated from the latent space of a VAEAC trained on MNIST. and are drawn from (O, I).
In order generate artificial data, we draw samples from the auxiliary latent space, map them back the VAEAC's latent space and then map them to the input space. This allows for high-quality sample generation. this way, a single VAEAC can be used for both ancestral sampling and conditional sampling. In addition, it allows us estimate the log-likelihood of inputs as:
log (x) = log / pe, (k/z))p,CC ())(())) dz du
(8)
In g parameter subscripts refer to the outer (1s level) and inner (2nd level) networks. In order to preserve computational tractability, approximate po, (z/u) u) with point estimate placed at its mean
31

---

Published as a conference paper at ICLR 2021
P8: (z|u) = 6(2 Moz (z/u)). We further approximate 8
with importance sampling:
K
po, (x (2)
log Px) ~ log
(u/x)
(9)
K
q( (ux(x)
I.1 COMPARISON OF METHODS UNDER A GROUND TRUTH DGM
The two-level VAEAC setup described above partially addresses the concern that our synthetic data might not be diverse enough to highlight differences among the methods being compared. Indeed, our results from Table and Table 2 show noticeable differences in performance across methods.
We address the opposite concer;; methods that leverage auxiliary VAEs might be unfairly advantaged under our functionally grounded framework, as the generative process of our synthetic data is also VAE-based. Because VAEs are very flexible neural network based generative models, using them as a ground truth provides relatively little inductive biases for auxiliary DGMs to take advantage of. Additionally, our ground truth VAEAC captures the joint distribution of inputs and targets. The metric of interest, AHgt. only depends on the conditional distribution over targets Por (y/x). Our auxiliary DGMs only model inputs.
Two of the methods we evaluate, U-FIDO and CLUE, leverage auxiliary DGMs. Thus, both would be equally advantaged. The Aggs vs A Pog ground truth VAEAC. However, we observe the largest difference between CLUE and U-FIDO on in this metric.
Pgt metric from Table 2 the most dependent on the
J DETAILS ON USER STUDY
J.1 ADDITIONAL DETAILS TABULAR USER STUDY
For our pilot point selection procedure, take points from each dataset's test set that above the uncertainty rejection thresholds described Appendix[B.. B.4as uncertain points. Points below the thresholds labeled as certain points. Pilot procedure participants, referred as participant A in the main text, were not informed that the pools were split up by the points' certainty with respect to the BNN being explained.
Suppose you shown following examples.. left, uncertal esample wi nec background. In the middle, we have cartain different between uncersain red text feerures an "the feet that needed to change for the Al system certain." are same
interetedd dentarcing fow tumene percine ancerall my trailipence informatie
hee presene
Your
The red text which examples. can about
kape complensy
The stade slo dil ie wnowdtt wicuts coeplito you tark condee The monu well withdie eu ane point surig the stady, ln serve, and thher prejotion.
have evammplss. cannot
Rasse rara tur surey dircusees dara vas consain the and succet intil the comad for taking servis ii okun an the sel neceeee wes systers.
determ ne the certalitty of the test example, However, looks like, when the uGdA
tharefore.
petertally defect blkare systems.
could conclude system be certain the text example
fo vess connerr eveergarre the discus dlis www.tatroiittttaa.o.oooo anto
Uncertain
Certain
My aikeg the sutien tekone nige but pour pud on stude vette youriccul
stucy arp and for reusee
LSAT
41.0
LSAT
39.0
LSAT
34.0
UGPA
3.7
UGPA
3.0
UGPA
3.4
Do you consent to pa partielpating in the study?
Race
White
Race
White
Race
White
Sex
Female
parteipatee
(a) Consent Form for the Tabular Main Survey
(b) Attention Check for the Tabular Main Survey
Figure 28: Setup of tabular user studies.
We now go through the various sections of the main survey. In Figure 28a include the consent form used our user studies. This user study was performed with the approval the University of Cambridge's Department of Engineering Research Ethics Committee. Only three participants who
32

---

Published as a conference paper at ICLR 2021
the uncartaen enampl on left the cartain ecample in middle, will the Al be cersain on the now esarppe on the right?
Given uncertain the and the certain the middle. will the Al be certain on the new ecample on the right?
Uncertain
Certain
LSAT
41.0
LSAT
LSAT
41.0
26-45
20
Race
Caucalin
Race
Cauc
2.5
3.3
Race
White
Race
White
Race
Currett Charge
Defore
Charge
fatore
Folieg
Reoffrneed
Cureet Chane
Fabity
Sax
Male
Sex
Male
Sex
Male
Corvictine
Convittine
Prir Corvicion
Days Served
Days Served
Days Served
uncertate on the scampi right.
acample
right
Yes,
example right.
Glven the uncersain enampia plu on the left and the cartain ceample in the middlo, will the All be certain on the now esamppe on the right?"
Givan the uncertain example on the left and the certain example the middle, be certain on exammple the righ??
Uncertain
Certain
?
Gertin
LSAT
37.0
LSAT
37,3
LSAT
37.0
Race
Race
Nalva Nnacial
UOPA
4,0
UGPA
3,8
UQPA
3.4
Charge
Race
White
Race
Race
Asian
Ferfonece Bobue
Yei
Reefhndee
Redtteded Entrre
Oveection
Dovections
Sex
Female
Female
Sex
No, the Al be uncertain on the examppee on right.
No. the Al bnunnertenn oe the acampi on right.
Al becentan coumpleon thriigh.
Yoo, the All will be certein on the womppl on the right.
(a) Two LSAT questions with certain points generated (b) Two COMPAS questions with certain points generated by CLUE
by CLUE
Figure 29: Example Tabular Main Survey questions
were asked to take the survey did not provide consent and thus exited the form. We still ensured that at least ten participants took each the four survey variants.
We then include an example question for each dataset, called an "attention check." An example is shown in Figure 28b) Note that the answer to this example question is provided in line. Later in the survey, we ask participants this exact same question. We ask one attention check per dataset. If participants get the attention check wrong for both datasets, we void their results. We only had to void one result. This did not affect criteria of ten completed surveys per variant. The consent form and attention check questions were the same for all survey variants. The main survey participants were first asked the LSAT questions followed by the ten COMPAS questions: decision since the dimensionality of LSAT is lower than that of COMPAS, easing participants into the task. Examples of questions from the CLUE survey variant are shown in Figure 29)
made this design
J.2 MNIST USER STUDY
In order to validate CLUE on image data, we create modified MNIST dataset with clear failure modes for practitioners to identify. We first discard all classes except four, seven, and nine. We then manually identify forty sevens from the training set which have dashes crossing their stems. Using K-nearest-neighhors, we identify the twelve sevens closest to each of the ones manually selected. We delete these 520 sevens from our dataset. We repeat the same procedure for fours which have a closed, triangle-shaped top. We do not delete any digits from the test set. We train a BNN on this new dataset. Our BNN presents high epistemic uncertainty when tested on dashed sevens and closed fours consequence of the sparsity of these features in the train set.
We evaluate the test set of fours, sevens, and nines with our BNN. Datapoints that surpass our uncertainty threshold are selected candidates to be shown in our user study uncertain context examples or test questions. We show example CLUES for a four and a seven that display the characteristics of interest in Figure 30
33

---

Published as a conference paper at ICLR 2021
Example
CLUE
Changes
7
Uncertain True
Uncertain: False
Example
CLUE
Changes
4
4
Uncertain True
Uncertain: False
Figure 30: Examples of high uncertainty digits containing characteristics that are uncommon in our modified MNIST dataset. Their corresponding CLUEs and ACLUEs are displayed beside them.
Leveraging the modified MNIST dataset, we run another user study with 10 questions and two variants. Unlike our tabular experiments, we show practitioners a set of five context points to start, opposed to pair. This set of context points is chosen at random from the training set. The first variant involves showing users the set of context points, labeled with their uncertainty surpasses our predefined threshold. We then ask users predict if new test points will be certain uncertain to the BNN. The second variant contains the same labeled context points and test datapoints. However, together with uncertain context points, practitioners shown CLUEs of how the input features be changed such that the BNN's uncertainty falls below the rejection threshold. The practitioners are then asked decide if points' predictions will be certain not. If CLUE works intended, practitioners taking the second variant should be able to identify points on which the BNN will be uncertain more accurately.
Here a of examples lbeled with make the All uncertaln not. comrspondiin
information. uncertain on new dig to. Note that no CLUEs shown with the questions, Feel
mutaeqqunnt qusttions, you be identify
new point
Durrpfe
CLUE
Example
Example
④
9
4
Burglee
CLUE
Change
4
4
Example
Uncenainn
Dumrgee
CLUE
Certain
(b) Example question
(a) Example Context Set with CLUES
Figure 31: MNIST User Study Setup
The first variant was shown to 5 graduate students with machine learning expertise who only received context points and rejection labels (uncertain or not). This group was able correctly classify 67% of the new test points as high or low uncertainty. The second variant was shown to 5 other graduate students with machine learning expertise who received context points together with CLUEs cases of high uncertainty. This group was able to reach an accuracy of 88% on new test points. This user study suggests CLUES are useful for practitioners image-basee based settings as well.
34


Reviews:
[{"review_id": "xBknESk3wi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of making Ai/ML-systems more trustworthy making the uncertainty associated with a model, in this case BNN, visible, more interpretable. Interpretability and knowing the limitations and uncertainties associated with a model are definitely very interesting research challenges. These topics are very relevant for ML, AI, Explainable AI etc., but I still think that they are also for ICLR (even if many conferences in the ML/AI/XAI will also fit this paper)\n\nI find the main ideas innovative, the paper is well-written, explained and even includes some kind of “small” user study. The authors also provide a framework for evaluating the counterfactual explanations of uncertainty provided, using informativeness, and they carry out well-designed experiments for validating CLUE.\n\nP. 7, under section 5.2. Can the first sentence be referred to Hoffman? I don’t think so. Are the participants used in the user study be good representatives of the possible users/practitioners of CLUE (as also stated in the conclusions)?\n\nThere has come a recent survey on counterfactuals, that I think it is relevant for this work:\n\nVerma, S., Dickerson, J., & Hines, K. (2020). Counterfactual Explanations for Machine Learning: A Review. arXiv preprint arXiv:2010.10596.\n\nI understand that it is out of the scope of this paper, but the notion of counterfactuals used in the ML community, like the one used in this paper (section 2.3), is quite narrow compared to how we use counterfactuals and contrastive explanations in real life. I think the richness and complexity of counterfactual explanations is well illustrated in \n\nByrne, R. M. (2019, August). Counterfactuals in Explainable Artificial Intelligence (XAI): Evidence from Human Reasoning. In IJCAI (pp. 6276-6282).\n\nPerhaps this is something to discuss in the future.\n\n(just to make clear: I am not involved in any of the references given, just thought that they can be of interest for this paper).", "extra_scores": {}, "extra_text": {}}, {"review_id": "wUcTgPRR7JP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of explaining the uncertainty of a prediction made by a differentiable probabilistic model (as opposed to the prediction itself) through counterfactual explanations. They propose a technique, CLUE, which optimizes their counterfactual metric in the latent space of a deep generative model. To validate their approach, they introduce a quantitative evaluation for uncertainty metrics and conduct a user study, while also analysing CLUE's reliance on the auxiliary DGM.\n\nStrengths:\nThe empirical validation of the approach is strong. \n\nThe authors deserve particular credit for \"creating their own baseline\" by adapting a previous counterfactual approach (FIDO) to the problem at hand. \n\nThe user study is well-executed, and produces a pretty stark improvement over baselines, including human-selected counterfactuals.\n\nWhile the introduced approach does require the non-trivial complexity of training a DGM, conceptually the method is an elegant way of dealing with one of the big challenges for counterfactual explanations - staying on the data manifold.\n\nWeaknesses:\n1. The framing (abstract/introduction) of the paper took a while to wrap my head around, and could probably be improved. In particular, for classification problems, the \"simple, stupid\" approach of looking for the most negative feature attributions for a prediction seems like it would produce an explanation of uncertainty (though not a counterfactual one, so not competitive to CLUE). This makes Figure 1 pretty puzzling, as it's not clear to me why standard attributions aren't useful for uncertain predictions.\n2. Qualitatively, how would CLUE compare to a standard counterfactual explanation?\n3. Why isn't U-FIDO included in the user study? Given that it performed the best in section 5.1 on the datasets used, the results would be interesting.\n3. In the user study, is the test example linked to the two context points at all? I wouldn't expect unrelated context points to be that useful in classifying a random test example.\n\nNitpicks:\n- In the appendix, CLUE is compared against Shapley/LIME on MNIST. LIME is a pretty strange choice, and has never been shown/claimed to be remotely SOTA on neural networks. Something like integrated gradients would be more relevant/interesting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ij9NqCc8Mt3", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe authors consider the problem of post-hoc explainability for decisions rendered by machine learning models.  They focus on addressing uncertain model predictions, producing counterfactual data that is both likely under a generative model of the data, as well as more certain in the classification task.  They present both experimental evidence, as well as a user study geared towards practitioners, that show the benefits of counterfactual explanations targeting uncertainty.\n\n## Strong points\n\nWhat really sets this paper aside for me is its explicit focus on uncertain model decisions, as well as its inclusion of a user study.  \n\n- While other works do use the point estimates of model softmax outputs as part of their methods (e.g Progressive Exaggeration), they do not focus on providing counterfactual explanations specifically for those points where the model is uncertain.\n\n- Most works in explainable AI, even those targeted towards practitioners, do not evaluate the utility of their tools in front of an audience of machine learning model developers.  Bravo for undertaking this work which is (currently) under-valued in the ML community.\n\n## Weak points\n\n- In step 4 of Algorithm 1, why do you need to focus on a BNN for obtaining $H(y|x)$?  Going back to my previous point about limiting usefulness of CLUE, there are many methods that yield $P(y | x)$, even by approximate Bayesian inference.    \n\n- Is it the case that as the method converges (if it converges?), $H(y|x)$ is likely to decrease, while $d(x, x_0)$ is likely to increase?  Do you attempt to anneal the relative contribution to the loss to account for this?  Furthermore, do you ensure that the relative contribution to the loss is balanced between $d(.,.)$ and $H(y|x)$??  If these become imbalanced, one will surely dominate the direction of $\\nabla_Z \\mathcal{L}$\n\n- In the description of the baselines used for experiments in section 5, the localized uncertainty sensitivity analysis seems artificially weak; why not include a more robust ensemble of models that produce softmax output over class assignments?  Or a proper probabilistic model like a GP?\n\n## Recommendation\n\nThe effectiveness of CLUE, and indeed of every counterfactual explanation method cited that makes use of an auxiliary generative model of the data is bounded by the faithfulness of this DGM to model the density.  This is a more fundamental limit on the applicability of these methods, not specific to CLUE, but worth stating IMO.\n\nI disagree with the statement detailed in section 4 after the evaluation procedure involving $\\mathcal{H}_{gt}$ capturing ground truth aleatoric uncertainty: $p_gt(y|x)$ is just another generative model trained by estimation of the data, it’s not special.  A better measure of aleatoric uncertainty would involve the variance of $p(y | x)$ taken over multiple independent models (see Snoek et al.).  I also find the argument about adversarial weakness that immediately follows is a bit confusing \"*Approaches that exploit adversarial weaknesses in the BNN will not transfer to the g.t. VAEAC, failing to reduce uncertainty on error*\".   \n\nFundamentally, though I see areas where the work could be improved, I believe the work is sufficiently different to existing counterfactual explanation methods to be accepted.\n\n## Questions for the authors\n\n- In section 3 where you penalize the distance $d(x, x_0)$, given that the motivation of having a penalty on $d(x, x_0)$ is to try and ensure minimal changes, what about instead ensuring this by doing projected gradient onto the space of plausible data?\n\n- Again in section 3, is $d_y(f(x), f(x_0))$ intended to be high, or low?  Traditional understanding of counterfactual explanations in the literature would suggest $f(x) != f(x_0)$, but I can see the value in not caring about enforcing this to focus on driving down $H(y | x)$.  Could you spend some space touching upon this design decision here?\n\n- In Section 5.1, I'm curious why FIDO was chosen as a baseline?  If memory serves, the FIDO objective has additional constraints to try and ensure the B form a contiguous set, which your U-FIDO formulation (equations 6,7) does not admit.\n\n## Suggestions for improving the paper\n\n-  Algorithm 1 presents a minor nomenclature issue: the output $x_{clue}$ produced might not be an actual counterfactual in the sense of Wachter et al., in that no effort is made to orient the latent space edits (z) towards crossing a decision boundary for Y.\n\n- At the beginning of section 4, I find step 4 of the procedure is not clearly presented.  Step 4 suggests it’s used as a way to measure whether the discovered x_c are likely given the density of the model.  Is that so? If not, could you be more clear why evaluation of $\\tilde{x_c}$ by the VAEAC is helpful? \n\n- At the end of section 3, it would be helpful here to consider recent efforts to encourage counterfactual explanations to be confined to small contiguous regions (cf. Dabkowski and Gal 2017, Chang et al 2019).  This issue of potentially large, disparate, sparse signals was a flaw of original gradient based saliency maps, and would be well addressed here.\n\n- In section 5.1 where you discuss the criteria for evaluation of counterfactuals (*We would like counterfactuals to explain away as much uncertainty as possible while staying as close to the original inputs as possible*).  This is achieved, albeit indirectly, by other counterfactual generation methods (e.g Progressive Exaggeration https://openreview.net/forum?id=H1xFWgrFPS), which vary latent representations along a continuum of class output probabilities.  You could use their method as a comparator by selecting uncertain points and generating counterfactuals that move away from the decision boundary instead of towards it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WOq0JFuz8Lt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces CLUE -- a method to explain uncertainty estimates.  The method utilizes a VAE trained on the original data set to search effectively for low confidence instances.  The method utilizes a gradient based search through the latent space of the VAE.  The authors assess their approach on a variety of tabular data sets and MNIST.  They evaluate along change in uncertainty of the counterfactual as well as human evaluation.  They generally find improvements using their method over baselines.  Additionally, their method works much better in human evaluations.  \n\nComments + Questions\n\nSection 1:\n- The authors argue that CLUE can be used to complement feature importance techniques like LIME, Saliency Maps, etc. They point out that when the model is confident, you can use a feature importance technique.  When it is not confident, you can use CLUE.  However, the motivation behind why this dual approach is useful is not quite clear.  With feature attribution methods, the goal is to understand what features the model is locally relying on.  If the goal is to understand how the model behaves and the model is uncertain for a particular point, it could still be quite insightful to use feature attribution methods.  The authors could better argue why their complementary technique is useful and make explicitly clear why you wouldn't want to use feature importance methods for uncertainty data instances like they suggest. Right now, it is not so clear. \n- One minor point is that in the second paragraph, the authors motivate their method by saying CLUE can be useful to understand features contributing to uncertainty for instances underrepresented in the training data.  This motivation doesn't connect quite so clearly to the example in figure 1 --- we can see that the data instance is somewhat ambiguous.  Would the solution here to be to collect more ambiguous 6's? This motivating example could flow more clearly if it were a tabular instance because it would immediately connect to the motivating scenario given immediately before.\n\nSection 3:\n- In section 3, we see CLUE \"aims to find points in latent space which generate inputs similar to an original observation x0 but are assigned low uncertainty.\"  However, in the introduction it was stated that “CLUEs answer the question: What is the smallest change that could be made to an input, while keeping it in distribution, so that our model becomes more certain in its decision for said input?” These two claims seem slightly at odds.  Should the claim in the introduction be revised? \n\nSection 4:\n- The evaluation procedure claims using data generated through a VAE as the training data will reduce the possibility of clue exploiting adversarial vulnerabilities.  Recent work has pointed out adversarial vulnerabilities might be part of the training data for image data sets as nonrobust features [1] and in this way could be captured by the VAE making this procedure less effective.  The bulk of this work is focused on tabular datasets and MNIST where this is less likely an issue, so I am not too concerned.  However, scaling this procedure up to larger image data sets could produce issues.\n- A larger concern is that by only using data produced through a VAE, CLUE has a bit of an unfair advantage over methods like localized sensitivity which don't explicitly require the use of a VAE.  Meaning, the representational capacity of the VAE trained for CLUE is limited and in this way may not find certain diverse data points with low confidence.  A method that doesn't use a VAE might be able to find these points -- though search could be more challenging.  By forcing the set of images we're considering to be those produced by a VAE, this technique is shifting the playing field in favor of CLUE in what feels like a bit unfair way.  The evaluation should take into consideration that the requirement to train a VAE could be a disadvantage of CLUE but establish it is worthwhile nonetheless.  \n\nSection 5:\n- From table 1, CLUE's performance seems relatively well balanced with U-FIDO in many of the tabular tasks.  The authors point out at the bottom of page 6 that CLUE performs better on higher dimensional data sets.  However, this is only apparent in the MNIST data set.  If CLUE's merits lie with higher dimensional data sets like images, it could be better to provide more evaluation in these settings.  \n-  In the reference for appendix h.2 and the $log p_{gt}(\\cdot)$ test, it again feels like the assessment is a bit unfairly advantaged to CLUE; it feels very likely for CLUE to produce the best counterfactuals according to this metric because we assess data likelihood using a VAE.  At the same time, CLUE only generates counterfactuals on the VAE manifold.  \n- Maybe the authors could consider including some metric like nearest neighbor distance to the original training set for both clue and the baselines (where the baselines are run without being restricted to VAE generated data)? This could help us better understand the limitations imposed by using a VAE with CLUE. I think that an additional metric that disentangles the effect of the VAE within CLUE is needed here.\n- The human study results add a lot of merit to the CLUE approach.  It's clear from these that the counterfactuals produced by clue are much more human interpretable. \n- Though the right hand side of figure 10 helps use understand the limitations of using a VAE and is much appreciated, I still think an additional evaluation metric is needed for the 5.1 experiments.  An additional metric would help understand if the section 4 technique is giving CLUE an unfair advantage over the baselines.  \n\nOverall:\nI am convinced after reading the paper that the method exhibits useful performance in finding human meaningful uncertainty focused counterfactuals.  Further, there are a number of strong experiments in the paper -- I particularly liked the human evaluation and found this convincing. That said, there are a number of weaknesses that I'll mainly divide into two categories:\n(1) Introduction: per my comments in the introduction, I think this section could be significantly strengthened.  Most importantly here, the authors describe their method being used in an explanation workflow where uncertain instances are explained with clue and certain instances with something like LIME.  This thread, which seems like a key focus initially, is dropped for the rest of the paper.  It's currently unclear why this workflow makes sense and warrants much more justification.  Further, it could be worthwhile just to motivate CLUE as a method to explain uncertainty estimates in its own right because the connection with methods like LIME isn't explored in the rest of the paper.\n(2) Evaluation technique from section 4: Using data generated from a VAEAC as the set of legitimate images could give CLUE an unfair advantage over baselines because it reduces the potential effects caused by representation capacity of the CLUE VAE. It would make this section much stronger to include another metric to try and isolate these effects.  I would appreciate some author clarification here as well, in case I am misunderstanding something about the evaluation technique.\n\nOne final minor point is the authors claim their method works better than baselines in higher dimensional data.  However, they only evaluate using MNIST where VAE's tend to be very strong.  To fully substantiate this claim, it could be worthwhile to consider a few slightly more challenging data sets for VAEs (street view house numbers, celeba, etc). \n\nMy sentiments are currently leaning towards reject mainly due to the motivation and experimental issues described in (1) and (2).  If the authors could remedy these concerns, I'd be inclined to raise my score because I think their are a number of potential valuable contributions in the work.\n\nOne related method that isn't discussed is https://arxiv.org/abs/2002.10248.  The authors generate instances at certain levels of prediction confidence --- though different it could be good to bring up.\n\n[1] https://arxiv.org/abs/1905.02175 \n\n---- update ----\nIn response to the author's comments and extensions, I've raised my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an uncertainty quantification method that is conceptually interesting and practical. All reviewers are in consensus regarding the quality and significance of this manuscript. 
Decision: Accept (Oral)