Explanation Uncertainty with Decision Boundary Awareness
Authors: Davin Hill
Aria Masoomi
Sandesh Ghimire
Max Torop
Jennifer Dy
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainability
Interpretability
XAI
Feature Importance
Explanation Uncertainty
Reliability

Abstract: Post-hoc explanation methods have become increasingly depended upon for understanding black-box classifiers in high-stakes applications, precipitating a need for reliable explanations. While numerous explanation methods have been proposed, recent works have shown that many existing methods can be inconsistent or unstable. In addition, high-performing classifiers are often highly nonlinear and can exhibit complex behavior around the decision boundary, leading to brittle or misleading local explanations. Therefore, there is an impending need to quantify the uncertainty of such explanation methods in order to understand when explanations are trustworthy. We introduce a novel uncertainty quantification method parameterized by a Gaussian Process model, which combines the uncertainty approximation of existing methods with a novel geodesic-based similarity which captures the complexity of the target black-box decision boundary. The proposed framework is highly flexible—it can be used with any black-box classifier and feature attribution method to amortize uncertainty estimates for explanations. We show theoretically that our proposed geodesic-based kernel similarity increases with the complexity of the decision boundary. Empirical results on multiple tabular and image datasets show that our decision boundary-aware uncertainty estimate improves understanding of explanations as compared to existing methods

Under review as a conference paper at ICLR 2023
EXPLANATION UNCERTAINTY WITH DECISION BOUNDARY AWARENESS
Anonymous authors Paper under double-bling review
ABSTRACT
Post-hoc explanation methods have become increasingly depended upon for derstanding black-box classifiers in high-stakes applications, precipitating a need for reliable explanations. While numerous explanation methods have been pro- posed, recent works have shown that many existing methods can be inconsistent unstable. In addition, high-performing classifiers can exhibit complex behavior around the decision boundary, leading to brittle or misleading local explanations. Therefore, there is an impending need to quantify the uncertainty of such explanation methods in order to understand when expla- nations are trustworthy. We introduce novel uncertainty quantification method parameterized by Gaussian Process model, which combines the uncertainty ap- proximation existing methods with a novel geodesie-based similarity which captures the complexity of the target black-box decision boundary. The proposed framework is highly flexible; it can used with any black- -ook classifier and fea- ture attribution method to amortize uncertainty estimates for explanations. We show theoretically that our proposed geodesic-based kernel similarity increases with the complexity of the decision boundary. Empirical results multiple tab- ular and image datasets show that our decision boundary-aware uncertainty esti- mate improves understanding of explanations as compared existing methods.
often highly nonlinear and
INTRODUCTION
Machine learning models are becoming increasingly prevalent in wide variety of industries and ap- plications. In many such applications, the best performing model is opaque; post-hoc explainability methods are one of the crucial tools by which we understand and diagnose the model's predictions. Recently, many explainability methods, termed explainers, have been introduced in the category of local feature attribution methods. That is, methods that return a real-valued score for each feature of a given data sample, representing the feature's relative importance with respect to the sample prediction. These explanations are local in that each data sample may have a different explana- tion. Using local feature attribution methods therefore helps users better understand nonlinear and complex black-box models, since these models are not limited to using the same decision rules throughout the data distribution.
Recent works have shown that existing explainers can be inconsistent or unstable. For example, given similar samples, explainers might provide different explanations (Alvarez-Melis & Jaakkola 2018 Slack et al. 2020)). When working in high-stakes applications, it is imperative to provide the user with an understanding of whether an explanation is reliable, potentially problematic, or even misleading. A way to guide users regarding an explainer's reliability is to provide corresponding uncertainty quantification estimates.
One can consider explainers function approximators; such, standard techniques for quantifying the uncertainty of estimators can be utilized to quantify the uncertainty explainers. This is the strategy utilized by existing methods for producing uncertainty estimates of explainers (Slack al.) 2021 Schwab & Karlen 2019). However, we observe that for explainers, this is not sufficient; because in addition to uncertainty due to the function approximation of explainers, explainers also have to deal with the uncertainty due to the complexity of the decision boundary (DB) the black- box model in the local region being explained.

---

Under review as a conference paper at ICLR 2023
Physical Activity (PA)
Predict: Negative
Pedien ponten
Patient A Feeture Importance
Patient B Feature Importance
BMI
Predict: Positive
PA
Body Mass Index (BMI)
Figure 1: Illustrative example of how similar data samples can result in very different feature impor- tance scores given black-box model with nonlinear decision boundary. Here, two similar patients with similar predictions are given opposing feature importance scores which could result in mis- guided recommendations. We define similarity based on the geometry of the decision boundary between any two given samples (red line). While the two patients are close together in the Euclidean sense, they are dissimilar under the proposed WEG kernel similarity. Using GPEC would return high uncertainty measure for the explanations, which would flag the results for further investigation.
Consider the following example: are using a prediction model for medical diagnosis using two features, level of physical activity and body mass index (BMI) (Fig. D. In order to understand the prediction and give actionable recommendations to the patient, we use a feature attribution method to evaluate the relative importance of each feature. Because of the nonlinearity of the prediction model, patients and B show very similar symptoms, but are given very different recommendations, Note that while this issue is related to the notion of explainer uncertainty. measures of uncertainty that only consider the explainer would not capture this phenomenon. This suggests that any notion of uncertainty is incomplete without capturing information related to the local behavior of the model. Therefore, the ability to quantify uncertainty for DB-related explanation instability is desirable.
We approach this problem from the perspective of similarity: given two samples and their respective explanations, how closely related should the explanations be? From the previous intuition, we define this similarity based Specifically, we propose a novel geodesic- Exponential Geodesic (WEG) kemel. The WEG kernel encodes expectation that two samples close in Euclidean space may not actually be similar if the DB within a local neighborhood of the samples is highly complex.
geometric perspective of the DB complexity between these two points. kernel similarity metric, which we call the Weighted
Using our similarity formulation, we propose the Gaussian Process Explanation UnCertainty (GPEC) framework, which is instance-wise, model-agnostic, and explainer-agnostic method to quantify the uncertainty of explanations. The proposed notion of uncertainty is complementary to existing quantification methods. Existing methods primarily estimate the uncertainty related to the choice in model parameters and fitting the explainer, which we call function approximation uncer- tainry, and does not capture uncertainty related to the DB. GPEC can combine the DB-based uncer- tainty with function approximation uncertainty derived from any local feature attribution method.
In summary, we make the following contributions:
We introduce a geometric perspective on capturing explanation uncertainty and define novel geodesic-bart based similarity between explanations. We prove theoretically that the proposed simi- larity captures the complexity of the decision boundary from a given black-box classifier.
We propose a novel Gaussian Process-based framework that combines A) uncertainty from deci- sion boundary complexity and B) explainer-specin uncertainty to generate uncertainty estimates for any given feature attribution method and black box model.
Empirical results show GPEC uncertainty improves understanding of feature attribution methods.
RELATED WORKS
Explanation Methods. A wide variety of methods have been proposed for the purpose of improv- ing transparency for pre-trained black-box prediction models (Guidotti et al., 2018 Barredo Arrietal et al. 2020). Within this category of post-hoc methods, many methods focus local explanations, that is, explaining individual predictions rather than the entire model. Some of these methods gen- erate explanations through local feature selection (Chen al. 2018 Masoomi et al. 2020). In this
2

---

Under review as a conference paper at ICLR 2023
Prediction Model
Explanation Model
Explainer
Explanations
GPEC Explanation Uncertainty
Black Box
Dectsion Boundary
Samples
Geodasic Similarity
Noivy Explanations
Uncertainty
Prediction
Figure 2: Schematic for the GPEC uncertainty estimation process. GPEC can be used in con- junction with black-box classifier and explainer to derive estimate of explanation uncertainty. GPEC takes samples from classifier's decision boundary plus (possibly noisy) explanations from the explainer and fits a Gaussian Progress Regression model with Weighted Exponential Geodesic Ker- nel. The variance of the predictive distribution combines uncertainty from the black-box classifier complexity and the approximation uncertainty from explainer.
work, we focus primarily on feature altribution methods, which return real-valued score for each feature in the sample. For example, LIME (Ribeiro et al. al.] (2016) trains a local linear regression model to approximate the black-box -bxx model. Lundberg & Lee (2017) generalizes LIME and five other fea- ture attribution methods using the SHAP framework, which fulfill a number of desirable axioms. While some methods such LIME and its SHAP variant, KernelSHAP,
model-agnostic, others are designed for specific model architectures, such as neural networks (Bach et al. 2015 Shrikumar let al. 2017: Sundararajan et al.. [017 Erion et al.) (02T), tree ensembles (Lundberg et al.) (020), or Bayesian neural networks (Bykov et al.) 2020). taking advantage of those specific architectures. Another class of method involves the training of separate surrogate models explain the black-box model (Dabkowski & Gal
et al. 2018 Schwab & Karlen 2019;Gur et al. [0088] Jethani al. 2022). Once trained, surrogate-bar based methods are typically very fast; explanation generation generally only requires single inference step from the surrogate model.
Explanation Uncertainty. One option for improving the trustworthiness of explainers to quantify the associated explanation uncertainty. Bootstrap resampling techniques have been proposed as way to estimate uncertainty from surrogate-base based explainers (Schwab Karlen 2019 [Shuuzz al.) 2022). Guo et al. (2018) 8) also proposes surrogate explainer parameterized with Bayesian mixture model. Alternatively, Bykov al.] (2020) and Patro et al. (2019) introduce methods for explaining Bayesian neural networks, which can be transferred to non- non-Bayesian neural networks. Covert & Lee (2021) derives an unbiased version of KerneISHAP and investigates an efficient way of estimating its uncertainty. ZZhng al.) (2019) categorizes different sources of variance in LIME estimates. Several methods also investigate LIME and KernelSHAP in a Bayesian context; for example calculating a posterior over attributions (Slack al. (02II, investigating the use of priors for explanations (Zhao et al. 2021 or using active learning to improve sampling (Saini & Prasad 2022).
However, all existing methods for quantifying explanation uncertainty only consider the uncertainty of the explainer as a function approximator. This work introduces an additional notion of uncertainty for explainers that takes into account the uncertainty of the explainer due the DB.
3 UNCERTAINTY QUANTIFICATION FOR BLACK-BOR EXPLAINERS
We now outline the GPEC framework, which we parameterize with a Gaussian Process (GP) regres- sion model. We define a vector-valued GP which is trained on data samples as input and explana- tions as labels. More concretely, consider data sample z E X C Id that want to explain in the context of a black-box prediction model F X + [0, 1]. in which the output corresponds to the probability for the positive class. For simplicity,
consider the binary classifier case; extend We apply any given feature attribution method E X - R E(z).
to the multiclass case in Section where is the dimension of the explanations
Let us draw samples ,,....,
......,.py. IM from the data distribution and generate their respective explanations Cl eM- We assume that each explanation en is generated from unobserved latent function &
3

---

Under review as a conference paper at ICLR 2023
plus some independent Gaussian noise 7mm-
Cm {zmm + Mm s.t.
E {zm) GP(O, kx2,2')
~ ~0 mm!
(1)
Decision Boundary-Aware Uncertainty
Function Approximation Uncertainty
where k(.-)) is the specified kernel function and Tm O. We decompose each explanation into Two components, m)) and 7/m which represent two separate sources of uncertainty: 1) decision boundary- -ware uncertainty which we capture using the kernel similarity, and 2) a functioo approx- imation uncertainty from the explainer. After specifying (z and 7m, we can combine the two sources by calculating the posterior predictive distribution for the GP model; we take the 95% con- fidence interval of this distribution to be the GPEC uncertainty estimate. Due to space constraints, we provide an overview GP regression in App. B.2
Function Approximation Uncertainty. The 7/m. component of Eq. represents the uncertainty stemming from explainer specification. For example, this uncertainty can be captured from in- creased variance from undersampling (perturbation-basee explainers) from the increased boot- strap resampling variance from model misspecification (surrogate- basee explainers).
We can therefore define based on the chosen explainer. Explainers that include some estimate of uncertainty (c.g. BayesLIME, BayesSHAP. CXPlain) can be directly used to estimate т. For example, CXPlain returns distribution of feature importance values for sample Fm; the variance of this distribution can be directly used as the variance of 7mm- For other stochastic explanation methods that do not explicitly estimate uncertainty, we can estimate т empirically by resampling explanations for the same data sample:
K
K
im
(E.(am) - E(am))? s.t. Σ
E(xm)
E, (2m)
K
where each (3m) is sampled explanation for Zm. Alternatively, for deterministic explanation methods we can omit the noise and assume that the generated explanations are noiseless.
Decision Boundary-Aware Uncertainty. In contrast, the ((mm) component of Eq. draws possible functions from the GP prior that could have generated the observed explanations, with the function space defined by the choice of kernel. Intuitively, the kernel encodes a priori definition of how similar two explanations should be given the similarity of the inputs. In other words, how much information do expect given point provide for nearby point xe' with respect to their explanations? We use this kernel specification to define source of uncertainty dependent on the behavior of nearby explanations. the complexity of the DB in local neighborhood of the samples; this is detailed in Section 4 For any given kernel, we can interpret the distribution of possible functions E {(xm) as an estimate of uncertainty based on the set of previously observed sampled explanations.
particular, consider a novel kernel formulation that reflects
4 WEIGHTED EXPONENTIAL GEODESIC KERNEL
Intuitively, the GP kernel encodes the assumption that each individual explanation gives some infor- mation about the explanations around it; the choice of kernel defines the neighborhood and magni- tude of this shared information. In the GP framework introduced in Section 3 the kernels define the relationship or similarity between two explanations E(x) and E(x') based soley on some function of their inputs x and x'. For example, stationary kernels assign the same similarity for any two inputs regardless of where in the data manifold they are located. Instead, we want to encode the assumption that when the DB is complex, knowing an explanation E(x) gives limited information about other nearby explanations (see Fig.
4.1 GEOMETRY AND GEODESICS OF THE DECISION BOUNDARY
We want to relate kernel similarity to the behavior of the black-box model: specifically, the com- plexity or smoothness of the DB. Given any two points on the DB, the relative complexity of the boundary segment between them can be approximated by the segment length. The simplest form that the DB can take is linear boundary connecting the points; this is exactly the minimum distance between the points. As the complexity of the DB grows, there is a general corresponding increase in segment length.
4

---

Under review as a conference paper at ICLR 2023
X (-3,3)
X (-0.5,3)
X (0.5,0)
(3,0)
RBF Kernel
IAMA
(a)s(a) Decision Doundary
MMAM
WEG
Figure 3: Comparison of RBF and WEG kernel similarity neighborhoods. The gray highlighted region (o.s(z) {x' k(x, x') 2 0.8} indicate points where the kernel similarity is greater than 0.8 with respect to the red point æ. The black line is the decision boundary for the classifier
fz x2) = 2 cos 20) I2, where 222 2 O indicates class 1, and ((11 < 0 indicates
class O. We see that for GPEC, (o.s(1) shrinks as moves closer to segments of the decision boundary which are more complex.
Concretely, we define the DB as a (d - 1)-dimensional Riemannian manifold embedded in Rd. Given the binary classifier F, we define . {x TRd F(x) = 2) representing the DH of F. Given this interpretation, we can define distances along the DB as geodesic distances in M..
min S [I(()](at Vm, m' E
dgeo (m, m²) =min
The mapping [] + M, defined such that 7(0) = m and 7(1) = m', is a parametric repre- sentation of 1-dimensional curve on M. We can adapt geodesic distance in our kernel selection through the exponential geodesic (EG) kernel (Feragen et 20155), which is generalization of the Radial Basis Function (RBF) kernel substituting & distance with geodesic distance:
kgo(E, x²) exp( -dggeo x²))
KRBF(E, x') cxp(- -||||= =x111)
The EG kernel has been previously investigated in the context of Riemannian manifolds (Feragen) eet al. 2015; Feragen & Hauberg (016). In particular, while prior work shows that the EG kernel fails to be positive definite for all values À in non-Euclidean space, there exists large intervals of A > 0 for the EG kernel to be positive definite. Appropriate values can be selected through grid search and cross validation; we assume that a valid value of À has been selected.
Note that the manifold are interested in is the DB; applying the EG kernel on the data manifold (i.e. directly using Kg in the GPEC formulation) would not capture model complexity. In addition, naively using the EG kernel on the DB manifold would not capture the local complexity with respect to given explanation; the similarity would be invariant to observed explanations. We therefore need relate the geodesic distances on M to samples in the data space X.
4.2 WEIGHTING THE DECISION BOUNDARY
Consider probability distribution (M) with its support defined over M. We weight p(M) ac- cording to the lz distance between M and a fixed data sample x:
q(M [z, p) exp[-plla- M M1(
We evaluate the kernel function k(xx, a²) by taking the expected value over the weighted distribution.
(W = / expp Addeo (m, m')] Q[m|z p) q(ml (e', dmdd''
(6)
Without loss of generality, we assume that the decision rule for the classifier is set he
5

---

Under review as a conference paper at ICLR 2023
Our formulation is example of marginalized kernel Tsuda et al. 2002): kernel defined on two observed samples z, x and taking the expected value over some hidden variable. Given that the underlying EG kernel is positive definite, it follows that the WEG kernel forms valid kernel.
To evaluate the WEG kernll, we thcoretically investigate two properties of the kernel. Theorem shows that the WEG kernel is extension of the EG kernel for data samples not directly on the DB.
Theorem 1. Given two points x' M, then limx- EG (z, x²) (gG
Proof details are in App- C.1 Intuitively, as we increase X, the manifold distribution closest to the points 2; x' becomes weighted increasingly heavily. At the limit, the weighting becomes concen- trated entirely on the points z, x' themselves, which recovers the EG kernel. Therefore we see that the WEG kernel is simply weighting of the EG kernel, which is controlled by A.
Theorem 2 establishes the inverse relationship between DB complexity and WEG kernel similarity. Given black-box model with piecewise linear DB, show that this DB represents local maximum with respect to WEG kernel similarity; i.c. as we perturb the DB to be nonlinear, kernel similarity decreases. We first define perturbations on DB, Note that int(S) indicates the interior of set S, i represents the function f restricted to S, and id indicates the identity mapping.
Definition 1. Let (Ua)aE) be charts of an atlas for manifold P IR where 7 is of indices. Let P and P be differentiable manifolds embedded in Rd where is a Piecewise Linear (PL) manifold. Let R chart if R satisfies the following two conditions. 1) There exists compact subset K; C U; s.t. R Pp ())) = id p()n(KK)) and (K,)) * id (K,)" 2) There exists a linear homeomorphism between an open subset U, U with R which contains K₃.
+ be a diffeomorphism. We say is a perturbation of on the 7th
Theorem 2. Let P be (d-I)-dimenssinn PL manifold embedded in R.d. Lei P be perturbation of P and define x(axx²) and k(x,x') as the WEG kernel defined on and P respectively. Then ((xxxx) < k(x, x') Vxx x! E Bd.
Proof details are in App. C.2 Theorem implies that, for any two fixed points x, x, their kernel similarity kWEG EG xx x') decreases as the black-box DB complexity increases. Within GPEC, the explanations for 2; x' become less informative for other nearby explanations and induce higher explanation uncertainty estimate.
To improve the interpretation of the WEG kernel, apply normalization scale the similarity values to be between [0, 1]. We construct the normalized kernel ki BweG as follows:
kw k
kiw kWEG(E / Fo FC EG((',
4.3 WEG KERNEL APPROXIMATION
In practice, the integral in Eq. o is intractable; we approximate the expected value using Monte Carlo (MC) sampling with K samples:
K
K
kw kWEG(I, Ze Zeme K2
m3) exxeex(ppppxxx m,,1--mm;,11-11 +1 -mm (8)
==1==1
Zm, Znd are the normalization constants for q(M Jr, p) and q( qM'lly, p), respectively. We can simi-
larly estimate these values using MC sampling:
K Σ t=1
7m =
exppl eepp-=pla= m ml12p((m dm R
expl_pplaz 1m
m.13
K
GPEC Implementation Details. Due to space constraints, the full GPEC training algorithm is detailed in Alg. [ in App. D To use the GPEC framework, we first train the GP regression model with a set of training samples. Once the GP is trained, we can derive uncertainty estimates for test samples from the 95% confidence interval of the posterior predictive distribution. The WEG kernel requires additional setup time to use the GP. We first calculate the EG kernel matrix by sampling the black-box DB, then weigh each element of the matrix per Eq. 5 The estimation of

---

Under review as a conference paper at ICLR 2023
GPEC- WEG
GPEC- - RBF
BayesSHAP
BayesLIME
CXPlain
Houu Hours pur soek
350
Consus
300
022
500
250
250
2:20
500
2.00
2.00
14
0.0
0.00
355
20
1.16
20
1000
10
Age
Age
Age
Age
Age
Census
Education hum 1
3:30
100
c00
0:50
8:00
3:40
1:00
1.0
-200
c000
19.00
0.00
2.00
Age
Age
Age
Age
Age
German Credit
Crudk Agu (Daritts)
3.50
c.00
3:00
1.16
2000
230
250
$.14
15 2.
00
Cede Ane
3.50
1.00
150
53
100
150
100
150
Credit Amoun (00)
Credht Amount (00)
Credit Amount (100)
Credit Amount 1001
Credit Credit.Amoi Amount (100)
Figure 4: Visualization of estimated uncertainty of explanations for different models and competing methods. The heatmap represents level of uncertainty for grid of explanations for the feature on the x-axis; darker areas represent higher uncertainty. Red points represent training samples for GPEC and CXPlain as well the reference samples for BayesSHAP and BayesL The black line represents the black-box DB. The heatmap shows that GPEC-WEG is the only method that captures uncertainty from the DB, due to the WEG kernel similarity formulation.
a classifier's DB is an ongoing area of research; number of methods have been proposed in the literature (e.g. Yan & Xu (2008); Karimi et al.] (2019)). Once GPEC is trained, training cost is amortized during inference; estimating uncertainty for test samples using a GP generally has time complexity of O(n"), which can be reduced to O(n²) using BBMM (Gardner et al. 2018 and further with variational methods (e.g.. Hensman et al. (2015).
EXPERIMENTS
We evaluate how well GPEC captures 1) DB-aware uncertainty and 2) functional approximation uncertainty on a variety of datasets and classifiers. In section 5.2 we compare the DB-aware un- certainty and approximation uncertainty components of GPEC. Section 5.3 evaluates how GPEC captures DB complexity. Section 5.4 evaluates how well GPEC combines sources of uncertainty. Due to space constraints we have additional results in the appendix, including sensitivity analysis for GPEC parameters (F.) execution time comparison F.T. and additional experiments on combining uncertainty (F.3). All experiments were run on an internal cluster using AMD EPYC 7302 16-Core processors, and all source code will be made public.
5.1 EXPERIMENTAL SETUP
Datasets and Models. Experiments are performed on three tabular datasets (Census, Online Shop- pers (Sakar et al. 2019). German Credit) from the UCI data repository (Dua & Graffi 2017) and
two image datasets (MNIST (LeCun & Cortes 2010) and fashion-MNiST (I-MNIST) (Xiao al.
2017h. GPEC can be used with any black-box model; for our experiments we use XGBoost (Chen & Guestrin 2016) with log-loss for the tabular datasets and 4-layer Multi-Layered Perception (MLP) model for the image datasets. Additional dataset details are outlined in App. E.T
Implementation Details. For comparison purposes, we consider GPEC using two different kernels: GPEC-WEG (WEG kernel) and GPEC-RBF (RBF Kernel). Unless otherwise stated, we choose
A = 1.0 and = 0.1 (see App. F.5 for experiments on parameter sensitivity). For the Uncertainty Visualization (Sec. 5.2) and Regularization Test (Sec. 5.3) we use GPEC with the KernelSHAP
explainer. We train the GP parameterizing GPEC with BBMM (Gardner et al., 2018). Samples
from the DB are drawn using DBPS Yan & Xu 2008) for tabular datasets and DeepDIG (Karimi
ct al. 2019) for image datasets. Geodesic distances are estimated using ISOMAP Tenenbaum et al. (2000) Additional implementation details available in App. D.
7

---

Under review as a conference paper at ICLR 2023
German Credit
Census
Ooll Sitere
GPEC-WEG
WEG
GPEC-WEG
GPEC-WEG
A)
00
0
-1:0
210
0/0
100
30
CI Aemuatt 2001
Credd Arrount (00)
Ape
Cree Aee Norte
GPEC-WEG
U
GPEC-
GPEC-WEG BayesSHAp
GPEC-WEG
00
B]
000
699
Arouul (10)
Credkt Amount (20)
Age
Age
Oval (Mavae)
Cwtt (60 (Mn)te))
Educomton-- Nam
0.40
0
0
100
0.00
12
000
53
100
150
50
130
150
22
30
90
72
42
30
73
Figure 5: Evaluation of GPEC's ability to combine DB-aware uncertainty and functional approxi- mation uncertainty. Row (A) visualizes the GPEC-WEC uncertainty estimate using only DB-aware uncertainty. Row (B) combines the DB-aware uncertainty (A) with the functional approxima- tion uncertainty from the two explainers: BayesSHAP and SSV. Row (C) visualizes the change in uncertainty estimate between (A) and (B).
Competing Methods. We compare GPEC to three competing explanation uncertainty estimation methods. BayesSHAP and BayesLIME (Slack al.] [021] are extensions of KerneISHAP and LIME, respectively, that fit Bayesian linear regression models to perturbed data samples. After fitting, 95% credible intervals can be estimated by sampling the posterior distribution of the feature attributions. CXPlain (Schwab & Karlen (019) trains separate explanation model using a causal- based objective, and applies a bootstrap resampling approach to estimate explanation uncertainty. We report the 95% confidence interval from the set of bootstrapped explanations. Unless otherwise stated, we use the default settings the provided implementation for all three of these methods.
5.2 UNCERTAINTY VISUALIZATION
In order to visualize explanation uncertainty, we train XGBoost binary classifiers using two selected features, which we take to be the black-box model. Both GPEC and competing methods are used to quantify the uncertainty for the feature attributions of the test samples: a grid of 10,000 samples over the data domain. Uncertainty estimates for the x-axis feature for the y-axis variable are included in App. F.4). where darker values in the heatmap indicate higher uncertainty. GPEC-WEG, GPEC-RBF, and CXPlain require training samples for their amortized uncertainty estimates; these samples, also used as the reference distribution for BayesSHAP and BayesLIME, are plotted in red. The DB is plotted as the black line. We ablate the function approx- imation uncertainty component of GPEC-WER and GPEC-RBF in order to evaluate the effects of their respective kernels. Comparing GPEC-WEG and GPEC-RBF shows that using the WEG ker- nel attributes higher uncertainty to samples near nonlinearities in the DB. In contrast, GPEC-RBF provides uncertainty estimates that relate only to the training samples; test sample uncertainty is pro- portional to distance from the training samples. The competing methods BayesSHAP, LMY, and CXPlain results relatively uniform uncertainty estimates over the test samples. CXPlain shows areas of higher uncertainty for Census, however the magnitude of these estimates are small. The uncertainty estimates produced by these competing methods are unable to capture the properties of the black-box model.
plotted in Figure 4 (results
5.3 REGULARIZATION TEST
In this experiment we compare the average uncertainty of explanations as the model is increasingly regularized in order to assess the impact of restricting model complexity. For XGBoost models, we vary the parameter 7, which penalizes the number of leaves in the regression tree functions (Eq. 2 in Chen & Guestrin (2016)). For neural networks, we use two types of regularization. First, we add an lg penalty to the weights; this penalty increases as the parameter À increases. Second, we change the
8

---

Under review as a conference paper at ICLR 2023
Dotaset
Census
Online Shoppers
German Credit
Regularization
7
Magnitude
5
10
5
10
GPEC-WEG
1,573
1,177
1.158
0.209
0.123
0.092
2,065
1.747
0.250
GPEC-RBF
0.493
0.472
0.466
1.740
1.731
1.734
0.153
0.081
0.073
BayesSHAP
0.037
0.037
0.037
0.031
0.031
0.031
0.019
0.019
0.018
BayesLIME
0.097
0.096
0.095
0.098
0.097
0.093
0.085
0.066
0.045
CXPlain
0.064
0.064
0.069
0.004
0.003
0.006
1.7e
1.1e
4.36
Dutaset
MNIST
Fashion MNIST
Regudarizaionn
Soliplux 8
Soliplus /
Magnitude
10c
0.5
0.25
1c
10c
0.5
0.25
GPEC-WEG
0.236
0.157
0.078
0.087
0.073
0.056
0.378
0.187
0.063
0.112
0.075
0.061
GPEC-RBF
0.439
0.301
0.232
0.226
0.232
0.424
0.247
0.261
0.250
BayesSHAP
0.025
0.016
0.008
0.013
0.010
0.014
0.007
0.010
0.009
BayesLIME
2,452
1.573
0.737
0.868
0.866
0.721
2,605
1.364
0.666
1.178
0.861
0.779
CXPlain
0.1e
5.0e
8.6
5,30
8.0e
5.4e
7.20
4.8e
6.2
9.0c
6.60
9.6e
Table 1: Average explanation uncertainty estimates all features for given classifier with varying levels of regularization. Higher regularization (increasing left to right) generally results in smoother and simpler classifiers.
ReLU activation functions Softplus; smooth approximation of ReLU with smoothness inversely proportional to a parameter B, which we vary (Dombrowski et al. 2019). We observe in Table Ithat the average uncertainty estimate generated by GPEC-WEG decreases as regularization increases, suggesting that the uncertainty estimates reflect the overall complexity of the underlying black- box model. For the tabular datasets, the estimates for BayesSHAP, BaYesLIME, and CXPlain stay relatively flat. Interestingly, the estimates from these methods decrease for the image datasets. We hypothesize that the regularization for the neural network model also increases overall stability of the explanations. GPEC can capture both the uncertainty from WEG kernel and also the estimated uncertainty from the function approximation for the explainer, which we demonstrate in Section 5.4
5.4 COMBINING APPROXIMATION UNCERTAINTY
Using the WEG kernel with noisy explanation labels enables GPEC to incorporate uncertainty es- timates from the explainer. Explainers such BayesSHAP provide estimate of uncertainty for their explanations, which can be used as fixed noise in the Gaussian likelihood for GPEC. For ex- plainers with no such estimate, and take the explanation variance as an estimate of uncertainty. Here, we compare GPEC uncer- tainty results before and after applying explanation noise estimates. We combine GPEC with two different explainers: BayesSHAP and Shapley Sampling Values (SSV) (Strumbelj & Kononenko, 2013). The two selected explainers are different methods of approximating SHAP values, however the former has an inherent method for quantifying its uncertainty while the latter does not. In Figure row (A) we plot the heatmap for uncertainty estimates using GPEC using only the WEG Kernel and no added noise from the explainers. In row (B) we add the additional uncertainty estimates from BayesSHAP and SSV. The difference in quantified uncertainty is visualized in the row (C). We observe that combining the uncertainty estimates from BayesSHAP and SSV in the GPEC Gaussian Process formulation increases the overall uncertainty estimate.
can empirically resample explanations for the same test sample
LIMITATIONS AND CONCLUSION
Generating uncertainty estimates for feature attribution explanations are essential for building reli- able explanations. We introduce a novel GP-based approach that can be used with any black-box classifier and feature attribution method. GPEC generates uncertainty estimates for explanations that capture the complexity of the black-box model. Experiments show that capturing this uncertainty improves understanding of the explanations and the black-box model itself. Regarding limitations, GPEC relies on DB estimation methods which is ongoing area of research. Due to the time com- plexity of DB estimation, this can result in a tradeoff between computation time and approximation accuracy or sample bias. However, the effects DB sampling time minimized during inference as the DB only needs to be sampled during training. Additionally, in its current implementation GPEC is limited to black-box classifiers; leave the extension regression future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, 0as/18806080499 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Roben Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10, 2015.
Naman Bansal, Chirag Agarwal, and Anh Nguyen. SAM: The Sensitivity of Attribution Methods to Hyperparameters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8673-8683, 2020.
Alejandro Barredo Arricta, Natalia Díaz- Rodriguzz, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AL. Information Fusion, 58:82-115, 2020. ISSN 1566-2535. doi: (0.10166).ifffus220099. 12.011.
Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and Aggregating Feature-based Model Explanations. arXiv:2005.000 001:22005500631 [cs, stat], May 2020.
Dmitri Burago, Yuri Burago, and Sergei Ivanov. course in metric geometry, volume 33. American Mathematical Society, 2001.
Kirill Bykov, Marina M-C Höhne, Klaus-Robert Müller, Shinichi Nakajima, and Marius Kloft. How much can I trust You?- quunttuatifyiing uncertainties in explaining neural networks. arXiv preprint 2XXi:20066099000 2020.
George Casella and Roger Berger. Statistical Inference. Duxbury Resource Center, June 2001. ISBN 0-534-24312-6.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theoretiee perspective model interpretation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 883-892. PMLR, July 2018
Tianqi Chen and Carlos Guestrin. XGBoost: scalable tree boosting system. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 11. pp. 785-794, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978- 978-1-4503-4232-2 doi: 10. 00.11455993997222999755
Ian Covert and Su-In Lee. Improving kernelSHAP: Practical shapley value estimation using linear regression. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of the 24th Interna- tional Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3457-3465. PMLR, April 2021.
Ian Covert, Scott Lundberg. and Su-In Lec. Feature Removal Is Unifying Principle for Model Explanation Methods. 0rXiv:2011.0362. [cs, stat], November 2020.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Knn-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 170.:11700080000 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httt:://arcciv/ ics. uc. edd/ml
10

---

Under review as a conference paper at ICLR 2023
Gabriel Erion, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, and Su-In Lee. Im- proving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence, 3(7):620-631, July 2021. ISSN 2522-5839. doi: 1110388/42255--02--0044444
Andre Esteva, Alexandre Robicquet, Bharath Ramsundar, Volodymyr Kuleshov, Mark DePristo, Katherine Chou, Claire Cui, Greg Corrado, Sebastian Thrun, and Jeff Dean. A guide to deep learning in healthcare. Nature medicine, 25(1):24-29, 2019.
Aasa Feragen and Søren Hauberg. Open Problem: Kernel methods on manifolds and metric spaces. What is the probability of positive definite geodesic exponential kernel? In Vitaly Feldman, Alexander Rakhlin, and Ohad Shamir (eds.), 29th Annual Conference on Learning Theory, vol- ume 49 of Proceedings of Machine Learning Research, pp. 1647-1650, Columbia University, New York, New York, USA, June 2016. PMLR.
Aasa Feragen, François Lauze, and Søren Hauberg. Geodesic exponential kernels: When curvature and linearity conflict. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3032-3042, 2015. doi: 10.1 10.1109CVPR.201552289...
Jacob Gardner, Geoff Pleiss, Kilian Q Weinberger, David Bindel. and Andrew G Wil- son. Gpytorch: Blackhox matrix-matrix gaussian process inference with gpu accelera- tion. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018 URL https:: htttps::/pprceeedingg new.eeedddinns.................. http://wwwcccooo.eeeee.../eeaaap
pcppps///ccpppprr//////eee 2 TT52e177136661003300007777722222 Papp:paa7....7222eeaaaa.p.pp Paper. pdf
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of Neural Networks Is Fragile. Proceedings of the AAAI Conference Artificial Intelligence, (3(00)):681-33)) July 2019.
ISSN 2374-3468. doi: 10. wttp///aaivv3301..33333
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. Acm Computing Surveys, 51 (5), August 2018. ISSN 0360-0300. doi: 10.1145/3236009.
Wenbo Guo, Sui Huang, Yunzhe Tao, Xinyu Xing, and Lin Lin. Explaining deep learning models a bayesian con-parametric approach. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
James Hensman, Alexander Matthews, and Zoubin Ghahramani. Scalable Variational Gaussian Process Classification. In Guy Lebanon and S. V. N. Vishwanathan (eds.). Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Pro- ceedings of Machine Learning Research, pp. httptttsss:///ppeeeeeeeee ngs mlr presss/ppe/////eeeeoooo w38/hensman 1.mmml
San Diego, California, USA, 09 12 May
2015. PMLR.
Cheng- Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim, Sanjiv Ku- mar, and Cho-Jui Hsieh. Evaluations and Methods for Explanation through Robustness Analysis. arXiv:2006.0000 00440 [cs, stat), April 2021.
Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. FastSHAP: Real-time shapley value estimation. In International Conference Learning Representattinss 2022.
Hamid Karimi, Tyler Derr, and Jiliang Tang. Characterizing the decision boundary of deep neural networks. arXiv preprint arXiv: /9/2. 1460, 2019.
Zulqarnain Khan, Aria Masoomi, Davin Hill, and Jennifer Dy. Analyzing the effects of classifier lipschitzness explainers. arXiv preprint arXiv:2206. 12481, 2022.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un)reliabilliy of Saliency Methods, pp. 267- 280. Springer International Publishing, Cham, 2019. ISBN 978 678-3-030-28954 doi: 10. 108-3-030-28544 14. URL https: /ts:://doo. //w.i org/
14

---

Under review as a conference paper at ICLR 2023
Kivane Kose, Alican Bozkurt, Christi Alessi-Fos Melissa Gill, Caterina Longo, Giovanni Pellacani, Jennifer G Dy. Dana H Brooks, and Milind Rajadhyaksha. Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via multiscale encoder-decoder network (med- net). Medical Image Analysis, 67: 17:101 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90 90, 2017.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. UtLhttpp///aan
lecun. wttp///e////m//////////////////t
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garneut (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1):56-67, January doi: 10. 10008//42256--09--013--99999
2020. ISSN 2522-5839.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL htt.tttts///peeeeeeeeee wwt.eiiwwwweeeeeeeeeee...e.ee www//err/oo/ooooo Jzd=rJzIBB
Aria Masoomi, Chieh Wu, Tingting Zhao, Zifeng Wang, Peter Castaldi, and Jennifer Dy. Instance- wise feature grouping. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13374-13386. Curran As- sociates, Inc., 2020. URL 0b110a99dddeb07111100 0ag.aeeeeee..1000000055222222333 Pawp:22aaaaaaeaaaa..pppp. pddf
cC httpsssecppppppppeeeee paper/2020/Ei1a
Aria Masoomi, Davin Hill, Zhonghui Xu, Craig P. Hersh, Edwin K. Silverman, Peter J. Castaldi, Stratis loannidis, and Jennifer Dy. Explanations of black-box models based on directional feature interactions. In International Conference on Learning Representations, 2021.
Badri N. Patro, Mayank Lunayach, Shivansh Patel, and Vinay Namboodiri. U-cam: Visual expla- nation using uncertainty based class activation maps. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes Machine Learning. doi: 10.755 //mipress/3206.000..00 URL
The MIT Press, 11 2005. ISBN 9780262256834. ht httpps://dol www.ooogoo...........oooo......o 755 75w5///mmmt//ssa.... 3206. 001. 0001
Jonas Rauber, Wieland Brendel, and Mathias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. In Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning, 2017. URL BRRLhttp:// hettp://arxiv.ort wwwp:ii.orrg/bb.... 1707. 04131
Jonas Rauber, Roland Zimmermann, Matthias Bethge, and Wieland Brendel. Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensor- flow, and jax. Journal of Open Source Software, 5(53):2607, 2020. doi: 10.21 (0.21105/joss.02607 URL https::/doi......
10. 201221111sssss 02607
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Ex- plaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD Interna- tional Conference Knowledge Discovery and Data Mining, KDD '1.. pp. 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978-1-4503--232-2. doi: 10.1145/2939672.293977
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Nassir Navab. Joachim Homegger, William M. Wells, and Alejan dro F. Frangi (eds.). Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319.2457444.
12

---

Under review as a conference paper at ICLR 2023
Walter Rudin. Principles of Mathematical Analysis. McGraw-Hie Hilll Book Co., New York, third edition, 1976.
Aditya Saini and Ranjitha Prasad. Select wisely and explain: Active learning and probabilistic local post-hoe explainability. In Proceedings of the 2022 AAAT/ACM Conference on AI, Ethics, and Suciety, AIES '22, pp. 599-608, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978-1-4503-9225 9247-1.
doi: 10. N0.114573114094:5344999
C. Okan Sakar, S. Olcay Polat, Mete Katircioglu, and Yomi Kastro. Real-time prediction of online shoppers' purchasing intention using multilayer perceptron and LSTM recurrent neural networks. Neural Computing and Applications, 31(10)668336608, October 2019. ISSN 1433-3058. doi:
Jonas Schulz, Raul Santos-Rodright Rodriguez, and Rafael Poyiadzi. Uncertainty quantification of surro- gate explanations: An ordinal consensus approach. Proceedings of the Northern Lights Deep Learning Workshop, volume 3, 2022.
Patrick Schwab and Walter Karlen. CXPlain: Causal explanations for model interpretation under uncertainty. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buus Buc, E, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Mohammad Ahmad Sheikh, Amit Kumar Goel, and Tapas Kumar. An approach for predic- tion loan approval using machine learning algorithm. 2020 International Conference on Electronics and Sustainable Communication Systems (ICESC), pp. 490-494, 2020. doi: 10. 1091 2.200.911551
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3145-3153, PMLR, August 2017
Ashutosh Singh, Christiana Westlin, Hedwig Eisenbarth, Elizabeth A. Reynolds Losin, Jessica R. Andrews-Hanna, Tor D. Wager, Ajay B. Satpute, Lisa Feldman Barrett, Dana H. Brooks, and Deniz Erdogmus. Variation the norm: Brain state dynamics evoked by emotional video clips. In 2021 43rd Anmual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 6003-6007, 2021. doi: 10. 121100//BBCC44162222211212
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: Adversarial attacks post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Sociery, AIES '20, pp. 180-186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978-1-4503-7110-0. doi: DD.1145/337562733355555
Dylan Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable Post hoc Ex- planations Modeling Uncertainty in Explainability. In Neural Information Processing Systems (NeurIPS), 2021.
Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41:647-665, 2013.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 3319-3328. JMLR.org, 2017.
J. B. Tenenbaum, V. de Silva, and J. C. Langford. A global geometric framework for nonlinear di- mensionality reduction. Science (New York, N.Y.), 29015500) December 2000. ISSN 0036-8075. doi: 10. 2ttl/ccieccc220.5500.222
Koji Tsuda, Taishin Kin, and Kiyoshi Asai. Marginalized kernels for biological sequences. Bioinformatics (Oxford, England), 18(suppl, ):S268-) -275, July 2002. ISSN 1367-4803. doi: 10. 10.1003/hioonfomaaii///..... Issppp 1.S268.
13

---

Under review as a conference paper at ICLR 2023
Giorgio Visani, Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. Sta- tistical stability indices for LIME: Obtaining reliable explanations for machine learning models. Journal of the Operational Research Society, 73(1):91-101, 2022. 2022. doi: 10.10800016056622200000 1865846.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/1708.07747, 2017. URL [ttL:t//axx.v..orr
Zhiyong Yan and Congfu Xu. Using decision boundary to analyze classifiers. In 2008 3rd Inter- national Conference on Intelligent System and Knowledge Engineering, volume 1, pp. 302-307, Nov 2008. doi: 10.1 00.1109/SEE.200.444500000 4730945.
Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. On the (In)fidelity and Sensitivity for Explanations. UrXiv:I901.000 fcs, stat), November 2019.
Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. "why should you trust my explanation?" understanding uncertainty in lime explanations. arXiv preprint 1xXiv:/904.129/1 2019.
Xingyu Zhao, Wei Huang, Xiaowei Huang, Valentin Robu, and David Flynn. BayLIME: Bayesian local interpretable model-agnostic explanations. In Cassio de Campos and Marloes H. Maathuis (eds.), Proceedings of the Seveey-Seeenth Conference Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pp. 887-896. PMLR, July 2021.
14

---

Under review as a conference paper at ICLR 2023
A SOCIETAL IMPACTS
As machine learning models are increasingly relied upon in a diverse set of high-impact domains ranging from health-care to financial lending (Esteva et al. 2019; Kose et al. 202T Doshi Velez & Kim 2017, Sheikh et al. 2020; Singh et al. (021), ), it is crucial that users of these models can accurately interpret why predictions are made. An understanding of why model is making certain prediction is important for users to trust it - for instance a doctor may wish to know if a skin- cancer classifier's high test-set accuracy comes from the leveraging of truly diagnostic features, or a specific imaging device artifact. However, further spurred by the advent of deep learning's increasing popularity (Krizhevsky et al.) 2017), many of the models deployed in these high-stakes fields are complex black box s; producing predictions which are non-trivial to explain the reasoning behind. The development of many methods for explaining black-box predictions has arisen from this situation (Ribeiro et al. 2016 Lundberg & Lee 2017; Covert et al.) 22000] Masoomi et al.) 2021], but explanations may have varying quality and consistency. Before utilizing explanations in practice, it is essential that users know when, and when not, to trust them. Explanation uncertainty is one proxy for this notion of trust, in which more uncertain explanations may be deemed less trustworthy. In this work, we explore new way to model explanation uncertainty. in terms of local decision-boundayy complexity. In tandem with the careful consideration of domain experts, our methodology may be used to assist in determining when explanations are reliable. Our theoretical results provide new insights towards what explanation uncertainty entails, and open the door for future methods expounding upon our formulation.
B BACKGROUND
B.1 RELATED WORKS: RELIABILITY OF EXPLANATIONS
While feature attribution methods have gained wide popularity, a number of issues relating to the reliability of such methods have been uncovered. Mllvrrez & Jaakkola (2018) investigate the notion of robustness and show that many feature attribution methods are sensitive to small changes in input. This has been further investigated in the adversarial setting for perturbation-base meth- ods (Slack et al. 2020) and neural network-based methods. (Ghorbani et al., 2019). Kindermans) al. (2019) show that many feature attribution methods are affected by distribution transformations such as those common in preprocessing. The generated explanations can also be very sensitive to hyperparameter choice Bansal et al. (2020). number of metrics have been proposed for evaluating explainer reliability, such as with respect to adversarial attack (Hsieh et al. 2021), local perturba- tions (Alvarez-Meih) & Jaakkola 2018 Visani al.. 2022), black-box smoothness (Khan et al.]
2022
fidelity to the black-box model Yeh et al. 2019), or combinations of these metrics (Bhatt)
ct al. 2020).
B.2 GAUSSIAN PROCESS REVIEW
A single-output Gaussian Process represents distribution over functions : -
f(x) ~ GP(m(z)),(m))) k(r,,')).
(10)
Here m X' -> IR and k (X + R are the mean and kernel (or covariance) functions re- spectively, which are chosen a priori to encode the users assumptions about the data. The kernel function k(x, x²) reflects notion of similarity between data points for which predictive distribu- tions over f(x), I (x') respect. The prior m(xx - frequently considered to be less important is commonly chosen to be the constant m(x) 0.
Specifically,, a GP is an infinite collection of R.V's f(x), each indexed by an element x E X. Importantly, any finite sub-collection of these R.V's
f(Xer) (f(z1) ..., f(zn)) E Rd.
(11)
corresponding to some index set X {2i311 X, follows the multivariate normal (MVN) distribution, i.c.
f(X,,) (m((((e)) ((Xrr
(12)
Here the mean vector m(X.,) (m(x;),... (x,))) E R" represents the mean function applied on each Xxr and the covariance matrix K E R also known the gram matrix, contains
15

---

Under review as a conference paper at ICLR 2023
each pairwise kernel-based similarity value K5 2z);, Kernel function outputs correspond to dot products in potentially infinite dimensional expanded feature space, which allows for the encoding of nuanced notions of similarity; e.g. the exponential geodesic kernel referenced in this work (Feragen et al. 2015).
Making predictions with GP is analogous to simply conditioning this normal distribution data. Considering a set of input, free label pairs
D ( f [(:))
(13)
we may update our posterior over any subset of the R.V's f(x) by considering the joint normal over the subset and D and conditioning on D. For instance, when choosing singleton index set {ao), the posterior over f fzob) is another normal distribution which may be written as
f(zu) ~ N(((zo), ((f(z)))
(14)
where
f(IO) K X1.) K( Xtr) f(x)r(
(15)
v[((z))] = k(zo, 20) - K (zo, XI,) )² K (( x -K Xtr)
(16)
and K (x00 xxr) E Rd is defined element-wise by K (20; Xtr); x()
Now we may consider the situation where our labels are noisy:
= ((x,, y:)), = f(x;) + 6 6~ N(0, o² TR..
(17)
Here y/ is equal to the quantity we wish to model, f(x)), with the addition of noise variable e. The conditional is still a MVN, but the mean and variance equations are slightly modified
f(xa) K K X (xr))(((XX,, Xtr) +00
(18)
[=f(=)) k(zo, xo) - K ( (K(()rr X + o21)-1K(an) Xtr);
(19)
where E R" has elements Y; = V/-
Notice how the variance o² I is added to [(Xtr, Xtr) in the quadratic form in Eq. 19 resulting in smaller eigenvalues after matrix inversion. Since this quadratic form is subtracted. the decision to model labels noisy increases the uncertainty (variance) of estimates the GP posterior provides. This agrees with the intuition that noisy labels should result in more uncertain predictions.
While GPs may also be defined vector valued functions, in this work the independence of each output component assumed, allowing for modeling with c 2 1 independent GPs. For more details Ch.2 of Rasmussen & Williams 22055), from which the notation and content of this section were inspired.
C PROOF OF THEOREMS AND MULTICLASS EXTENSION
C.1 THEOREM RELATION TO EXPONENTIAL GEODESIC KERNEL
k(r,u) = 11 expl- Adgea (m, m ()(())) p)g(m' ly. p) dm
s.t. q(m/a, p) x expl-pllaz ml mllāpp(m)
Note that p controls how to weight manifold samples close to We take lim pomi
{
x = m and y = m
lim g(m|a, p) yy p) poo
Otherwise
Therefore the function within the integral of k(x,, y) evaluates to zero at all points except x = m and y m'. Since x, y E M evaluate the integral:
k( expl Lopp-xech Adgoo (z, v)]
assuming prior m(z)
16

---

Under review as a conference paper at ICLR 2023
C.2
THEOREM 2: KERNEL SIMILARITY AND DECISION BOUNDARY COMPLEXITY
From definition given any perturbation P on P, there must exist a compact subset K; C U, s.t. R PVin(()) (X.)(K.)) and R lln(K.) # id in((k)) Furthermore there exists linear homeomor- phism between an open subset U, S U3 with Rd which contains K,
We parametrize K; using smooth function T + K2 s.t. g(t) E OK; E
We further define (t) = g + cny(t), for some perturbation E R and smooth function 1) T Rd-1 We also restrict 7) such that (t) 0 vt E öT and E\t to E T s.t. n7(t)) * g(ta). In other words, 1 smooth function where g. (t) (t) Ve > 0, Vt ÖT, but is not identical to g for all I E T. Using (4), we define the manifold P. {ae (1) E
To complete the proof, we want to show that the kernel similarity between any two given points y E Rd is lower when using the manifold P. for > O) as opposed to the manifold Po- We therefore want to compare the two respective kernels Ke (x. and (o y). Note that in this proof consider the local effects of P on the kernel similarity through Po and P. exclusively, ignoring the manifold P Uo Using Euler- Euler-Lagrange, we can calculate lower bound for dgeo(ge (t)), In particular, for any t' T, dgeo (g. (t)), (t'))) 2 (go (o())))
dgoo (ge (t), (t)) 2 dgeo (go (t),
(20)
expl-Adgeo (g. (t), (t)) S expl- cxpp--Adgeloo0().0000)))
(21)
J expl xodpl->doo (I) didr' s J xx0 didr'
(22)
Note that in Eq. 22 we are integrating over all possible values of t, t', therefore the inequality is tight iff ge (f) ()(
(t) vt T: i.e. (see proof in C.2.Ip. The case of 0 is trivial; instead
assume > 0, in which case we can establish the following strict inequality:
dtdt' I expl (dooo)) go(')) dtdt'
(23)
Define uniform random variables T, over the domain of g, i.e. T,T T UT. Then we have:
TT [exp[- Ad (T), < BT.T' [expl Ad (g0 (o)
(24)
EM. (AM. (MMM)))))MMMM/////MM (exp (M, (M, M')]]
<
(25)
We define the random variable M ge (T) with distribution pe (M). The distribution pe (M) repre-
sents the uniform distribution 247 mapped to the manifold P. using ge (7)) The step from Eq. 24 to
Eq. 25 uses a property of distribution transformations (Eq. 2.2.5 in Casella & Berger (200T).
Next, compare either side of Eq. 25 to our kernel formulation shown below in Eq. 26 The kernel ke (x,, ylp, A) takes expected value over ge (M ar, p) and qe(M (y. p), which are equivalent to P. (M) and Pe (M') weighted with respect to x, y. and hyperparameter 2 0.
k ke vlp,)) EMSg. ( -ddel (M, M']]
(26)
s.l. ge (M xr, p) x exp[-pllx - M 21pp
s.t. (M y. p) x expl-plla plyy - M' (E)P)(
17

---

Under review as a conference paper at ICLR 2023
Note that when p is set zero, g(M x. 0) p(M) and (v. = p(M)). Therefore Eq. 25 equivalent to the inequality k. (2, y 00 A) (z, y/0, (000, 2).
We next want prove that the inequality ke ylp, A) ko(z, vlp, A) also holds for non-zero values of p. For convenience, define
f(p) = ko (z, vl
(27)
Under this definition, we want to prove there exists po 0 such that f(p) Vp < po. From Eq. 25 we established that f(0) > 0. Assume that
lim f(p) =c p-+
(28)
therefore follows that > In addition, note that f(p) is continuous with respect to p (see proof in section C.2.3). Therefore for any there exists ô > 0 s.t. p < ö implies (f( cl <
We choose e and define the corresponding be po. Therefore:
p < 7 lf(p) - <cc
(29)
p Pm (p) <c
(30)
Since this result holds for any follows that the piecewise linear manifold local minimum under any perturbation along specific chart or combination of charts with respect to the kernel similarity k(x;, y) Vx, y R.a.
C.2.1 PROOF: EQ. 23
We want to prove:
Jr expl 10 didd' = -f//////ooo expl expl->dlgco (go (9)( didi'
-exp
(31)
(1) go()) VET
Consider the LHS of Eq. 31
S/ - (.) dtdt' = - expl- -dl--ddgeo (g) go). dtdt'
(32)
[ expl-xMeooooo00000 h(t.e)
expl- Addgoo (g.( didt'
(33)
Define h(t, E') as the function inside the integrals in Eq. 33 From Eq. 21 h(t,t)) ow, E T.
Since continuous (see proof in C.2.2) and f J h didt' = 0, it follows that h(t,t') = Vt, l' E T (Ch.6 Rudin (976)).
It therefore follows that:
expl Sap[--deen (9)()))
Vt,t' ET
(34)
From the definition of (7()) in g. (t) = g()) + en()), there must exist E T s.t. nit) # 0. Therefore must be zero for Eq. 34 to hold. It follows that (1) = (0( Vi T.
18

---

Under review as a conference paper at ICLR 2023
C.2.2 PROOF: CONTINUITY OF h(t,t')
We prove that h(t,t' continuous with respect t,t'. First note that by definition, (c continuous parametrization of the manifold P.. From Burago et al. (200T). it follows that for any two points ge (1), De (t") P., (g (l), Se (t")) continuous. Since the exponential functional
preserves continuity and the sum of continuous functions are also continuous, it follows that h(t, L') is continuous.
C.2.3 PROOF: CONTINUITY OF k(z, y) WITH RESPECT To p
We prove that k(æ, y) is continuous with respect to p.
/ expl Ad (m, m')) g(m|ar, p) q(m'lu.p) dmddm'
k(x,)) v)
(35)
Zum (p) (p) / Aexppl- -(ll= -m| + 118 m 1
dmdm'
(36)
2(p)
(p) / expl-plla -mm(mpmmm) dm
s.t.
Zm (p) = [ eepl- plly m (1)lp(mm) dm'
A = expl -ddooo m m'))p(m)p(mm)
Define h(p) pB, where B is constant. Consider h(p) h((p),, where po is fixed positive
constant:
(h(p) (po)) -IpB PoB
(37)
= (p- - -p 8|
(38)
It follows that V > 0, = [B] continuous for all p I
> 0 such that |p pol < 8 + [h(p) - h(po)l < Therefore
We set B to be m l m' and || - Ily-m' 1| which shows that (p), Zm (p), and (p) are also continuous, respectively. It then follows that the entirety of Eq. 36 is continuous,
C.3 EXTENDING TO MULTICLASS CLASSIFIERS
the multiclass case we define a black-box prediction model F: : X + IRS We consider the one-vs- all DB for every class y y {1....,c) c). defined as M, {Ir E Rd Fz(x) maxey F.(z) max (x))), where F3 indicates the model output for class k. We then apply the GPEC framework separately to each class using the respective DB. The uncertainty estimate of the GP model would be of dimension x 8.
D IMPLEMENTATION DETAILS
D.I ALGORITIM
The GPEC training algorithm is outlined in Alg. GPEC is parametrized using multi-output Gaussian Process Regression model using the explanations as labels. Once the explanations L, explanation uncertainty U, and WEG kernel matrix K generated from Alg. [ we directly use these values to update the GP posterior and calculate the prediction variance for new test samples
(Eq. 19).
19

---

Under review as a conference paper at ICLR 2023
D.2 ADVERSARIAL SAMPLE FILTERING MULTI-CLASS MODELS
We elect to sample from multi-class neural network decision boundaries by running binary search on train-point adversarial example pairs. Specifically, given test-point æo R and model prediction
= x F (ro), decision boundary points may be generated by the following procedure:
First, for each class v E V a set of My points is randomly sampled from the set of train points on which the model predicts class u:
X, {ar argmax F(x) v, x X [& M,,
(39)
Ve E V. An untargeted adversarial attack using some lp norm and radius is run on each point in Ky. the set of points with the same class prediction as 2o. Each attack output Attackun (2, c) E IRd paired with its corresponding input, resulting in the set
Xy = {xx Attacku (x. (x,c)) x E &
(40)
where for an element (a, b) E Xy we have a F(a) y, argmaxkey F(b) * y. where v is an unspecified class.
Likewise targeted adversarial attack, with target class y. is run each point in each of the sets of points that are not predicted as class y. Each attack output Attacky (x, c) E IRd may be paired with its input resulting in sets
X = (1x, Attack, (z, <)) x E X.}
(41)
Ve # y E y. Here, for an element (a, X. we have argmax;ey F(a) v, argmax F(b) y.
Thus, we have generated diverse set of Evsy Me pairs of points that lie opposite sides of the decision boundary for class y. The segment between any pair from a given set X, V + will necessarily contain point on the class v.s. class y decision boundary. Likewise, in the interest of further diversity, segments between any pair from the set Xy will contain a point on the class v.s. class y decision boundary, where A y E y is unspecified. A binary search may be run on each pair to find the boundary point in the middle.
In practice the entire procedure may be amortized for each class, and ran for all classes as a single post-processing step immediately after training. This results in a dictionary of boundary points which may be efficiently queried on demand via the model predicted class of any given test point.
Each adversarial attack is attempted multiple times, once using each radius value in the list: [0.0, 2e
1e 1, 1e 1.5e 3, 2e 3e 1e 1e 3e 5e 1.0). For given input, the output of the successful attack with smallest is used. If no attack is successful at any radius, the input is discarded from further consideration.
4. 5e 4, 8e -4-
In this work the Foolbox Rauber et al. (0017) 2020) implementation of the Projected Gradient De- scent (PGD) Madry et al. (2018) attack with the Tox norm was used for both targeted and untargeted attacks. The M. values used for the relevant datasets are indicated below in Appendix E.
E EXPERIMENT SETUP
E.1 DATASETS AND MODELS
Census. The UCI Census dataset consists of 32,561 samples from the 1994 census dataset. Each sample is a single person's 's response to the census questionaire. An XGBoost model is trained using the 12 features to predict whether the individual has income 2 $50k.
Online Shopper. The UCI Online Shoppers dataset consists of clickstream data from 12,330 web sessions. Each session is generated from a different individual and specifies whether a revenue- generating transaction takes place. There are 17 other features including device information, types of pages accessed during the session, and date information. An XGBoost model is trained to predict whether purchase occurs.
German Credit. The German Credit dataset consists of 1,000 samples; each sample represents an individual who takes credit from bank. The classification task is to predict whether individual is
20

---

Under review as a conference paper at ICLR 2023
Shoppers
German Credit
E-NNIT
GPEC-WEG
0.11
0.37
0.07
12.90
18.15
GPEC-RBF
0.00
0.00
0.02
8.95
7.41
CXPlain
0.05
0.06
0.04
9.76
18.18
BayesSHAP
140.40
54,56
4.86
42,467
42,361
Bayc:LIME
91,29
34.60
4.83
41,832
41,99 11,992
Table 2: Execution time comparison for estimating the uncertainty for all features for 100 samples (in seconds). For MNIST and f-MNIST datasets, results represent execution time for calculating uncertainty estimates with respect to all ten classes. For GPEC-WEG, GPEC-RBE, and CXPlain methods, the results show inference times.
considered good or bad risk. Features include demographic information, credit history, and infor- mation about existing loans. Categorical features are converted using one-hot encoding, resulting in 24 total features.
MNIST. The MNIST dataset (LeCun & Cortes 2010) consists of 70k grayscale images of dimension 28x28. Each image has single handwritten numeral, from 0-9. A fully connected network with and ReLU activation functions was trained and validated on on 50, 000 and 10, 000 image label pairs, respectively. Training lasted for 30 epochs with initial learning rate of 2 and a learning rate decay of γ 0.5 when training loss is plateaued. During adversarial example generation we used M,, 500 and M, 50 Vc y.
layer sizes 784-700-400-200-100
Fashion MNIST The Fashion MNIST dataset (Xiao et al. 2017) contains 70k grayscale images of dimension 28x28. There are 10 classes, each indicating different article of clothing. We train a MIP model with the same architecture used for the MNIST dataset, however we increase training to 100 epochs and increase the initial learning rate to 3. During adversarial example generation we used My 500 and M. 50 Ve # y.
E.2 COMPETITOR IMPLEMENTATION DETAILS
BayesLIME and BayesSHAP. Slack et al. 2021 extend the methods LIME and KernelSHAP to use Bayesian Framework. BME and BayesSHAP are fit using Bayesian linear regression models on perturbed outputs of the black-box model. The posterior distribution of the model weights are taken as the feature attributions instead of the frequentist estimate that characterizes LIME and KemnelSHAP. We take the expected value of the posterior distribution as the point estimate for feature attributions, and the 95% credible interval as the estimate of uncertainty. To implement Baye and BayesSHAP we the public implementation We the number of samples 200, disable discretization for continuous variables, and calculate the explanations over all features. Otherwise, we use the default parameters for the implementation.
CXPlain. Schwab & Karlen (2019) introduces the explanation method CXPlain, which trains surrogate explanation model based on causal loss function. After training the surrogate model, the authors propose using bootstrap resampling technique to estimate the variance of the predictions. In our experiments we implement the publicly available code We use the default parameters, which include using 2-layer UNet model Ronneberger al. (2015) for the image datasets and 2-layer MLP model for the tabular datasets. We take 95% confidence interval from the bootstrapped results as the estimate of uncertainty.
F ADDITIONAL RESULTS
F.1 EXECUTION TIME COMPARISON
In Table we include an execution time comparison between the methods implemented in this paper. Results are averaged over 100 test samples. For MNIST and f-MNIST datasets, results evaluate the time to calculate uncertainty estimates with respect to all classes. All experiments were run on an internal cluster using AMD EPYC 7302 16-Core processors. We observe from the results that
21

---

Under review as a conference paper at ICLR 2023
the methods that amortization methods (GPEC-WEO GPEC-RBF, CXPlain) are significantly faster than perturbation methods BayesLIME and BayesSHAP.
F.2 TOY EXAMPLE: EVALUATING GPEC ON A LINEAR CLASSIFIER
In order to understand how the GPEC-WEG uncertainty estimate behaves for linear models, use a toy example shown in Fig. F2 In the top row, we visualize the GPEC-WEG uncertainty; we see that the uncertainty estimate a small, constant value for the linear model (left) whereas uncertainty increases for the nonlinear model (right). Intuitively, GPEC derives its uncertainty estimate by eval- uating the distribution of explanations with respect to the black-box model. Since it is generally infeasible to evaluate the space of all possible explanations, GPEC will typically estimate "base- line" amount of uncertainty for every explanation, which depends on the sampling of the training distribution and is minimized for linear model.
In the bottom two figures we visualize the magnitude of the gradient for the two respective models, which is noiseless estimate of feature importance. We see that even using this deterministic feature importance method, the estimates can fail to be robust due to the nonlinearity, i.c. nearby samples within the same class can have very different explanations.
F.3 VISUALIZING EFFECTS OF EXPLAINER UNCERTAINTY IN GPEC ESTIMATE
In section 5.4 we evaluate GPEC's ability to combine uncertainty from the black-box decision boundary and the uncertainty estimate from BayesSHAP and SSV explainers. In Figure we extend this experiment to evaluate how well GPEC can capture the explainer uncertainty. We calculate the combined GPEC+explainer estimate using different numbers of approximation samples.
Both BayesSHAP and SSV depend on sampling to generate their explanations; having fewer sam- ples increases the variance of their estimates. As we decrease the number of samples from 200 (Row A) to 5 (Row B) we would expect that the explainer uncertainty, and consequently the com- bined GPEC uncertainty, would increase. We see in Row A that the results follow our intuition; uncertainty increases for most of the plotted test points and uncertainty does not decrease for any points.
F.4 ADDITIONAL RESULTS FOR UNCERTAINTY VISUALIZATION EXPERIMENT
Results for Y-Axis Feature In Figure X we visualize the estimated explanation uncertainty as a heatmap for grid of explanations. The generated plots only visualize the uncertainty for the feature on the x-axis. Due to space constraints, we list the results for the y-axis feature the appendix, in We can that the results are in line with those from the x-axis figure.
Figure
Black-box box model output For reference, Figure 9 we plot the probability output of the XGBoost models used in the visualization experiment (Figure
F.5 SENSITIVITY ANALYSIS WEG KERNEL PARAMETERS
The WEG kernel formulation uses two parameters, and A. The parameter controls the weighting between each datapoint and the manifold samples. As p increases, the WEG kernel places more weight on manifold samples close in &g distance to the given datapoint. The parameter À acts as bandwidth parameter for the exponential geodesic kernel. Increasing À increases the effect of the geodesic distance along the manifold. Therefore decision boundaries with higher complexity will have an increased effect on the WEG kernel similarity. Figures 10 [1 and 12 we plot heatmaps for various combinations of and À parameters to evaluate the change in the uncertainty estimate. The black line is the decision boundary and the red points are the samples used for training GPEC. Please note that the heatmap scales not necessarily the same for each plot.
22

---

Under review as a conference paper at ICLR 2023
Algorithm GPEC Model Training
Input: :
GPEC Training Samples E Rx Explainer Hyperparamelers (# Samples), (
optional: Samples for Functional Approximation Uncertainty estimate).
Output Explanations L E I x Explanation Uncertainty U R WEG Kereel K E [0, M
\\ Calculate Function Approximation Uncertainty Initialize L E I RMM U E R xxx
if Explainer returns uncertainty estimate then
for I,2....M
Ta Us f (x)).)
expllaaattnsoons (I4,)) and explanation uncertainty (U,,) from Explainer
end
end
else if Explainer is stochastic and J then for I.2....M do
Initialize Q _
for I.2....J do
Q₁ (();.)
Draw stochastic explanations for same data sample
end
J
Ii; +- j Σe
Qj:
1=1
Us. + } Σ
(Q..: - L,:)²
Empirical uncertainty estimate (Eq.
end
end
else if Explainer is deterministic then
for M. do
L. E(X..)
end
+-
0
\1 Sct functional approximation uncertainty to zero
end
\\ Calculate EG Kernel Matrix
B + Decision Boundary Samllert Sampler(F, P) Initialize G E [0,
Draw P DB samples of dimension B E Pxd
PxP 1
for 1.2....P do
for j I.2....
Gi f- exp( -ddooo (Bi,;, Bj:))
Eq
end
end
W Calculate WEG Kernel Matrix Initialize W [0, MXP for 1.2....M do
\\ Initialize weighting matrix
for j 1,2.... do
W. + expl- pop(-p1- XI. B)
end
W
W.,
1) Normalize weighting distribution (Eq.
Σ Wi,
end
K T
Apply weighting EG Kernel Matrix
Return L, U.K
23

---

Under review as a conference paper at ICLR 2023
GPEC-WEG: Linear
GPEC-WEG: Cos
10
10
0.25
0.25
5
0.20
0.20
0.15
0.15
0.10
0.05
0.05
-10
-0,00
-10
0..00
-10
-5
10
110
5
10
Gradient Norm: Linear
Gradient Norm: Cos
100.00
80.00
80.00
60.00
-60.00
x2
X
-40.00
-0.000
-10
-0.00
-0.00
x1
x1
Figure 6: Comparison of GPEC uncertainty estimates linear and nonlinear toy models: funear (11, T22) = 22 and fex (21, 22) = 2 cos( 10 T2. TOP: Uncertainty estimate from GPEC. Applying GPEC on a linear model results a small, relatively constant variance for the test expla- nations. As the DB becomes more complex, in fcos. the uncertainty estimate increases around the nonlinearities the DB. BOTTOM: Gradient norm, which is an estimate of feature importance. The feature importance estimate becomes unstable (i.e. nearby samples of the same class have have very different explanations) due to the nonlinearity of foos.
German Credit
Census
GPEC + BavesSHAP
GPEC SSV
GPEC + BayesSHAP
GPEC SSV
Ordil Aoo.App Cooktpateeens
0
A)
Creell Amourt (00)
Chwdit Arsount (00)
Age
Age
Coest (lovrte) 8 B
Costre (Warra)
0.20
0
0
B)
62
00.0
CrodRAnoor.. (00)
Crodk-Anooutt (00)
Ago
Age
Orof Mereg
Oraf Ago Moreol
8 8
0
3.10
0.28
405
-ore
Aruue (00)
(00)
Age
Age
Figure 7: Comparison of the change in quantified uncertainty of explanations we change the number samples for BayesSHAP and SSV. Row (A) visualizes the combined uncertainty estimate using GPEC and either BayesSHAP or SSV, using 200 samples for approximating the BayesSHAP SSV explanation. In Row (B) we decrease the number of samples to and recalculate the estimated uncertainty. Row (A) represents the change in uncertainty estimate between (A) and (B), We see that the average uncertainty changes as we decrease the number of samples, which indicates that GPEC is able to capture the uncertainty arising from BayesSHAP SSV approximation.
24

---

Under review as a conference paper at ICLR 2023
GPEC WEG
GPEC- RBF
BayesSHAP
BayesLIME
OXPlain
Cenzus Houre per weet
4.20
330
350
4.0
-020
60
2.58
1.5
60
2.00
1.0
1.00
69
0:20
0.19
00.5
00.4
0.2
0.16
0.01
3.1
3.00
3.00
0.06
Age
AgM
AgM
Age
Age
0.22
3.0
0.30
0023
Consus
0.0
3.00
035
12
3.00
3.00
033
0.0
9.50
030
3.00
3.00
Age
Age
Age
Age
Age
German Credit
Creft Syg (ltrrtn)
2.30
2.50
2:38
3:50
09
2.00
250
60.5
0.23
0.20
6.17
-036
-034
0.22
001
1.3
030
0.56
0.50
0.56
20
Credit
Credit (100)
Crede (00)
(00)
Figure 8: Complement to Figure Visualization of estimated explanation uncertainty where the heatmap represents level of uncertainty for the feature on the y-axis.
Census
Census
German Credit
0.0
0
0
6.70
0.53
0.10
240
Credit Amount (10)
Figure 9: Output of binary classifiers used in the experiments. The color indicates the predicted
probability of each point. Points with probability 2 0.5 are classified as class 1, and points with probability < 0.5 are classified class O. The decision boundary, which is the set of points {m m E IR2 fmm = 0.5) where f is the classifier, represented by the black line.
Lambda: 0.01
Lambda: 0.5
Lambda: 1.0
Lambda:: 0.01. Rho:
Lembda: Rho:
Lembda: Rho:
35
35
35
00
30
9
30
30
Rho: 0.1
25
20
2.5
25
20
20
20
10
100
1.0
4.5
-.5
-05
50
100
150
50
100
150
50
100
150
Lambda: 0.01. Rho: 0.01
Lambda: 0.5, Rno: 0.01
Lambda: 100, Rno: 0.01
2.00
355
355
30
30
Rho: 0.01
1.5
075
2.0
255
15
25
2.0
1.
0.50
Lambda: 0.01, Rho: 0,001
Lambda: Rho: 0.001
Lambda:
Rho: 0.001
0.05
0.30
0.32
0.55
0.5
0.23
Rho: 0.001
20
aus
20
015
20
a01
50
100
150
50
100
150
50
100
150
Figure 10: Hyperparameter sensitivity analysis for the German Credit Dataset. Heatmap of esti- mated uncertainty for the x-axis variable under different and À parameter choices.
25

---

Under review as a conference paper at ICLR 2023
Lambda: 0.01
Lambda: 0.5
Lambda: 1.0
Lembdd: 0.01, Rho: 0.1
Lambde: 0.5, Rho:
Lambda: 1.0, Rho: 0.1
25
35
20
60
15
Rho: 0.1
20
40
40
40
30
60
Lambda:
0.
20
2.5
0.5
15
15
Rho: 0.01
02
40
10
40
1.0
6%
01
0.
4%
20
20
50
40
Lambda: 0.01 Rho: 0.001
Lambbda: 0.5. Rho: 0.001
Lamoda: 1.0. Rho: 0.001
0.997
0.0175
0.20
0.006
D.016
Rho:
0.944
0.0100
0.003
0.0775
0.00
0.001
0.0025
30
40
50
50
70
30
40
50
60
70
20
30
40
50
60
70
Figure 11: Hyperparameter sensitivity analysis for the Census Dataset. Heatmap of estimated un- certainty for the x-axis variable under different p and parameter choices.
Lambda: 0.01
Lambda:
Lambda:
Lambda: 0,01, Rho: 0.1
Lambdd: 0.5, Rho: 0.1
Lambda: 1.0. Rhe:
14
0.66
007
14
0.5
6.6
14
0.8
005
04
0.6
Rho: 0.1
0.4
001
10
0.
02
50
40
70
50
Lambda: 0.011 Rho: 0.01
Lambda: 0.5, Rho: 0.01
Lambda: 1.0, Rho: 0.01
15
0.0035
14
0.0010
0.0038
14
c.0025
0.0030
6.0000
0.004
0.003
Rho: 0.01
B000e
0.0015
0.002
10
10
0.0010
0.0005
0.001
50
40
30
50 50
70
Lambba: 0.01, Rho: 0.001
Lambda: 0.5. Rhos 0.001
Lambda: 1.0. Rho: 0.001
16
0.00335
16
0.0007
34
14
0.0006
0.00025
0.0003
Rho:
000000
0.0003
10
000200
17
0.00010
10
60
640
0.0001
16
0,0001
0.0002
20
30
80
50
60
70
20
40
50
60
70
30
40
60
70
Figure 12: Hyperparameter sensitivity analysis for the Census Dataset. Heatmap of estimated un- certainty for the x-axis variable under different p and parameter choices.
26


Reviews:
[{"review_id": "g6hPcXk0cKA", "reviewer": "Reviewer_R7Fs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a model that estimates the confidence of an explanation model of a classification model. The proposed model takes into account the variance in explanations of the explanation model, as well as the brittleness of the decision boundary around the sample that is to be explained. Samples that lie around a _complex_ decision boundary are assigned a high uncertainty. The complexity of the decision boundary is here measured as the length of the shortest path on the decision boundary that connects two close samples. This definition requires the computation of a geodesic distance, which is approximated over a Monte-Carlo sampling method. The experiments compare uncertainty methods for explanations on MNIST, fashion MNIST and three UCI datasets.", "summary_of_the_review": "Nice idea with a somewhat flawed application, trying to build upon the flawed explaining models. Clarity should be improved in the rebuttal. Training and Inference complexity is possibly awfully high. Because I like the idea, I would argue for a weak reject, although I see many issues with the paper as is.\n\n# After Rebuttal Thoughts\nThe authors accomodated my request for clarity regarding the final method. I still have my concerns about the proposed approach, because it creates another layer of trust, that is however not certified. This is my main issue with the paper. The issue with the inference complexity also seems to consist, although I truly appreciate that the authors added a discussion about this.", "strength_and_weaknesses": "# Strengths\n* The model choices are generally well substantiated\n* The authors explain well the (computational) challenges and limits of the method and possible remedies\n* The figures are helpful to illustrate the ideas\n* The proposed idea to use the geodesic kernel in the Gaussian process is sound and quite neat\n# Weaknesses\n* Although I feel like I understand the idea well and the reasoning behind it, I have problems to really understand the whole training and inference process. There is this one paragraph summarizing the method, but it depicts the method only textually and I am missing a more technical description of the method. After all, a lot of things come here together.\n* I am sorry, but I have to ask: do we really need extra methods to derive uncertainty for explaining models of the actual model? It appears to me that the explainer is not really needed that much here after all, only for the variance term. Can't the proposed method be used as a confidence measure of the black-box model itself? There is a whole subsection describing well the issues with explaining models, their insufficiency, and their lack of actually explaining what is happening in the model. Why would we want to fix these flawed models, instead of scraping them and doing something proper? The proposed method itself relies on multiple hyper-parameters and it introduces another layer of possible inaccuracies, introducing possibly another false sense of trust. I believe that when the model could be used at least directly to replace the absolute failing uncertainty measure of softmax, that this would be a good contribution. I wonder if this is possible or if the inference would then take too long. This brings me to the next point:\n* The sample complexity seems to be high, especially when brittle decision boundaries are supposed to be detected. I approve that there is a complexity analysis, but I wonder if in practice the time to train and to conduct inference for this model is so high that it prevents the usage of big datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D_s6c0WM8e9", "reviewer": "Reviewer_6BBB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of reliable explanations. The authors find that existing methods can be inconsistent or unstable. Therefore, there is an impending need to quantify the uncertainty of such explanation methods in order to understand when explanations are trustworthy. The authors introduce a novel uncertainty quantification method parameterized by a Gaussian Process model, which combines the uncertainty approximation of existing methods with a novel geodesic-based similarity which captures the complexity of the target black-box decision boundary.\n", "summary_of_the_review": "See *Strength And Weaknesses*\n", "strength_and_weaknesses": "Strengths:\n1. The authors introduce a geometric perspective on capturing explanation uncertainty and define a novel geodesic-based similarity between explanations.\n2. The authors prove theoretically that the proposed similarity captures the complexity of the decision boundary from a given black-box classifier.\n3. This paper proposed a novel Gaussian Process-based framework.\n\n\nWeaknesses:\n1.  Since the proposed method is a combination of several components, an ablation study is necessary to identify the contribution of each component. Otherwise, it is difficult to evaluate the effectiveness of the proposed method.\n\n2. Performance is lacking on large-scale benchmarks. It is better to show some experimental results in large-scale datasets (e.g., CIFAR100, ImageNet) to demonstrate the effectiveness of the proposed method.\n\n3. Some important related work missing, such as [1, 2]\n\n[1] Zhang, Yujia, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. \"Why Should You Trust My Explanation?\" Understanding Uncertainty in LIME Explanations.\" arXiv preprint arXiv:1904.12991 (2019).\n[2] Patro, Badri N., Mayank Lunayach, Shivansh Patel, and Vinay P. Namboodiri. \"U-cam: Visual explanation using uncertainty based class activation maps.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7444-7453. 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CaYnPk0CxZ", "reviewer": "Reviewer_dnzT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to quantify uncertainty of explanations by modelling explanations via a Gaussian process regression, and by defining a kernel called the \"weighted exponential geodesic\" (WEG) kernel. This kernel computes the similarity between two data points via their distances to the decision boundary, and the geodesic distance (of their \"projections\") on the decision boundary. Experiments show that the proposed measure indeed places larger uncertainty close to the decision boundary, and smaller uncertainty elsewhere.", "summary_of_the_review": "While the proposed WEG kernel is interesting, the experiments are underwhelming, and do not convincingly demonstrate that the proposed approach achieves its goals of telling users when to trust an explanation method. As a result, I am leaning toward a reject.", "strength_and_weaknesses": "**Strengths**:\n\n+ The paper tackles an important problem of estimating the uncertainty of explanations, which can be used to (potentially) judge whether or not trust the underlying explanation.\n\n+ The proposed WEG kernel is a nice definition of the geodesic distance between two data points with respect to an underlying decision boundary. Arguments made to show that it constitutes a well-defined kernel, and that it is related to the exponential geodesic kernel are appreciated.\n\n**Weaknesses**:\n\n**Weak experiments; unclear if advances are practically relevant** \n\nThe experiments in this paper consist of (1) qualitative uncertainty visualizations for two selected features, (2) showing that uncertainty decreases for regularized models. Either of these experiments are not convincing demonstrations of the usefulness of uncertainty quantification, as they do not deliver on the promise of \"whether or not to trust an explanation\". In particular, it is still unclear whether there is a clear, concrete setting where the proposed method is truly advantageous. Some examples of experiments closer to this goal include, (1) studying the faithfulness or robustness of explanations, and comparing them with this uncertainty metric. Does the proposed metric provide a cheaper way to identify data points whose explanations are likely to be unreliable based on these (or other) metrics? (2) A toy example where the explanations and the underlying uncertainties associated with explanations are known in advance, and a demostration of whether the proposed approach is able to correctly recover these. Note that these are mere suggestions, and that any experiment that can convince the readers of potential usefulness and practical utility of these methods is welcome.\n\n**Missing discussion of scalability and statistical efficiency of proposed approach**\n\nThe core components of the proposed algorithm involve sampling from the decision boundary and computation of the geodesic distance. These are computationally expensive for large problems, as due to the curse of dimensionality one would expect the cost of characterizing the decision boundary (by sampling) to be exponential in the data dimensionality. The authors claim that they \"apply adversarial attacks to reduce search space size\" for MNIST, however it is unclear if this heuristic results in generation of independent samples from the decision boundary as required for the monte carlo sampling approach. Some discussion on the statistical properties of this approach and its computational complexity would improve the paper in my view.\n\n**Distance from DB may not be as important as its curvature**\n\nConsider the case of \"explaining\" a linear model, where some methods (gradients, smoothgrad) return the weight vector itself as an explanation (https://arxiv.org/abs/1711.06104), and thus there is no uncertainty in this case. However the proposed method still assigns non-zero uncertainty values to explanations based on both the geodesic distance as well as distance from the DB, which seems to contradict the ground truth. Thus the proposed approach may not be ideal to quantify uncertainty, and some discussion on this counterexample is welcome.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject