On Locality of Local Explanation Models
Authors: Sahra Ghalebikesabi
Lucile Ter-Minassian
Karla DiazOrdaz
Christopher C. Holmes
Venue: NeurIPS 2021 Poster (2021)
Keywords: Shapley values
model interpretability
reference distribution
neighbourhood sampling

Abstract: Shapley values provide model agnostic feature attributions for model outcome at a particular instance by simulating feature 
absence under a global population distribution. The use of a global population can lead to potentially misleading results when local model behaviour is of interest. Hence we consider the formulation of  neighbourhood reference distributions that improve the local interpretability of Shapley values. By doing so, we find that the Nadaraya-Watson estimator, a well-studied kernel regressor, can be expressed as a self-normalised importance sampling estimator. Empirically, we observe that Neighbourhood Shapley values identify meaningful sparse  feature relevance attributions that provide insight into local model behaviour, complimenting conventional Shapley analysis. They also increase on-manifold explainability and robustness to the construction of adversarial classifiers. 

On Locality of Local Explanation Models
Sahra Ghalebikesabi University Oxford sahra. ghalebikesabi @atats.ox. ac. uk
Lucile Ter-Minassian" University of Oxford
www.eeteeemmiaassnnntttaaa ac. uk
Karla Diaz-Ordaz
The London School of Hygiene & Tropical Medicine & The Alan Turing Institute
Chris Holmes
University of Oxford The Alan Turing Institute
Abstract
Shapley values provide model agnostic feature attributions for model outcome a particular instance by simulating feature absence under global population distri- bution. The of global population lead to potentially misleading results when local model behaviour is of interest. Hence we consider the formulation of neighbourhood reference distributions that improve the local interpretability of Shapley values. By doing so, we find that the Nadaraya- Watson estimator, a well-studied kernel regressor, can expressed as a self-normalised importance sampling estimator. Empirically, we observe that Neighbourhood Shapley values identify meaningful sparse feature relevance attributions that provide insight into local model behaviour, complimenting conventional Shapley analysis. They also in- crease on-manifold explainability and robustness to the construction of adversarial classifiers.
1 Introduction
The ability to correctly interpret a prediction model is increasingly important as we move to widespread adoption of machine learning methods, in particular within safety critical domains such as health care [7]. In this paper, we consider the task of attributing the features {l...., m] of complex machine learning model f : R - R', abstracted as a function that predicts a response given a test instance æ E Ih" given only black-box access to the model. We especially focus on two popular model-agnostic feature removal based local explanation models, namely LIME [32 and SHAP 226. However our findings are applicable other local explanation models that we do not consider this paper. As these methods are often described as fitting a local surrogate model to black box 34 natural question is: how 'local' are local explanation methods?
As simple motivating example as why this question matters, consider black box model given by f(x) = [[(x; > 0)223 I[2) S 0)æ² where II()) denotes the indicator function. When attributing the local feature importance at a test instance x (æ1, 2). with .2 fixed at 2, we would expect Feature-1 to receive a higher absolute attribution when x. is closer to the decision boundary at = 0. In Figure [ we report the results on this example from LIME and SHAP as well as for our proposed 'Neighbourhood SHAP' approach. We observe that Neighbourhood SHAP assigns Feature-1 smaller attribution, the higher the absolute value x is. SHAP and LIME, however, assign Feature-1 an attribution which is constant either side of æ 0 which illustrates that these methods capture global
equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

LINE Neightcuhoodd Local Lincar
-2-
Figure I: Attributions at = (I1, 2) with 21 varying for reference distribution of X Normal 1)
and black box f(x:) = I[x1 > 0)2x3 I(m1 - 0)z3.
model behaviour. The figure also shows that training a local linear approximation to the black box 30 & is misleading since Feature-2 receives significantly positive attribution for [-0.4, 0), even though Feature-2 contributes clearly negatively to the model outcome whenever I1 < 0.
This motivates the following contributions
1. We propose Neighbourhood SHAP (Section 3 which considers local reference populations for prediction points as a complimentary approach to SHAP. By doing so, we show that the Nadaraya- wataon estimator x can interpreted as an importance sampling estimator where the expectation is taken over the proposed neighbourhood. Empirically, we find that greater locality increases the number model evaluations on the data manifold and with this the robustness of the attributions against adversarial attacks.
2. We consider how smoothing can also be used to stabilise SHAP values (Section 4 We quantify the loss in information incurred by our smoothing procedure and characterise its Lipschitz continuity.
2 Background
We begin with short introduction to Shapley values the quantity of interest of the SHAP optimisa- tion procedure. For pre-defined value function v(T,x) x)) that takes a set of features T C {1, .1., m} as input, the Shapley value Φ (j, æ) of feature j measures the expected change in the value function from including feature j into a random subset of features S (1, (1....,m] \, (i} (without j)
v.(j.z) = E (u(SU()) - v(S,x) P(S)
where the expectation is taken over the feature coalitions whose distribution is defined by P(S) = | - This choice of probability distribution ensures that sampling set of size & has the same probability as sampling one of size I, P(S [ = k}) = P(S |||= (}) for k I
(0, ..., 1).
The choice of value function for explanation- based modelling feature attributions an instance has been the subject of recent debates 11 25 27 The consensus is to take the expectation of the black box algorithm f at observation over the not-included features S using a reference distribution r(X / x) such that
v(S,
E. r(X
[[ X3)]
for S := === (1,....., (1,...,m3// and the operation (xs, xg) denoting the concatenation of its two arguments. Marginal Shapley values 226 25 define r(X: x) = p(X²) where denotes the marginal data distribution. Conditional Shapley values 1 set the reference distribution equal to the conditional distribution given æs. r(x; .) X x.S). All in all, the Shapley value Φ(j,))) a) is characterised by the expected change in model output, comparing the output when include j in the model, i.e. integrate out some randomly sampled features 3 (i). with the model output where feature j is not included, i.e. we integrated out some randomly sampled features including S

---

((j 2)
E P(S)
E r( Bun 1x)
(
Sun E x(xp Aglle ((ss)
As we see, Shapley values are computed by estimating the change in model outcome when some features are integrated out over the reference distribution r r(X²5 x) which has so far been defined as either the marginal or conditional global population. For marginal Shapley values, the interpretation simplifies: The Shapley value of feature is the expected change in model outcome when we sample random individual x" from the global statistical population and set its feature j equal to zj (after we already set random set of features S E {1..... m) {5) equal to æs). This motivates our proposal in Section 3 of neighbourhood distributions where we instead sample random individual from the immediate neighbourhood of z, outlined in the next section.
Computing Shapely values challenging high-dimensional feature spaces, which motivates the widely adopted KernelSHAP approach [26] that estimates the Shapley values of all features by empirical risk minimisation of
E 1 E [f(xs. - P(S) r(Xg 12)
ΣΣ wf((((s, 9(S))) 1=11-
(1)
where g(S) = ou + Σド= + [[j S) is a linear explanation model with the Shapley values as its coefficients, (-/,3,) M are i.i.d. T. draws from the respective global reference distributions, [5:) is set of size C of sampled coalitions, and the weights are defined by the KernelSHAP weights (26). LIME optimises similar generalised expectation also sampling references from global distribution. To improve local fidelity of Tabular LIME, [32 propose to define the weights as w; = -m /o²) for a bandwidth a. While this weighting increases the importance of f(ss ".ss) proportional to the size of S, it however does not ensure that higher weights are assigned model evaluations for observations closer to ø.
simple solution to the locality problem is to a local linear approximation in the form of a tangent line that predicts the black box in small neighbourhood around 2;, in 30 g 43 Such
approach has however several drawbacks compared to SHAP (and thus Neighbourhood SHAP) such as higher instability, less interpretability, and assuming a fixed parametric form. While SHAP (and Neighbourhood SHAP) does not make any assumptions on the form of f in the feature space, local linear approximations assume linearity of the black box in neighbourhood. As consequence, this See Supplement A for
may result in misleading attributions, as was demonstrated in Figure [ detailed discussion of local approximating models versus local reference populations.
Neighbourhood SHAP
Shapley values similarly to other feature removal methods employ a global reference distribution when computing attributions. This can lead to surprising artefacts as illustrated in Figure 2 To increase the local fidelity of Shapley values, we propose sample from well-defined local reference population instead. Having selected a distance metric D, such as the Euclidean distance or the more powerful Random Forests 77.. we define distance-based distribution : RT + IR that is centred around æ, such as the exponential kernel d( Fg) exp(-D((r xt 12/021)))) Further, we define the local neighbourhood distribution n( xt) = nh d(r zz) rra x)) where r(r" 2)) can be any marginal conditional reference distribution and n. the normalising constant. This choice ensures that we sample neighbourhood values not only considering the metric space w.r.t z but also the data distribution. This leads to a proposed change to the optimisation problem of eq. (I) to the following Neighbourhood SHAP minimisation
E ((S)
E Red(Xg (2)) (2) [f]s g(())
3

---

gotre
100
donn
183 (2)
Figure 2: When sampling { (black dots) from reference distribution r r(X ax) (here S 0), the Shapley value e at is positive since f(x) is larger than Ex x) x5) In contrast, Neighbourhood SHAP dynbrd is negative since En(x 2) x)) is larger than f(x). This difference results from the fact that, first, the model outcome has local minimum æ, and second, f(ts, X takes its smallest values at the tails of the data distribution (right-skewed density of f(rs, x5) when X ~
(55~p(X3) p(X3), black line on the left). SHAP only captures that f()) is higher than
the average model outcome but not that f f() is smaller at than it is for any other close observation this is reflected by Neighbourhood SHAP.
Instead of estimating the neighbourhood distribution, we approximate the expectation of the model outcome in the neighbourhood around using self-normalised importance sampling [13] with proposal distribution rXX3 a))
[no d(X²) [s)s(as Ele d(a Σ d(x ( Tzg)
E [f(xs m(x
= r(xg s)
E
12)
While our proposal, Neighbourhood SHAP, weights the f( based on distance metric to x, KernelSHAP uses uniform weights, ie. d(z' rz) = We note that the proposed local neighbourhood sampling scheme has a convenient form which corresponds to the well-known Nadaraya-Watson estimator (28 42 35 used for kernel regression. Kernel regression is a parametric technique to model the non-linear relationship between a dependent variable Y (here, f x and an independent variable Z (here, X5), by approximating the conditional expectation X3).. While the form of the Nadaraya Watson estimator has so far been justified from a kernel theory perspective (Supplement we show that it can be interpreted an importance sampling estimator.
E[Y Z] (here, Ex(x;
Proposition 1. The Nadaraya- Watson estimator YE = Et
- d(2,1=")
Ei d(=,,-)) where d(2)2²)
kernel function, is consistent self-normalised importance sampling estimator of Y(Z) with proposal distribution p(z) and desired distribution proportional to p(=)d(z|2²).
As pointed out in Supplement B all Shapley axioms 26 39
still hold true for the Neighbourhood SHAP. Now, by linearity, we can quantify the difference between SHAP and Neighbourhood SHAP 'Anti-Neighe Anti-Neighboorrhod SHAP" (see Supplement D). Looking at this difference might be of value to characterise the information loss when contrasting an instance to the global population instead to a local neighbourhood. Finally, also derive variance estimator of Shapley values computed with the Shapley formula Supplement[]
On-Manifold Explainability. A major disadvantage of marginal Shapley values and LIME is that the concatenated data vectors (æs, x x7,5) for a sampled reference 2 do not necessarily on the data manifold [5] I0 This has two serious ramifications: 1) the model is evaluated in regions that lie off the data manifold where might behave unexpectedly, and be unrepresentative for the data population; similar problem was described by (24] who note that removal based methods induce

---

bias in model explanations if removal is modelled by imputing with observations that are far from the actual test instance; and 2) adversaries can use an out-o--distribution (OOD) classifier trained to distinguish real dat- from simulated concatenated data and, through this, construct model whose Shapley values look fair even though the model is demonstrably unfair on the real-data domain [37]. To circumvent this problem, [16] [ propose the use of conditional instead of marginal reference distributions. However, using conditional reference distributions changes the interpretability of Shapley values - i.e. unrelated features get non-zero attribution and thus, their use is controversial [25] A marginal Neighbourhood SHAP approach in contrast can achieve on-manifold explainability while keeping the properties of marginal Shapley values for small enough abrd if the data manifold is to some extent coherent (see Figure 3).
Figure 3: Concatenated data (pink dots) used for model evaluations for the computation of Ker- neISHAP (left) and Neighbourhood SHAP (Anbrd 0.1, right) at randomly sampled instance (maroon dots) where the data manifold is ring in RR,. Even though the background references (blue dots) lie on the data manifold, marginal Shapley values are evaluated at instances that lie off the data manifold.
Choice of Bandwidth. For andrd - oo, Neighbourhood SHAP will be equal to KerneSHHA,, while it converges to for abbrd + O. Small neighbourhoods thus induce regularisation in the predictions which we also observe empirically in Section 5 While SHAP values add up to ΣØ=1 "
m @uhrd (j. x)
f(x) Er(x- | 2) [](X))]. Neighbourhood SHAP attributions add up to f(x) - Exxx [X*)]
Hence, care needs to be taken when comparing SHAP and Neighbourhood SHAP, since the scales might differ. In this case, both SHAP values (standard and neighbourhood) can be divided by either the sum of their absolute values or by their standard deviation, to represent relative attribution measures. As commonly observed with kernel regression approaches, there some drawbacks, such as the additional hyperparameters (distance function, bandwidth) and increased variability especially in data sparse regions for small bandwidths. These problems can be tackled by choosing adaptive bandwidth methods. For instance, Unbrd could be chosen such that the 25% % closest observations to x assigned more than 75% of the weight mass. We propose plot the Neighbourhood SHAP values of the normalised features over a range of bandwidths, from anted [0, 3m). This provides a powerful diagnostic and information tool.
The computational burden changing anbrd is not as large as it might first appear. Our importance sampling approach has the desirable property that En(xy I> (f)s X3)) is estimated on the same set of references {ai H for each anbrd, and that only the importance weights vary with the band mindwidth. As a result, there are no additional model evaluations required when Neighbourhood SHAP is computed for a different unbrd This stands in contrast to other neighbourhood schemes proposed in the XAI literature such as KDEs 19, GANs (36] or Gaussian perturbations (33 where the black box
must be evaluated an additional C L times for each new bandwidth where C' denotes the number of sampled coalitions. Please refer to Supplement for a theoretical and empirical complexity analysis.
4 Smoothed SHAP
In the previous section, we discussed neighbourhood sampling as useful tool to understand feature relevance through feature removal. We have also seen that the proposed neighbourhood sampling approach relates to kernel smoothers such as the Nadaraya- Watson estimator. This result can give us insights consider a Smoothed SHAP that locally averages neighbouring SHAP values
2 aamdd (j, = Σ iiz damid (2
1
gomid ( x)0(i.
5

---

where {- are samples from the reference distribution and d is kernel function. Such smoothing procedures have been applied before in the explainability literature, e.g. for gradient- based methods [3 44 and can be of interest when the interpretability of SHAP values suffers under the The smoothing induces can be captured by a Lipschitz constant whose upper bound decreases with the bandwidth aemtd-
high instability of the black box E 19 21).
Theorem 2. For every To E I.T with || - zo vool=& 6, there exists a constant O- < S max, (J()) -
Ex(x- ()]f(x*))) ()))( such that ll (j. x) - @ (j, اااما)ل L||z - =ool the smoothed Shapley value estimator @ō (j, x) (2) with d(x,, x,) = exp(-11x - xz xx(l2/o3) if f() is bounded on { { where lu(o" is a fanction that decreases in o smt with l + 00 as a - 0.
Lipschitz continuity has been established as a favourable characteristic of explainability tools [. Introducing smoothness indeed lead to feature attributions that do not correctly reflect the model's behaviour. In certain circumstances, this disadvantage can be outweighed by the advantage of more "intuitive" explanations, i.e. when used to "explain to justify" (2)
With the tools from before, we can derive that Smoothed SHAP is an consistent importance sampling estimator of the SHAP values from the neighbourhood around x
aamid (jxx) n(X'
(ummdd( (i) - (omt p(S)
E
(3)
where the new value function is defined by u (S,x) x) = Ex(x' xx) [Er xg) xx)) X9)] This smoothed value function relates to the explicit modelling of feature inclusion and gives an interesting perspective on the meaning of smoothing, namely that a is measurement of the test instance variable X'. Exploring a smoothed summary of the SHAP values in the local neighbourhood around x highlights how local variability in f(z) drives changes in the SHAP feature attributions. This is interesting in its own right but particularly so if features are susceptible to reporting error. As an illustration, consider a black box algorithm that predicts the fitness level of an adult based on multiple covariates, including weight. The reported weight may be subject to error if unreliable scales are used. In addition, as weight varies constantly throughout the day. the individual might not be interested in the attribution for one particular weight at single point in time, but rather in the attribution that a range of weights per day receives. The test instance thus more appropriately described by a test distribution of X' around æ where X' is random variable that describes the volatility in the covariates of the test instance. If the test distribution is unknown, it can be estimated by setting it, earlier, equal to a neighbourhood distribution nmmt (X' I)) pammdd (X' 2 2dd (X') x)) where damad (X') z) encapsulates the prior belief on the variability of X' and y peem x) cap- tures the artefacts of the data distribution (i.e. skew, curtosis, high density regions). The kernel demod (x),,) x) = exp(-D2 (x), x)) smat Dxr,, x)) can now be defined with multivariate bandwidth Egmid = diag(o? 20ttd,1 .... 02 smtd.m. We can observe empirically that such choice can decrease the MSE of the estimation of Shapley values (Supplement K Building upon results from kernel regression, we can quantify the squared distance of Smoothed SHAP to o(j, x) (Supplement H). Finally, we also derive a variance estimator for Smoothed SHAP in Supplement[]
Choice of Smoothing Bandwidth. Prior information on the variability of the covariates of the test instance can be included in the definition of the bandwidth matrix. Fixed covariates, like age or season, are not expected to change and thus receive a bandwidth ammtd.j -> O, while volatile features like weight, temperature or windspeed are assigned a positive bandwidth. For bandwidths o smidd + 0o, the feature is treated as inherently missing. If ammid_ + oo for all features j. Smoothed SHAP equals the average of the Shapley values over all references which is often used as global explanation measure [1 6.. As Smoothed SHAP can be estimated efficiently once SHAP values have been computed for the reference population, we propose, again, computing it for several bandwidth choices, and using a plot with respect to the bandwidth as a visualisation technique to help inform the choice of bandwidth. The bandwidth induces bias- variance trade-off derived in Supplement H: the larger the bandwidth, the smoother the results, but also the less Smoothed SHAP reflects the model behaviour at x, especially if f is highly non-linear.
Connection to LIME Tabular LIME 32 1 provides the same explanation for any two instances [8] As such it is also an aggregated attribution measure, similar to Smoothed SHAP. Key differences are the treatment of different dimensions and no proven guarantees of Lipschitz continuity (see Supplement E
falling into the same quantile along each dimension

---

0.9
langth
Female
0.6
data
adult
hoston
soc Male
biie
deggrgg_ M
Iris
two, year. recid
0.3
prions count
SHAP No Negghourhood
4. Charrge degree
0.2
0.1
0.2
0.3
0.4
0.5
inlative attribution
(a) AUC from OOD LightGBM and RF over 10 runs with 95% Cls. Concatenated data was created by sampling as many coalition vectors as data and masking with random references. Where references are sampled locally (smaller aérdd), OOD classifiers per- form significantly worse.
(b) Adversarial black box predicts recidivism using the COM- PAS data. Absolute attributions obtained from Neighbourhood SHAP and KemelSHAP for comparability. The adversarial attack affects Neighbourhood SHAP (with anôrd = 0.5) less than KemelSHAP when aver- aged over 10 runs. Without adversarial attack, (Neighbourhood) SHAP attributes only race (not shown).
divided by the sum of attributions
Figure 4: Neighbourhood SHAP explains on-manifold and is robust to adversarial attacks.
5 Examples
We present comprehensive experiments on several standardised real-world tabular UCI data sets S of different sizes predicted with ensemble classifiers regressors, well as image classification task on the MNIST dataset. The experiments demonstrate some key attributes of Neighbourhood and Smoothed SHAP including: Neighbourhood SHAP increases on-manifold explainability and robustness against adversarial attacks; Neighbourhood SHAP also leads to sparser attributions than standard Shapley values; Smoothed SHAP tells us how Shapley values of neighbouring observations differ from the attribution of the test instance.
Since Neighbourhood SHAP, Smoothed SHAP and SHAP operate different scales, divide all attributions by their standard deviation (over features) unless otherwise specified. We present a subset of our results in this Section and refer the interested reader to Supplement K for thorough report of all experimental results (including simulated experiments), details and hyper-paramete settings.
On-Manifold Explainability and Robustness against Adversarial Attacks. For adversarial learning, train Random Forest and a LightGBM as OOD classifiers that distinguish true data from concatenated vectors used for model evaluations. We find that for small bandwidths anbrd. the adversary not able to distinguish between the test data and the concatenated test data (Figure 4a), leading to deterioration in their ability to discriminate true from concatenated vectors. Under able detect the true data manifold, we can thus claim that Neighbourhood SHAP relies more on observations from the data manifold than SHAP and LIME. Further, we mimicked the experimental setup of [37] on the COMPAS data set [4] an adversarial black box predicts recidivism based only on race if the data is predicted from the OOD classifier to be from the data manifold, and returns an unrelated column if not. As presented Figure 4b for 10 randomly sampled individuals, the unrelated column has no effect on Neighbourhood SHAP and race has higher relative attribution than it does for KernelSHAP.
the assumption that the classifiers
Increased Local Prediction Performance. As SHAP learns binary feature model g(S) = Φo Σj=1 TJ 5), we can sample feature coalitions and reference values to perturb test data and predict the model outcome at the perturbed data. To check local prediction performance, we weight the MSE at each test instance and each reference value with an exponential kernel of the distance between test instance and reference value. Its bandwidth signifies the size of the neighbourhood. Figure presents presents the MSE corresponding to an XGBoost model, applied to four different datasets. As expected, Neighbourhood SHAP with smaller bandwidth predicts data within small neighbourhood significantly better than Neighbourhood SHAP with larger bandwidth. Here we noticed that the difference between the bandwidths is larger where there are fewer features in the data set (such as
7

---

the iris dataset), We attribute the loss in performance to the difficulty of estimating meaningful distances in high dimensions.
festures)
features)
6032
23
3002
4020
25-
0.0-
1.5
3000
043
2.5
2020
15
1920
0.1
0.00
6:30
0.35
0.50
3.25
Nitghosurhood daca
Figure 5: MSE when predicting local model outcome of an XGBoost model averaged over 400 runs displayed with 95% confidence intervals. Neighbourhood SHAP with smaller bandwidth predicts neighbourhoods significantly better than with large bandwidths.
Interpretation of Neighbourhood SHAP. Neighbourhood SHAP computed with small kernel widths reflects feature attributions when contrasting with model behaviour at similar observations, whereas Neighbourhood SHAP computed with large kernel widths renders model behaviour contrast- ing at a population scale. Figure O shows the evolution of Neighbourhood SHAP across bandwidths on randomly picked observations across different data sets. The test instance in the bike data set, where XGBoost regressor predicts daily bike rentals, has a high normalised temperature of 0.82. As the median observation has temperature of 0.50, the neighbourhood of our test instance is expected to look considerably different to the global population. For small kernel widths, Neighbourhood SHAP computes a negative attribution for temperature, whereas marginal SHAP is positive. This sign "flip" is coherent with descriptive statistics: for subpopulation with temperatures +/-0.05 around 0.82, temperature is negatively correlated with outcome (correlation equal to -0.08) whereas overall, bike rental tends to increase Neighbourhood SHAP thus shows that a warmer temperature has in general a positive impact on the count of rental bikes, which reverses for very hot days. Standard Shapley values do not provide this type of fine- reeeggriied interpretation. Similarly, in the Boston data set (Figure 6 third column), our test instance is dwelling with high percentage of lower status population (LSTAT) equal to 18.76%. LSTAT gets positive Neighbourhood SHAP values for small kernel widths, whereas its marginal Shapley value is negative. This observation is consistent with the negative overall correlation, which is equal to -0.76, whereas for a restricted population with LSTAT +/- 1% it is equal to +0.15. For similar dwellings i.e. with high pupil-teacher ratio and many rooms, lower status populations do not decrease the value of the home as much as they do in general, and can even increase it.
warmer days (unconditional correlation equal to +0.47).
Interpretation of Smoothed SHAP. In contrast, Smoothed SHAP summarises marginal Shapley values (which contrast against the entire population) within a neighbourhood, instead of at a single instance. For example, consider the adult data (Figure first column). We chose a test instance for which the model performs poorly: its predicted probability of high income for this individual, aged 42, is equal to 0.09, when in actual fact the person has high income. is interesting to contrast the conventional Shapley value assigned to the person, which is obtained by Smoothed SHAP with asmtd - O, with the average Shapley values for individuals like them. We observe that Smoothed SHAP quickly assigns negative attribution to age and a positive attribution to education for ammid > 1, whilst SHAP values were positive and negative, respectively for the individual. This highlights local instability in the Shapley values, as the SHAP numbers for people similar to the predicted person are positive for education, and negative for the age feature. For the Boston data set we note that Smoothed SHAP of the Pupil/Teacher Ratio (PTRATIO) initially decreases for small a amtds as there are many dwellings with high PTRATIO the data neighbourhood of the test instance, while it then increases as the global attribution of this feature is in general higher.
Image Classification. We applied our Neighbourhood SHAP approach on KernelSHAP and also on DeepSHAP (26) which computes Shapley values for images based on gradients. After training a convolutional neural network on the MNIST data set, we explain digits with the predicted label given a background data set of 100 images with labels 3 and 8. As we see in Figure Neighbourhood DeepSHAP gives pixels close to the strokes attributions with the highest absolute values while DeepSHAP assigns less sparse and more blurry attributions. This is expected: DeepSHAP compares
8

---

06
200
-12
-44
19
Anoend reveial
-0.5
-16
LSTAT boatte cara
0.0
130
-25
140
100
LIT
-1.
Figure 6: Scaled attributions at three different test instances (see Supplements K for varying kernel widths computed with 2000 reference points in the adult, bike and Boston housing data sets. Bounds for LIME have been computed over 2000 runs, while the Shapley bounds have been estimated with their theoretical formula as outlined in Supplements J
each digit to random digit in the population, while Neighbourhood DeepSHAP only looks at images in the neighbourhood, i.e. images which have a similar stroke. As we show in the Supplement, the change in log odds of predicting a 3 after modifying the images depicting an 8 with the attributions (setting blue pixels to 0) is snon-significanll) higher for Neighbourhood DeepSHAP than it is for DecpSHAP. In contrast, Smoothed DeepSHAP leads to smaller change in log-odds which is expected since we lose information by smoothing. However we see that Smoothed DeepSHAP gives additional insights compared to DeepSHAP and Global DeepSHAP: In all images the lower left corner of the 8s is highlighted in blue only for Smoothed DeepSHAP. Thus, we know that there is at least one observation in the neighbourhood of these 8s that has strong negative attribution in that image area. This image however loses importance when aggregating over the whole data set. Note that LIME gives the sharpest results because we chose the hyperparameters such that the image is split into the largest number of super pixels. We however see that LIME gives counter-intuitiin results (i.e. lower right corner of the third 8 gets the lowest attribution, lower contour of first 8 gets highest attribution). Another popular explanation method called Integrated Gradients (IG) [40] was added to Figure 7 for comparison purposes. This method consists in computing the gradients along path from the input of interest to a bascline input. Thus it resembles reature-removal based approaches in so far as it requires baseline values be specified. The baseline is typically defined to be just the mean of the feature. If a fixed baseline value is chosen, IG suffers under the same globality problem that outlined for SHAP.
8
6
8
8
Original
LIME
SHAP
Neighbourhood
Smoothed
Globel
Anti Neighb.
Pixol Attribution
Figure 7: Randomly picked test images with explanations of the label 8. Red regions are pixels that increase the predicted probability of label 8 while blue regions decrease the predicted probability contrasted with the background data set.

---

Evaluation of the proposed methods The two proposed methods were evaluated using a deletion metric from the ROAR framework developed by Hooker et. [23]. For each observation, 20% of the most important features (measured by the magnitude of the absolute feature attributions) are imputed by their mean and the model retrained. Eventually, we compare the resulting changes in predictive accuracy. We distinguish between two settings: imputation with the (global) mean of the features and imputation with the local mean (i.e. the weighted mean of each feature where the weights were computed based on distance metric to the observation). Results are shown in the Supplementary Material (see K.7). Overall, Neighbourhood and Smoothed SHAP result in higher absolute change in test performance than Marginal SHAP.
6 Discussion
In this paper, we first highlighted the limitations of using SHAP when the local model behaviour is of interest. We then introduced Neighbourhood SHAP. While neighbourhood sampling has been applied in other areas of model explainabilityy such as image perturbations by adding noise [14], local linear approximations (see Supplements A). or rule-based models [20] 31 29]. has not been previously introduced for model agnostic additive feature models such as SHAP. Our contribution important as it provides a theoretical understanding of explanations of local model behaviour, which is often lacking in the explainable Al literature [18]. A secondary contribution of this work the analyses of how smoothing Shapley values can identify unstable feature attributions. While it is difficult to evaluate model explanations numerically, we provide an exhaustive comparison of different metries (adversarial robustness, prediction accuracy, and visual inspection). Neighbourhood SHAP and Smoothed SHAP both merit consideration, as they have considerable advantages compared to standard KemelSHAP. For comparability across experiments, we limited our analysis to the use of the euclidean distance as a distance metric. In high dimensional spaces, this choice can be misleading [2) and the use of more powerful distance metrics, such as one obtained by random forests, would be appropriate. We thus caution against exclusively relying on mathematical metrics for explaining models, and suggest comparing the un- un-weighted and weighted histograms before any judgement calls. While it can be difficult to choose adequate bandwidth, we see that having control over kernel width allows the user to have a precise understanding of model predictions, both locally and at a larger scale. LIME or KernelSHAP in their default implementation do not allow for such detailed analysis. Plots of Neighbourhood SHAP and Smoothed SHAP w.r.t the bandwidths anter and o ammad respectively are thus powerful tools that give additional insight into oblique dynamics of the black box.
Limitations and Societal Impact Due the increasing use of ML-based systems to assist to replace the human maces, there is need for transparency in algorithms. Developing robust methods that provide end-users with insightful explanations on the decisions made is key for building assurance in critical systems employing ML. However, we acknowledge that as all explainability tools, our method requires good model fit. Finally, caution against over-reliance on mathematical metrics for explaining models, as is no consensus evaluation tool for explainability models.
Acknowledgements
SG and LTM are students of the EPSRC CDT in Modern Statistics and Statistical Machine Learning (EP/S023151/11 SG receives funding from the Oxford Radcliffe Scholarship and Novartis. LTM receives funding from the EPSRC. KDO is funded by a Wellcome Trust/Royal Society Sir Henry Dale Fellowship 218554/Z/19/Z. CH is supported by The Alan Turing Institute. Health Data Research UK, the Medical Research Council UK the EPSRC through the Baycs4Haalth programme Grant SP/RO18561/ and AI for Science and Government UK Research and Innovation (UKRI). We would like to thank Brieuc Lehmann, Luke Merrick and Daniel Moss for helpful discussions.
10

---

References
[1] Aas, K., Jullum, M., and Loland, A. (2019). Explaining individual predictions when features are dependent: More accurate approximations to shapley values. arXiv preprint arXiv: I903. 10464.
[2] Adadi, A. and Berrada, M. (2018). Peeking inside the black-box: survey explainable artificial intelligence (xai). IEEE access, 1452138-52164
[3] Alvarez-Melis, D. and Jaakkola, T. (2018). On the robustness of interpretability methods. arXiv preprint
88669.
[4] Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). Machine bias. ProPublica, May. 139-19.
[5] Asuncion, A. and Newman, D. (2007). Uci machine learning repository.
[6] Bhargava, V., Couceiro, M., and Napoli, A. (2020). Limeout: An ensemble approach to improve process fairness. arXiv preprint arXiv:2006. 10531.
[7] Bloniarz, A., Talwalkar, A., Yu, B., and Wu, C. (2016). Supervised neighborhoods for distributed nonparametric regression. In Artificial Intelligence and Statistics, pages 1450-1459. PMLR.
[8] Botari, T., Hvilshøj, F., Izbicki, R., and de Carvalho, A. C. (2020), Melime: Meaningful local explanation for machine learning models. arXiv preprint urXiv:2009.05818
[9] Botari, T., Izbicki, R., and de Carvalho, A. C. (2019). Local interpretation methods to machine learning using the domain of the feature space. Joint European Conference on Machine Learning and Knowledge Discovery in Dutabases, pages 241-252. Springer.
[10] Chen, H., Janizek, J. D., Lundberg, S., and Lee, S.-I. (2020). True to the model or true to the data? arXiv preprint arXiv:2006. 16234.
[1I] Covert, I., Lundberg, S., and Lee, S.-I. (2020). Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33.
[12] Domingos, P. (2012). A few useful things to know about machine learning, Communications of the ACM, 55(10)778-87.
[13] Doucet, A., De Freitas, N., and Gordon, N. (2001). An introduction to sequential monte carlo methods. In Sequential Monte Carlo methods in practice, pages 3-14. Springer.
[14] Fong, R. C. and Vedaldi, A. (2017). Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, pages 3429-3437.
[15] Frye. C., de Mijolla, D., Cowton, L.,, Stanley, M., and Feige, I. (2020). Shapley-based explain- ability on the data manifold. arXiv preprint arXiv:2006.0127
[16] Frye, C., Feige, I., and Rowat, C. (2019). Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. arXiv preprint arXiv: 1910.0630.
[17] Gade, K. Geyik, S., Kenthapadi, K., Mithal, V., and Taly, A. (2020). Explainable ai in industry: Practical challenges and lessons learned. In Companion Proceedings of the Web Conference 2020, pages 303-304.
[18] Garreau, D. and von Luxburg, U. (2020). Looking deeper into lime. arXiv preprint 1111v22008.11092.
[19] Ghorbani, A., Abid, A., and Zou, J. (2019). Interpretation of neural networks fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681-3688.
[20] Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., and Giannotti, F. (2018). Local rule-based explanations of black box decision systems. arXiv preprint 18Xiv: J0820.
[21] Hancox-Li, L. (2020). Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 640-647.
11

---

[22] Holzinger, A., Biemann, C., Pattichis, C. S., and Kell, D. B. (2017). What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.09923.
[23] Hooker, G. (2004). Discovering additive structure black box functions. In Proceedings of the tenth ACM SIGKDD international conference Knowledge discovery and data mining, pages 575-580.
[24] Hsich, C.-Y., Yeh, C.-K., Liu, X., Ravikumar, P., Kim, S., Kumar, S., and Hsich, C.-. (2020). Evaluations and methods for explanation through robustness analysis. arXiv preprint ftv:22006000442
[25] Janzing, D., Minorics, L.,, and Blöbaum, P. (2020). Feature relevance quantification in explain- able ai: A causal problem. In International Conference on Artificial Intelligence and Statisties, pages 2907-2916. PMLR.
[26] Lundberg. S. and Lee, S.-L. (2017). A unified approach to interpreting model predictions. arXiv preprint arXiv:
[27] Merrick, L. and Taly, A. (2020). The explanation game: Explaining machine learning models using shapley values. In International Droms-Domain Conference for Machine Learning and Knowledge Extraction, pages 17-38. Springer.
[28] Nadaraya, E. A. (1964). On estimating regression. Theory of Probability & Its Applications, 91):141-142.
[29] Rajapaksha, D., Bergmeir, C., and Buntine, W. (2020). Lormika: Local cule-based model interpretability with k-optimal associations. Information Sciences, 540:221-241.
[30] Rasouli, and Yu, I. C. (2019). Meaningful data sampling for faithful local explanation method. In International Conference on Intelligent Data Engineering and Automated Learning. pages 38-38.55 Springer.
[31] Rasouli, P. and Yu, I. C. (2020). Explan: Explaining black-box classifiers using adaptive neighborhood generation. In 2020 International Joint Conference Neural Networks (IJCNN), pages 1-9, IEEE,
[32] Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "hhy should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
[33] Robnik-Sikonje M. and Bohanec, M. (2018). Perturbation- explanations of prediction models. In Human and machine learning, pages 1 159-175. Springer.
[34] Roscher, R., Bohn, B., Duarte, M. F., and Garcke, J. (2020). Explainable machine learning for scientific insights and discoveries. IEEE Access, 8:42200 42216.
[35] Ruppert, D. and Wand, M. P. (1994). Multivariate locally weighted least squares regression. The annals of statistics, pages 1370-
[36] Saito, S., Chua, E., Capel, N., and Hu, R. (2020). Improving lime robustness with smarter locality sampling. arXiv preprint arXiv:2006. 12302.
[37] Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020). Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Sociely, pages 180-186.
[38] Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: /06.0:88000055
[39] Sundararajan, M. and Najmi, A. (2020). The many shapley values for model explanation. In International Conference on Machine Learning, pages 9269-9278. PMLR
[40] Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR.
12

---

[41] Visani, G., Bagli, E., and Chesani, F. (2020). Optilime: Optimized lime explanations for diagnostic computer algorithms. arXiv preprint arXiv:2006.05714.
[42] Watson, G. S. (1964), Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pages 359-372.
[43] White, A. and Garcez, A. d. (2019). Measurable counterfactual local explanations for any classifier. arXiv preprint
03000..
[44] Yeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., and Ravikumar, P. (2019). On the (in) fidelity and sensitivity for explanations. arXiv preprint arXiv:1901.09392.
13


Reviews:
[{"review_id": "rz50vLVgMNt", "reviewer": "Reviewer_FFcC", "summary": "The paper addresses an important problem in model-agnostic local explanation methods, i.e., using global distribution to simulate feature attributions for explaining local model behaviors. To this extent, the authors consider the formulation of neighborhood reference distributions that improve the local interpretability of Shapley values. Further, empirical evaluations show that Neighborhood Shapley values identify relevant feature attributions and improve on-manifold explainability and adversarial robustness against adversarial attacks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The paper provides a theoretical understanding of attribution-based explanation methods for analyzing local model behavior.\n2. The paper is well-written with clear notations and motivations for NeighborhoodShap and SmoothedShap.\n3. The paper shows that how smoothing Shapley values can identify unstable feature attributions and improve output explanations.\n\nOpen questions\n1. One of the underlying assumptions to generate neighborhood samples using either the underlying data distribution process or from an estimator that approximates that distribution. In a black-box setting, we do not have any knowledge of the training data distribution, and for the estimator, there would be an error in approximating the underlying distribution. It would be great if the authors can comment on this.\n\n2. How practical is the assumption that \"models can detect the true data manifold?\"\n\n3. In the image classification results, Neighbourhood DeepSHAP looks at images in the neighborhood (i.e., images with similar stroke) which can be misleading. For instance, in Fig. 7, the neighborhood stroke for the top-left stroke of digit '8' can be similar to strokes of other digits (e.g., '2', '6', and '9').\n\n4. The proposed methods are extensively evaluated using diverse metrics such as adversarial robustness, prediction accuracy, and visual inspections. It would be great if the authors can comment why they did not use more established metrics from the literature like deletion/insertion metric?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "gh3qm70R2O-", "reviewer": "Reviewer_74h4", "summary": "The authors highlight on some of the flaws of SHAP for local explanations and propose a new variant, Neighbourhood SHAP, based on neighbourhood sampling and a kernel regressor.\n\nAuthors do a detailed theoretical and experimental analysis of the effects of the hyperparameters (i.g. kernel bandwith) of this technique and how they can be interpreted.\nThey consider several tabular datasets and one example on MNIST. They compare their insights with other common techniques (i.g. kernelSHAP).\n\nOverall, authors further the theoretical results of interpretability tools.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Work is original, clear, and very extensive. Furthermore is provides theoretical insights to a relevant method in XAI. I think this work should be accepted and would improve with some minor edits.\n\nSome points that authors should address and would make me consider to slightly improve their score (8 or 9):\n\n* The title is pretty broad \"On Locality of Local Explanation Models\", when it actually only addressing a class of local explanation models (\"SHAP\").\n* Authors might want to expand and contextualize their results within the broader family of XAI techniques like attribution/saliency techniques ( CAM/GradCAM, integrated gradients) or Layer-wise Relevance Propagation (LRP), especially since there are connections between these techniques and Shapley values [1].\n* The MNIST example is mostly anecdotal, stronger results would be quantitative. There is already a wave of works that are attempting to evaluate the quality of explanations [2-7]. Adopting some of these ideas/frameworks would give stronger evidence for claims.\n\nAuthors should write their Societal Impact section.\n\nCitations:\n[1] https://arxiv.org/abs/1903.10992\n[2] https://papers.nips.cc/paper/2020/hash/417fbbf2e9d5a28a855a11894b2e795a-Abstract.html\n[3] https://arxiv.org/abs/1907.09701\n[4] https://arxiv.org/abs/1904.11829\n[5] https://arxiv.org/abs/2003.07258\n[6] https://arxiv.org/abs/1806.10758\n[7] https://arxiv.org/abs/1806.10758", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors did not address it, they should be able to do it.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "YUx2gj1xciU", "reviewer": "Reviewer_RMYP", "summary": "SHAP computes feature attribution by estimating the expected change in model prediction when a feature is included into a random subset of the rest of features. To account for the \"removed features\", conventional SHAP employs a \"global reference distribution\" (typically the marginal or conditional global population) from which values are sampled for the missing (removed) features. This paper argues that such global reference distribution can be too coarse-grained as to understand the local behavior of model prediction. To increase local fidelity, the paper proposes Neighborhood SHAP which considers a more local reference distribution to sample from, which encompasses conventional SHAP as a special instance. The core idea of Neighborhood SHAP is the use of local reference distribution, which depends on not only the original marginal (or conditional) reference distribution considered in conventional SHAP but also a distance-based distribution that gives higher weights to values around the example to be explained. Such design allows Neighborhood SHAP to capture more fine-grained local behavior but also improves on-manifold explainability. The paper then discusses Smoothed SHAP (averaging SHAP around a local neighborhood) as a closely related variant. Experimental results demonstrate that Neighborhood SHAP increases local fidelity and possesses higher robustness to adversarial attacks, and Smoothed SHAP shows difference between attributions on the test instance and those on neighboring observations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Update**: Thank you to the authors for the well-thought responses, and conducting the suggested additional experiments. My questions have been addressed well, and I am happy to see the inclusion of additional discussions/experiments in the revision. I believe the paper cast an important notion on the \"locality\" of explanations, and has provided a comprehensive investigation on the topic.\n\nOverall: The paper presents a clean framework to view current SHAP approaches, and the proposed Neighborhood SHAP is well-motivated. The formulation of Neighborhood SHAP naturally leads to a closely related variant, Smoothed SHAP. While the experimental section can be improved by more ablations and comparisons (see below), current presentation of the paper does provide insights on how Neighborhood/Smoothed SHAP capture different explanation characteristics compared to the original SHAP.\n\nStrengths:\n- First of all, the paper is clearly-written and the organization makes it easy to follow.\n- The paper presents a comprehensive and clear overview of related work on SHAP. The overview provides a clean formulation on SHAP (line 62-63) and enables a clear viewpoint that current approaches to SHAP (marginal or conditional) are actually in a sense \"global\" reference distributions. By providing motivating examples illustrating the potential pitfalls of considering global reference distributions, the paper makes the problem of improving local fidelity well-posed.\n- The proposed Neighborhood SHAP is well-motivated, and through importance sampling, it could be viewed as a generalization of the original SHAP. This makes Neighborhood SHAP a natural extension of SHAP.\n- I appreciate that the authors have discussed how to choose the bandwidth for Neighborhood SHAP, as it could be critical to the success of Neighborhood SHAP.\n- I appreciate that the authors have included discussion on Smoothed SHAP in the paper in addition to Neighborhood SHAP, as Smoothed SHAP appears to be a very closely related and natural variant to think of.\n- Examples shown demonstrate the advantage of Neighborhood SHAP to the original SHAP: increased local fidelity and robustness to attacks. Interpretation on Neighborhood SHAP and Smoothed SHAP is also insightful.\n\nApart from the above, followings are some main question I have when reading the paper and suggestions on room to improve:\n- In Figure 1, does decreasing the size of local neighborhood (e.g. N(0, 1) --> N(0, 0.1)) affect (improves) the results for LIME and SHAP? Would gradient-based methods be better at capturing this kind of local behavior? In general, how would Neighborhood SHAP compare to gradient-based methods like Integrated Gradient?\n- In Figure 5, what is the exact metric (formal definition) used for the local prediction accuracy? How does the metric capture local fidelity better? How does Neighborhood SHAP compare to LIME on local fidelity (not shown in Figure 5)? How to explain why Neighborhood SHAP with small bandwidth can sometimes has better performance even when the neighborhood size of test data is large?\n- I'd expect to see more discussion and comparison between NeighborhoodSHAP and Smoothed SHAP. Can SmoothedSHAP be formulated as a special instantiation of Neighborhood SHAP (or vice versa)? Empirically. how does Smoothed SHAP compare to NeighborhoodSHAP on robustness against attacks / on local fidelity / and on explanation stability (sensitivity)? These seem to be missing now, and they are important to understand the different characteristics between the two.\n- Eq.2: Does Smoothed SHAP suffer from off-manifold problem?\n- Figure 6, first column, NeighborSHAP and SmoothedSHAP has very different attribution (different sign), how to explain this discrepancy and which one to use?\n- While the presented qualitative examples are illustrative, I believe more quantitative measurements on evaluating explanations (e.g., robustness to attacks or explanation sensitivity) could make the results much more convincing.\n- Recent work has pointed out that explanations using reference distribution could subject to unintentional biases [1], does Neighborhood/Smoothed SHAP mitigate this problem?\n\nMinor questions:\n- In line 79-81, it is mentioned that the weighting of LIME does not ensure higher weights are assigned to model evaluations for observations closer to $x$. Why is this the case? Could the authors elaborate more on this?\n- In line 85-87, why is SHAP not making any assumption on the form of $f$? SHAP does assume a linear additive model to fit the black-box model. Could the author discuss more on this?\n- Equation between Line 104-105: why does $r(X|x)$ disappear in the last (rightmost) term?\n\n[1] Evaluations and Methods for Explanation through Robustness Analysis. Hsieh et al. 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have pointed out some limitations of the work and how they managed to address them in the Discussion section.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "39SMwGUUhbr", "reviewer": "Reviewer_iHps", "summary": "The main contributions of this paper are two variants of SHAP explanation method, Neighborhood SHAP and Smoothed SHAP. These two methods are motivated from the prior work in extending SHAP only captures the global behavior of the model but leaves the local behavior unexplored. The author demonstrates that the proposed explanations, neighborhood SHAP and Smoothed SHAP provide better local explanations over other SHAP methods in several data distributions. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting and complete work in general. The authors have provided detailed theoretical analysis in extending the SHAP method with a distribution that is more useful to capture the model’s local behavior. The technical part is sound. My main questions are about the motivation of Smoothed SHAP and some analysis in the experiment section. \n\n### Originality\nThe paper extends SHAP explantion with distributions that can capture the model’s local behaviors, which is the motivation for Neighborhood SHAP. However, the transition to Smoothed SHAP seems to be a little sudden such that even without Smoothed SHAP, Neighborhood SHAP seems to stand on its own. The only relation explained in the paper is they are related to the same technology, Nadaraya-Watson estimator, in different ways (line 161-162). It would be interesting to discuss how the way of treating neighborhood samplers as Nadaraya-Waston estimator provides the reader a better understanding behind Smoothed SHAP and Neighborhood SHAP. For example, is there some unified view of these two approaches? \n\nOther than that, the related work is well-cited and the novelty of this paper is significant. \n\n### Quality\nThe theoretical part is sound and the authors have provided very useful appendix content for the readers. I have several questions about the analysis of Neighborhood SHAP and Smoothed SHAP. \n\n- How do we choose between Neighborhood SHAP and Smoothed SHAP? When analyzing these two approaches at page 8, the baseline approach is mostly the previous SHAP methods, e.g. Marginal SHAP for both methods, respectively. But what should the human user trust more when represented with Neighborhood SHAP and Smoothed SHAP as there are still obvious gap between the results shown in Figure 6. When the relation between Neighborhood SHAP and Smoothed SHAP is not well-described in the previous section, it would be very useful to discuss which method is better in terms of local explainability and what is the suggested way in this paper to interpret result of the Neighborhood SHAP against Smoothed SHAP in an explicit way.\n\n- Baseline choices in deep models. Comparing the neighborhood version and smoothed version of SHAP with LIME are reasonable choices for tree models or any models that are treated as black boxes. However, in the paragraph that shows Neighborhood DeepSHAP is more insightful than LIME seems to be less fair since DeepSHAP is based on DeepLIFT which has access to the model’s architecture and backpropagation while LIME is model-agnostic and therefore may not able to be faithfully capture the deep model’s behavior. The comparisons may be more fair if the baselines are also leveraging the gradient information, e.g. Integrated Gradient [1] and Smoothed Gradient [2]. \n\n### Clarity\nThe paper is well-organized. The writing is easy to follow even though there are some typos. The analysis in the experimental section is quite dense and there might be some ways to improve these paragraphs to make it easier to read. \n### Significance\nThis paper’s result is significant. Other researchers and especially the users of machine learning products may get benefit from the work. \n\n\n[1] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. ICML 2017\n[2] Smilkov, D., Thorat, N., Kim, B., Viégas, F.B., & Wattenberg, M. (2017). SmoothGrad: removing noise by adding noise. ArXiv, abs/1706.03825.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for a good back-and-forth discussion during the review process.  All reviewers recommend accepting this paper, appreciating the well-motivated method, the extensive complete original work, and the valuable the theoretical insights.  I encourage the authors to integrate all the suggestions from the reviewers into the camera ready.
Decision: Accept (Poster)