On Locality of Local Explanation Models
Authors: Sahra Ghalebikesabi
Lucile Ter-Minassian
Karla DiazOrdaz
Christopher C. Holmes
Venue: NeurIPS 2021 Poster (2021)
Keywords: Shapley values
model interpretability
reference distribution
neighbourhood sampling

Abstract: Shapley values provide model agnostic feature attributions for model outcome at a particular instance by simulating feature 
absence under a global population distribution. The use of a global population can lead to potentially misleading results when local model behaviour is of interest. Hence we consider the formulation of  neighbourhood reference distributions that improve the local interpretability of Shapley values. By doing so, we find that the Nadaraya-Watson estimator, a well-studied kernel regressor, can be expressed as a self-normalised importance sampling estimator. Empirically, we observe that Neighbourhood Shapley values identify meaningful sparse  feature relevance attributions that provide insight into local model behaviour, complimenting conventional Shapley analysis. They also increase on-manifold explainability and robustness to the construction of adversarial classifiers. 

On Locality of Local Explanation Models
Sahra Ghalebikesabi University Oxford sahra. ghalebikesabi @atats.ox. ac. uk
Lucile Ter-Minassian" University of Oxford
www.eeteeemmiaassnnntttaaa ac. uk
Karla Diaz-Ordaz
The London School of Hygiene & Tropical Medicine & The Alan Turing Institute
Chris Holmes
University of Oxford The Alan Turing Institute
Abstract
Shapley values provide model agnostic feature attributions for model outcome a particular instance by simulating feature absence under global population distri- bution. The of global population lead to potentially misleading results when local model behaviour is of interest. Hence we consider the formulation of neighbourhood reference distributions that improve the local interpretability of Shapley values. By doing so, we find that the Nadaraya- Watson estimator, a well-studied kernel regressor, can expressed as a self-normalised importance sampling estimator. Empirically, we observe that Neighbourhood Shapley values identify meaningful sparse feature relevance attributions that provide insight into local model behaviour, complimenting conventional Shapley analysis. They also in- crease on-manifold explainability and robustness to the construction of adversarial classifiers.
1 Introduction
The ability to correctly interpret a prediction model is increasingly important as we move to widespread adoption of machine learning methods, in particular within safety critical domains such as health care [7]. In this paper, we consider the task of attributing the features {l...., m] of complex machine learning model f : R - R', abstracted as a function that predicts a response given a test instance æ E Ih" given only black-box access to the model. We especially focus on two popular model-agnostic feature removal based local explanation models, namely LIME [32 and SHAP 226. However our findings are applicable other local explanation models that we do not consider this paper. As these methods are often described as fitting a local surrogate model to black box 34 natural question is: how 'local' are local explanation methods?
As simple motivating example as why this question matters, consider black box model given by f(x) = [[(x; > 0)223 I[2) S 0)æ² where II()) denotes the indicator function. When attributing the local feature importance at a test instance x (æ1, 2). with .2 fixed at 2, we would expect Feature-1 to receive a higher absolute attribution when x. is closer to the decision boundary at = 0. In Figure [ we report the results on this example from LIME and SHAP as well as for our proposed 'Neighbourhood SHAP' approach. We observe that Neighbourhood SHAP assigns Feature-1 smaller attribution, the higher the absolute value x is. SHAP and LIME, however, assign Feature-1 an attribution which is constant either side of æ 0 which illustrates that these methods capture global
equal contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

LINE Neightcuhoodd Local Lincar
-2-
Figure I: Attributions at = (I1, 2) with 21 varying for reference distribution of X Normal 1)
and black box f(x:) = I[x1 > 0)2x3 I(m1 - 0)z3.
model behaviour. The figure also shows that training a local linear approximation to the black box 30 & is misleading since Feature-2 receives significantly positive attribution for [-0.4, 0), even though Feature-2 contributes clearly negatively to the model outcome whenever I1 < 0.
This motivates the following contributions
1. We propose Neighbourhood SHAP (Section 3 which considers local reference populations for prediction points as a complimentary approach to SHAP. By doing so, we show that the Nadaraya- wataon estimator x can interpreted as an importance sampling estimator where the expectation is taken over the proposed neighbourhood. Empirically, we find that greater locality increases the number model evaluations on the data manifold and with this the robustness of the attributions against adversarial attacks.
2. We consider how smoothing can also be used to stabilise SHAP values (Section 4 We quantify the loss in information incurred by our smoothing procedure and characterise its Lipschitz continuity.
2 Background
We begin with short introduction to Shapley values the quantity of interest of the SHAP optimisa- tion procedure. For pre-defined value function v(T,x) x)) that takes a set of features T C {1, .1., m} as input, the Shapley value Φ (j, æ) of feature j measures the expected change in the value function from including feature j into a random subset of features S (1, (1....,m] \, (i} (without j)
v.(j.z) = E (u(SU()) - v(S,x) P(S)
where the expectation is taken over the feature coalitions whose distribution is defined by P(S) = | - This choice of probability distribution ensures that sampling set of size & has the same probability as sampling one of size I, P(S [ = k}) = P(S |||= (}) for k I
(0, ..., 1).
The choice of value function for explanation- based modelling feature attributions an instance has been the subject of recent debates 11 25 27 The consensus is to take the expectation of the black box algorithm f at observation over the not-included features S using a reference distribution r(X / x) such that
v(S,
E. r(X
[[ X3)]
for S := === (1,....., (1,...,m3// and the operation (xs, xg) denoting the concatenation of its two arguments. Marginal Shapley values 226 25 define r(X: x) = p(X²) where denotes the marginal data distribution. Conditional Shapley values 1 set the reference distribution equal to the conditional distribution given æs. r(x; .) X x.S). All in all, the Shapley value Φ(j,))) a) is characterised by the expected change in model output, comparing the output when include j in the model, i.e. integrate out some randomly sampled features 3 (i). with the model output where feature j is not included, i.e. we integrated out some randomly sampled features including S

---

((j 2)
E P(S)
E r( Bun 1x)
(
Sun E x(xp Aglle ((ss)
As we see, Shapley values are computed by estimating the change in model outcome when some features are integrated out over the reference distribution r r(X²5 x) which has so far been defined as either the marginal or conditional global population. For marginal Shapley values, the interpretation simplifies: The Shapley value of feature is the expected change in model outcome when we sample random individual x" from the global statistical population and set its feature j equal to zj (after we already set random set of features S E {1..... m) {5) equal to æs). This motivates our proposal in Section 3 of neighbourhood distributions where we instead sample random individual from the immediate neighbourhood of z, outlined in the next section.
Computing Shapely values challenging high-dimensional feature spaces, which motivates the widely adopted KernelSHAP approach [26] that estimates the Shapley values of all features by empirical risk minimisation of
E 1 E [f(xs. - P(S) r(Xg 12)
ΣΣ wf((((s, 9(S))) 1=11-
(1)
where g(S) = ou + Σド= + [[j S) is a linear explanation model with the Shapley values as its coefficients, (-/,3,) M are i.i.d. T. draws from the respective global reference distributions, [5:) is set of size C of sampled coalitions, and the weights are defined by the KernelSHAP weights (26). LIME optimises similar generalised expectation also sampling references from global distribution. To improve local fidelity of Tabular LIME, [32 propose to define the weights as w; = -m /o²) for a bandwidth a. While this weighting increases the importance of f(ss ".ss) proportional to the size of S, it however does not ensure that higher weights are assigned model evaluations for observations closer to ø.
simple solution to the locality problem is to a local linear approximation in the form of a tangent line that predicts the black box in small neighbourhood around 2;, in 30 g 43 Such
approach has however several drawbacks compared to SHAP (and thus Neighbourhood SHAP) such as higher instability, less interpretability, and assuming a fixed parametric form. While SHAP (and Neighbourhood SHAP) does not make any assumptions on the form of f in the feature space, local linear approximations assume linearity of the black box in neighbourhood. As consequence, this See Supplement A for
may result in misleading attributions, as was demonstrated in Figure [ detailed discussion of local approximating models versus local reference populations.
Neighbourhood SHAP
Shapley values similarly to other feature removal methods employ a global reference distribution when computing attributions. This can lead to surprising artefacts as illustrated in Figure 2 To increase the local fidelity of Shapley values, we propose sample from well-defined local reference population instead. Having selected a distance metric D, such as the Euclidean distance or the more powerful Random Forests 77.. we define distance-based distribution : RT + IR that is centred around æ, such as the exponential kernel d( Fg) exp(-D((r xt 12/021)))) Further, we define the local neighbourhood distribution n( xt) = nh d(r zz) rra x)) where r(r" 2)) can be any marginal conditional reference distribution and n. the normalising constant. This choice ensures that we sample neighbourhood values not only considering the metric space w.r.t z but also the data distribution. This leads to a proposed change to the optimisation problem of eq. (I) to the following Neighbourhood SHAP minimisation
E ((S)
E Red(Xg (2)) (2) [f]s g(())
3

---

gotre
100
donn
183 (2)
Figure 2: When sampling { (black dots) from reference distribution r r(X ax) (here S 0), the Shapley value e at is positive since f(x) is larger than Ex x) x5) In contrast, Neighbourhood SHAP dynbrd is negative since En(x 2) x)) is larger than f(x). This difference results from the fact that, first, the model outcome has local minimum æ, and second, f(ts, X takes its smallest values at the tails of the data distribution (right-skewed density of f(rs, x5) when X ~
(55~p(X3) p(X3), black line on the left). SHAP only captures that f()) is higher than
the average model outcome but not that f f() is smaller at than it is for any other close observation this is reflected by Neighbourhood SHAP.
Instead of estimating the neighbourhood distribution, we approximate the expectation of the model outcome in the neighbourhood around using self-normalised importance sampling [13] with proposal distribution rXX3 a))
[no d(X²) [s)s(as Ele d(a Σ d(x ( Tzg)
E [f(xs m(x
= r(xg s)
E
12)
While our proposal, Neighbourhood SHAP, weights the f( based on distance metric to x, KernelSHAP uses uniform weights, ie. d(z' rz) = We note that the proposed local neighbourhood sampling scheme has a convenient form which corresponds to the well-known Nadaraya-Watson estimator (28 42 35 used for kernel regression. Kernel regression is a parametric technique to model the non-linear relationship between a dependent variable Y (here, f x and an independent variable Z (here, X5), by approximating the conditional expectation X3).. While the form of the Nadaraya Watson estimator has so far been justified from a kernel theory perspective (Supplement we show that it can be interpreted an importance sampling estimator.
E[Y Z] (here, Ex(x;
Proposition 1. The Nadaraya- Watson estimator YE = Et
- d(2,1=")
Ei d(=,,-)) where d(2)2²)
kernel function, is consistent self-normalised importance sampling estimator of Y(Z) with proposal distribution p(z) and desired distribution proportional to p(=)d(z|2²).
As pointed out in Supplement B all Shapley axioms 26 39
still hold true for the Neighbourhood SHAP. Now, by linearity, we can quantify the difference between SHAP and Neighbourhood SHAP 'Anti-Neighe Anti-Neighboourhod SHAP" (see Supplement D). Looking at this difference might be of value to characterise the information loss when contrasting an instance to the global population instead to a local neighbourhood. Finally, also derive variance estimator of Shapley values computed with the Shapley formula Supplement[]
On-Manifold Explainability. A major disadvantage of marginal Shapley values and LIME is that the concatenated data vectors (æs, x x7,5) for a sampled reference 2 do not necessarily on the data manifold [5] I0 This has two serious ramifications: 1) the model is evaluated in regions that lie off the data manifold where might behave unexpectedly, and be unrepresentative for the data population; similar problem was described by (24] who note that removal based methods induce

---

bias in model explanations if removal is modelled by imputing with observations that are far from the actual test instance; and 2) adversaries can use an out-o--distribution (OOD) classifier trained to distinguish real dat- from simulated concatenated data and, through this, construct model whose Shapley values look fair even though the model is demonstrably unfair on the real-data domain [37]. To circumvent this problem, [16] [ propose the use of conditional instead of marginal reference distributions. However, using conditional reference distributions changes the interpretability of Shapley values - i.e. unrelated features get non-zero attribution and thus, their use is controversial [25] A marginal Neighbourhood SHAP approach in contrast can achieve on-manifold explainability while keeping the properties of marginal Shapley values for small enough abrd if the data manifold is to some extent coherent (see Figure 3).
Figure 3: Concatenated data (pink dots) used for model evaluations for the computation of Ker- neISHAP (left) and Neighbourhood SHAP (Anbrd 0.1, right) at randomly sampled instance (maroon dots) where the data manifold is ring in RR,. Even though the background references (blue dots) lie on the data manifold, marginal Shapley values are evaluated at instances that lie off the data manifold.
Choice of Bandwidth. For andrd - oo, Neighbourhood SHAP will be equal to KerneSHHA,, while it converges to for abbrd + O. Small neighbourhoods thus induce regularisation in the predictions which we also observe empirically in Section 5 While SHAP values add up to ΣØ=1 "
m @uhrd (j. x)
f(x) Er(x- | 2) [](X))]. Neighbourhood SHAP attributions add up to f(x) - Exxx [X*)]
Hence, care needs to be taken when comparing SHAP and Neighbourhood SHAP, since the scales might differ. In this case, both SHAP values (standard and neighbourhood) can be divided by either the sum of their absolute values or by their standard deviation, to represent relative attribution measures. As commonly observed with kernel regression approaches, there some drawbacks, such as the additional hyperparameters (distance function, bandwidth) and increased variability especially in data sparse regions for small bandwidths. These problems can be tackled by choosing adaptive bandwidth methods. For instance, Unbrd could be chosen such that the 25% % closest observations to x assigned more than 75% of the weight mass. We propose plot the Neighbourhood SHAP values of the normalised features over a range of bandwidths, from anted [0, 3m). This provides a powerful diagnostic and information tool.
The computational burden changing anbrd is not as large as it might first appear. Our importance sampling approach has the desirable property that En(xy I> (f)s X3)) is estimated on the same set of references {ai H for each anbrd, and that only the importance weights vary with the band mindwidth. As a result, there are no additional model evaluations required when Neighbourhood SHAP is computed for a different unbrd This stands in contrast to other neighbourhood schemes proposed in the XAI literature such as KDEs 19, GANs (36] or Gaussian perturbations (33 where the black box
must be evaluated an additional C L times for each new bandwidth where C' denotes the number of sampled coalitions. Please refer to Supplement for a theoretical and empirical complexity analysis.
4 Smoothed SHAP
In the previous section, we discussed neighbourhood sampling as useful tool to understand feature relevance through feature removal. We have also seen that the proposed neighbourhood sampling approach relates to kernel smoothers such as the Nadaraya- Watson estimator. This result can give us insights consider a Smoothed SHAP that locally averages neighbouring SHAP values
2 aamdd (j, = Σ iiz damid (2
1
gomid ( x)0(i.
5

---

where {- are samples from the reference distribution and d is kernel function. Such smoothing procedures have been applied before in the explainability literature, e.g. for gradient- based methods [3 44 and can be of interest when the interpretability of SHAP values suffers under the The smoothing induces can be captured by a Lipschitz constant whose upper bound decreases with the bandwidth aemtd-
high instability of the black box E 19 21).
Theorem 2. For every To E I.T with || - zo vool=& 6, there exists a constant O- < S max, (J()) -
Ex(x- ()]f(x*))) ()))( such that ll (j. x) - @ (j, اااما)ل L||z - =ool the smoothed Shapley value estimator @ō (j, x) (2) with d(x,, x,) = exp(-11x - xz xx(l2/o3) if f() is bounded on { { where lu(o" is a fanction that decreases in o smt with l + 00 as a - 0.
Lipschitz continuity has been established as a favourable characteristic of explainability tools [. Introducing smoothness indeed lead to feature attributions that do not correctly reflect the model's behaviour. In certain circumstances, this disadvantage can be outweighed by the advantage of more "intuitive" explanations, i.e. when used to "explain to justify" (2)
With the tools from before, we can derive that Smoothed SHAP is an consistent importance sampling estimator of the SHAP values from the neighbourhood around x
aamid (jxx) n(X'
(ummdd( (i) - (omt p(S)
E
(3)
where the new value function is defined by u (S,x) x) = Ex(x' xx) [Er xg) xx)) X9)] This smoothed value function relates to the explicit modelling of feature inclusion and gives an interesting perspective on the meaning of smoothing, namely that a is measurement of the test instance variable X'. Exploring a smoothed summary of the SHAP values in the local neighbourhood around x highlights how local variability in f(z) drives changes in the SHAP feature attributions. This is interesting in its own right but particularly so if features are susceptible to reporting error. As an illustration, consider a black box algorithm that predicts the fitness level of an adult based on multiple covariates, including weight. The reported weight may be subject to error if unreliable scales are used. In addition, as weight varies constantly throughout the day. the individual might not be interested in the attribution for one particular weight at single point in time, but rather in the attribution that a range of weights per day receives. The test instance thus more appropriately described by a test distribution of X' around æ where X' is random variable that describes the volatility in the covariates of the test instance. If the test distribution is unknown, it can be estimated by setting it, earlier, equal to a neighbourhood distribution nmmt (X' I)) pammdd (X' 2 2dd (X') x)) where damad (X') z) encapsulates the prior belief on the variability of X' and y peem x) cap- tures the artefacts of the data distribution (i.e. skew, curtosis, high density regions). The kernel demod (x),,) x) = exp(-D2 (x), x)) smat Dxr,, x)) can now be defined with multivariate bandwidth Egmid = diag(o? 20ttd,1 .... 02 smtd.m. We can observe empirically that such choice can decrease the MSE of the estimation of Shapley values (Supplement K Building upon results from kernel regression, we can quantify the squared distance of Smoothed SHAP to o(j, x) (Supplement H). Finally, we also derive a variance estimator for Smoothed SHAP in Supplement[]
Choice of Smoothing Bandwidth. Prior information on the variability of the covariates of the test instance can be included in the definition of the bandwidth matrix. Fixed covariates, like age or season, are not expected to change and thus receive a bandwidth ammtd.j -> O, while volatile features like weight, temperature or windspeed are assigned a positive bandwidth. For bandwidths o smidd + 0o, the feature is treated as inherently missing. If ammid_ + oo for all features j. Smoothed SHAP equals the average of the Shapley values over all references which is often used as global explanation measure [1 6.. As Smoothed SHAP can be estimated efficiently once SHAP values have been computed for the reference population, we propose, again, computing it for several bandwidth choices, and using a plot with respect to the bandwidth as a visualisation technique to help inform the choice of bandwidth. The bandwidth induces bias- variance trade-off derived in Supplement H: the larger the bandwidth, the smoother the results, but also the less Smoothed SHAP reflects the model behaviour at x, especially if f is highly non-linear.
Connection to LIME Tabular LIME 32 1 provides the same explanation for any two instances [8] As such it is also an aggregated attribution measure, similar to Smoothed SHAP. Key differences are the treatment of different dimensions and no proven guarantees of Lipschitz continuity (see Supplement E
falling into the same quantile along each dimension

---

0.9
langth
Female
0.6
data
adult
hoston
soc Male
biie
deggrgg_ M
Iris
two, year. recid
0.3
prions count
SHAP No Negghourhood
4. Charrge degree
0.2
0.1
0.2
0.3
0.4
0.5
inlative attribution
(a) AUC from OOD LightGBM and RF over 10 runs with 95% Cls. Concatenated data was created by sampling as many coalition vectors as data and masking with random references. Where references are sampled locally (smaller aérdd), OOD classifiers per- form significantly worse.
(b) Adversarial black box predicts recidivism using the COM- PAS data. Absolute attributions obtained from Neighbourhood SHAP and KemelSHAP for comparability. The adversarial attack affects Neighbourhood SHAP (with anôrd = 0.5) less than KemelSHAP when aver- aged over 10 runs. Without adversarial attack, (Neighbourhood) SHAP attributes only race (not shown).
divided by the sum of attributions
Figure 4: Neighbourhood SHAP explains on-manifold and is robust to adversarial attacks.
5 Examples
We present comprehensive experiments on several standardised real-world tabular UCI data sets S of different sizes predicted with ensemble classifiers regressors, well as image classification task on the MNIST dataset. The experiments demonstrate some key attributes of Neighbourhood and Smoothed SHAP including: Neighbourhood SHAP increases on-manifold explainability and robustness against adversarial attacks; Neighbourhood SHAP also leads to sparser attributions than standard Shapley values; Smoothed SHAP tells us how Shapley values of neighbouring observations differ from the attribution of the test instance.
Since Neighbourhood SHAP, Smoothed SHAP and SHAP operate different scales, divide all attributions by their standard deviation (over features) unless otherwise specified. We present a subset of our results in this Section and refer the interested reader to Supplement K for thorough report of all experimental results (including simulated experiments), details and hyper-paramete settings.
On-Manifold Explainability and Robustness against Adversarial Attacks. For adversarial learning, train Random Forest and a LightGBM as OOD classifiers that distinguish true data from concatenated vectors used for model evaluations. We find that for small bandwidths anbrd. the adversary not able to distinguish between the test data and the concatenated test data (Figure 4a), leading to deterioration in their ability to discriminate true from concatenated vectors. Under able detect the true data manifold, we can thus claim that Neighbourhood SHAP relies more on observations from the data manifold than SHAP and LIME. Further, we mimicked the experimental setup of [37] on the COMPAS data set [4] an adversarial black box predicts recidivism based only on race if the data is predicted from the OOD classifier to be from the data manifold, and returns an unrelated column if not. As presented Figure 4b for 10 randomly sampled individuals, the unrelated column has no effect on Neighbourhood SHAP and race has higher relative attribution than it does for KernelSHAP.
the assumption that the classifiers
Increased Local Prediction Performance. As SHAP learns binary feature model g(S) = Φo Σj=1 TJ 5), we can sample feature coalitions and reference values to perturb test data and predict the model outcome at the perturbed data. To check local prediction performance, we weight the MSE at each test instance and each reference value with an exponential kernel of the distance between test instance and reference value. Its bandwidth signifies the size of the neighbourhood. Figure presents presents the MSE corresponding to an XGBoost model, applied to four different datasets. As expected, Neighbourhood SHAP with smaller bandwidth predicts data within small neighbourhood significantly better than Neighbourhood SHAP with larger bandwidth. Here we noticed that the difference between the bandwidths is larger where there are fewer features in the data set (such as
7

---

the iris dataset), We attribute the loss in performance to the difficulty of estimating meaningful distances in high dimensions.
festures)
features)
6032
23
3002
4020
25-
0.0-
1.5
3000
043
2.5
2020
15
1920
0.1
0.00
6:30
0.35
0.50
3.25
Nitghosurhood daca
Figure 5: MSE when predicting local model outcome of an XGBoost model averaged over 400 runs displayed with 95% confidence intervals. Neighbourhood SHAP with smaller bandwidth predicts neighbourhoods significantly better than with large bandwidths.
Interpretation of Neighbourhood SHAP. Neighbourhood SHAP computed with small kernel widths reflects feature attributions when contrasting with model behaviour at similar observations, whereas Neighbourhood SHAP computed with large kernel widths renders model behaviour contrast- ing at a population scale. Figure O shows the evolution of Neighbourhood SHAP across bandwidths on randomly picked observations across different data sets. The test instance in the bike data set, where XGBoost regressor predicts daily bike rentals, has a high normalised temperature of 0.82. As the median observation has temperature of 0.50, the neighbourhood of our test instance is expected to look considerably different to the global population. For small kernel widths, Neighbourhood SHAP computes a negative attribution for temperature, whereas marginal SHAP is positive. This sign "flip" is coherent with descriptive statistics: for subpopulation with temperatures +/-0.05 around 0.82, temperature is negatively correlated with outcome (correlation equal to -0.08) whereas overall, bike rental tends to increase Neighbourhood SHAP thus shows that a warmer temperature has in general a positive impact on the count of rental bikes, which reverses for very hot days. Standard Shapley values do not provide this type of fine- reeeggriied interpretation. Similarly, in the Boston data set (Figure 6 third column), our test instance is dwelling with high percentage of lower status population (LSTAT) equal to 18.76%. LSTAT gets positive Neighbourhood SHAP values for small kernel widths, whereas its marginal Shapley value is negative. This observation is consistent with the negative overall correlation, which is equal to -0.76, whereas for a restricted population with LSTAT +/- 1% it is equal to +0.15. For similar dwellings i.e. with high pupil-teacher ratio and many rooms, lower status populations do not decrease the value of the home as much as they do in general, and can even increase it.
warmer days (unconditional correlation equal to +0.47).
Interpretation of Smoothed SHAP. In contrast, Smoothed SHAP summarises marginal Shapley values (which contrast against the entire population) within a neighbourhood, instead of at a single instance. For example, consider the adult data (Figure first column). We chose a test instance for which the model performs poorly: its predicted probability of high income for this individual, aged 42, is equal to 0.09, when in actual fact the person has high income. is interesting to contrast the conventional Shapley value assigned to the person, which is obtained by Smoothed SHAP with asmtd - O, with the average Shapley values for individuals like them. We observe that Smoothed SHAP quickly assigns negative attribution to age and a positive attribution to education for ammid > 1, whilst SHAP values were positive and negative, respectively for the individual. This highlights local instability in the Shapley values, as the SHAP numbers for people similar to the predicted person are positive for education, and negative for the age feature. For the Boston data set we note that Smoothed SHAP of the Pupil/Teacher Ratio (PTRATIO) initially decreases for small a amtds as there are many dwellings with high PTRATIO the data neighbourhood of the test instance, while it then increases as the global attribution of this feature is in general higher.
Image Classification. We applied our Neighbourhood SHAP approach on KernelSHAP and also on DeepSHAP (26) which computes Shapley values for images based on gradients. After training a convolutional neural network on the MNIST data set, we explain digits with the predicted label given a background data set of 100 images with labels 3 and 8. As we see in Figure Neighbourhood DeepSHAP gives pixels close to the strokes attributions with the highest absolute values while DeepSHAP assigns less sparse and more blurry attributions. This is expected: DeepSHAP compares
8

---

06
200
-12
-44
19
Anoend reveial
-0.5
-16
LSTAT boatte cara
0.0
130
-25
140
100
LIT
-1.
Figure 6: Scaled attributions at three different test instances (see Supplements K for varying kernel widths computed with 2000 reference points in the adult, bike and Boston housing data sets. Bounds for LIME have been computed over 2000 runs, while the Shapley bounds have been estimated with their theoretical formula as outlined in Supplements J
each digit to random digit in the population, while Neighbourhood DeepSHAP only looks at images in the neighbourhood, i.e. images which have a similar stroke. As we show in the Supplement, the change in log odds of predicting a 3 after modifying the images depicting an 8 with the attributions (setting blue pixels to 0) is snon-significanll) higher for Neighbourhood DeepSHAP than it is for DecpSHAP. In contrast, Smoothed DeepSHAP leads to smaller change in log-odds which is expected since we lose information by smoothing. However we see that Smoothed DeepSHAP gives additional insights compared to DeepSHAP and Global DeepSHAP: In all images the lower left corner of the 8s is highlighted in blue only for Smoothed DeepSHAP. Thus, we know that there is at least one observation in the neighbourhood of these 8s that has strong negative attribution in that image area. This image however loses importance when aggregating over the whole data set. Note that LIME gives the sharpest results because we chose the hyperparameters such that the image is split into the largest number of super pixels. We however see that LIME gives counter-intuitiin results (i.e. lower right corner of the third 8 gets the lowest attribution, lower contour of first 8 gets highest attribution). Another popular explanation method called Integrated Gradients (IG) [40] was added to Figure 7 for comparison purposes. This method consists in computing the gradients along path from the input of interest to a bascline input. Thus it resembles reature-removal based approaches in so far as it requires baseline values be specified. The baseline is typically defined to be just the mean of the feature. If a fixed baseline value is chosen, IG suffers under the same globality problem that outlined for SHAP.
8
6
8
8
Original
LIME
SHAP
Neighbourhood
Smoothed
Globel
Anti Neighb.
Pixol Attribution
Figure 7: Randomly picked test images with explanations of the label 8. Red regions are pixels that increase the predicted probability of label 8 while blue regions decrease the predicted probability contrasted with the background data set.

---

Evaluation of the proposed methods The two proposed methods were evaluated using a deletion metric from the ROAR framework developed by Hooker et. [23]. For each observation, 20% of the most important features (measured by the magnitude of the absolute feature attributions) are imputed by their mean and the model retrained. Eventually, we compare the resulting changes in predictive accuracy. We distinguish between two settings: imputation with the (global) mean of the features and imputation with the local mean (i.e. the weighted mean of each feature where the weights were computed based on distance metric to the observation). Results are shown in the Supplementary Material (see K.7). Overall, Neighbourhood and Smoothed SHAP result in higher absolute change in test performance than Marginal SHAP.
6 Discussion
In this paper, we first highlighted the limitations of using SHAP when the local model behaviour is of interest. We then introduced Neighbourhood SHAP. While neighbourhood sampling has been applied in other areas of model explainabilityy such as image perturbations by adding noise [14], local linear approximations (see Supplements A). or rule-based models [20] 31 29]. has not been previously introduced for model agnostic additive feature models such as SHAP. Our contribution important as it provides a theoretical understanding of explanations of local model behaviour, which is often lacking in the explainable Al literature [18]. A secondary contribution of this work the analyses of how smoothing Shapley values can identify unstable feature attributions. While it is difficult to evaluate model explanations numerically, we provide an exhaustive comparison of different metries (adversarial robustness, prediction accuracy, and visual inspection). Neighbourhood SHAP and Smoothed SHAP both merit consideration, as they have considerable advantages compared to standard KemelSHAP. For comparability across experiments, we limited our analysis to the use of the euclidean distance as a distance metric. In high dimensional spaces, this choice can be misleading [2) and the use of more powerful distance metrics, such as one obtained by random forests, would be appropriate. We thus caution against exclusively relying on mathematical metrics for explaining models, and suggest comparing the un- un-weighted and weighted histograms before any judgement calls. While it can be difficult to choose adequate bandwidth, we see that having control over kernel width allows the user to have a precise understanding of model predictions, both locally and at a larger scale. LIME or KernelSHAP in their default implementation do not allow for such detailed analysis. Plots of Neighbourhood SHAP and Smoothed SHAP w.r.t the bandwidths anter and o ammad respectively are thus powerful tools that give additional insight into oblique dynamics of the black box.
Limitations and Societal Impact Due the increasing use of ML-based systems to assist to replace the human maces, there is need for transparency in algorithms. Developing robust methods that provide end-users with insightful explanations on the decisions made is key for building assurance in critical systems employing ML. However, we acknowledge that as all explainability tools, our method requires good model fit. Finally, caution against over-reliance on mathematical metrics for explaining models, as is no consensus evaluation tool for explainability models.
Acknowledgements
SG and LTM are students of the EPSRC CDT in Modern Statistics and Statistical Machine Learning (EP/S023151/11 SG receives funding from the Oxford Radcliffe Scholarship and Novartis. LTM receives funding from the EPSRC. KDO is funded by a Wellcome Trust/Royal Society Sir Henry Dale Fellowship 218554/Z/19/Z. CH is supported by The Alan Turing Institute. Health Data Research UK, the Medical Research Council UK the EPSRC through the Baycs4Haalth programme Grant SP/RO18561/ and AI for Science and Government UK Research and Innovation (UKRI). We would like to thank Brieuc Lehmann, Luke Merrick and Daniel Moss for helpful discussions.
10

---

References
[1] Aas, K., Jullum, M., and Loland, A. (2019). Explaining individual predictions when features are dependent: More accurate approximations to shapley values. arXiv preprint arXiv: I903. 10464.
[2] Adadi, A. and Berrada, M. (2018). Peeking inside the black-box: survey explainable artificial intelligence (xai). IEEE access, 1452138-52164
[3] Alvarez-Melis, D. and Jaakkola, T. (2018). On the robustness of interpretability methods. arXiv preprint
88669.
[4] Angwin, J., Larson, J., Mattu, S., and Kirchner, L. (2016). Machine bias. ProPublica, May. 139-19.
[5] Asuncion, A. and Newman, D. (2007). Uci machine learning repository.
[6] Bhargava, V., Couceiro, M., and Napoli, A. (2020). Limeout: An ensemble approach to improve process fairness. arXiv preprint arXiv:2006. 10531.
[7] Bloniarz, A., Talwalkar, A., Yu, B., and Wu, C. (2016). Supervised neighborhoods for distributed nonparametric regression. In Artificial Intelligence and Statistics, pages 1450-1459. PMLR.
[8] Botari, T., Hvilshøj, F., Izbicki, R., and de Carvalho, A. C. (2020), Melime: Meaningful local explanation for machine learning models. arXiv preprint urXiv:2009.05818
[9] Botari, T., Izbicki, R., and de Carvalho, A. C. (2019). Local interpretation methods to machine learning using the domain of the feature space. Joint European Conference on Machine Learning and Knowledge Discovery in Dutabases, pages 241-252. Springer.
[10] Chen, H., Janizek, J. D., Lundberg, S., and Lee, S.-I. (2020). True to the model or true to the data? arXiv preprint arXiv:2006. 16234.
[1I] Covert, I., Lundberg, S., and Lee, S.-I. (2020). Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33.
[12] Domingos, P. (2012). A few useful things to know about machine learning, Communications of the ACM, 55(10)778-87.
[13] Doucet, A., De Freitas, N., and Gordon, N. (2001). An introduction to sequential monte carlo methods. In Sequential Monte Carlo methods in practice pages 3-14. Springer.
[14] Fong, R. C. and Vedaldi, A. (2017). Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, pages 3429-3437.
[15] Frye. C., de Mijolla, D., Cowton, L.,, Stanley, M., and Feige, I. (2020). Shapley-based explain- ability on the data manifold. arXiv preprint arXiv:2006.0127
[16] Frye, C., Feige, I., and Rowat, C. (2019). Asymmetric shapley values: incorporating causal knowledge into model-agnostic explainability. arXiv preprint arXiv: 1910.0630.
[17] Gade, K. Geyik, S., Kenthapadi, K., Mithal, V., and Taly, A. (2020). Explainable ai in industry: Practical challenges and lessons learned. In Companion Proceedings of the Web Conference 2020, pages 303-304.
[18] Garreau, D. and von Luxburg, U. (2020). Looking deeper into lime. arXiv preprint 1111v22008.11092.
[19] Ghorbani, A., Abid, A., and Zou, J. (2019). Interpretation of neural networks fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681-3688.
[20] Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., and Giannotti, F. (2018). Local rule-based explanations of black box decision systems. arXiv preprint 18Xiv: J0820.
[21] Hancox-Li, L. (2020). Robustness in machine learning explanations: does it matter? In Proceedings of the 2020 conference on fairness, accountability, and transparency, pages 640-647.
11

---

[22] Holzinger, A., Biemann, C., Pattichis, C. S., and Kell, D. B. (2017). What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.09923.
[23] Hooker, G. (2004). Discovering additive structure black box functions. In Proceedings of the tenth ACM SIGKDD international conference Knowledge discovery and data mining, pages 575-580.
[24] Hsich, C.-Y., Yeh, C.-K., Liu, X., Ravikumar, P., Kim, S., Kumar, S., and Hsich, C.-. (2020). Evaluations and methods for explanation through robustness analysis. arXiv preprint ftv:22006000442
[25] Janzing, D., Minorics, L.,, and Blöbaum, P. (2020). Feature relevance quantification in explain- able ai: A causal problem. In International Conference on Artificial Intelligence and Statisties, pages 2907-2916. PMLR.
[26] Lundberg. S. and Lee, S.-L. (2017). A unified approach to interpreting model predictions. arXiv preprint arXiv:
[27] Merrick, L. and Taly, A. (2020). The explanation game: Explaining machine learning models using shapley values. In International Droms-Domain Conference for Machine Learning and Knowledge Extraction, pages 17-38. Springer.
[28] Nadaraya, E. A. (1964). On estimating regression. Theory of Probability & Its Applications, 91):141-142.
[29] Rajapaksha, D., Bergmeir, C., and Buntine, W. (2020). Lormika: Local cule-based model interpretability with k-optimal associations. Information Sciences, 540:221-241.
[30] Rasouli, and Yu, I. C. (2019). Meaningful data sampling for faithful local explanation method. In International Conference on Intelligent Data Engineering and Automated Learning. pages 38-38.555 Springer.
[31] Rasouli, P. and Yu, I. C. (2020). Explan: Explaining black-box classifiers using adaptive neighborhood generation. In 2020 International Joint Conference Neural Networks (IJCNN), pages 1-9, IEEE,
[32] Ribeiro, M. T., Singh, S., and Guestrin, C. (2016). "hhy should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144.
[33] Robnik-Sikonje M. and Bohanec, M. (2018). Perturbation- explanations of prediction models. In Human and machine learning, pages 1 159-175. Springer.
[34] Roscher, R., Bohn, B., Duarte, M. F., and Garcke, J. (2020). Explainable machine learning for scientific insights and discoveries. IEEE Access, 8:42200 42216.
[35] Ruppert, D. and Wand, M. P. (1994). Multivariate locally weighted least squares regression. The annals of statistics, pages 1370-
[36] Saito, S., Chua, E., Capel, N., and Hu, R. (2020). Improving lime robustness with smarter locality sampling. arXiv preprint arXiv:2006. 12302.
[37] Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2020). Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Sociely, pages 180-186.
[38] Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Wattenberg, M. (2017). Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: /06.0:88000055
[39] Sundararajan, M. and Najmi, A. (2020). The many shapley values for model explanation. In International Conference on Machine Learning, pages 9269-9278. PMLR
[40] Sundararajan, M., Taly, A., and Yan, Q. (2017). Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR.
12

---

[41] Visani, G., Bagli, E., and Chesani, F. (2020). Optilime: Optimized lime explanations for diagnostic computer algorithms. arXiv preprint arXiv:2006.05714.
[42] Watson, G. S. (1964), Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A, pages 359-372.
[43] White, A. and Garcez, A. d. (2019). Measurable counterfactual local explanations for any classifier. arXiv preprint
03000..
[44] Yeh, C.-K., Hsieh, C.-Y., Suggala, A. S., Inouye, D. I., and Ravikumar, P. (2019). On the (in) fidelity and sensitivity for explanations. arXiv preprint arXiv:1901.09392.
13


Reviews:
[{"review_id": "uAmjazDHeTo", "reviewer": "Reviewer_dkVj", "summary": "The manuscript proposes Neighbourhood Shapley Values (and a smoothed version thereof) which is a modification of the existing KernelSHAP method. The main advantage is that instead of a global population density a local neighbourhood density is used. The authors demonstrate that this increases robustness of the explanation method and lead to sparser (and thus more interpretable) explanations.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript is well structured and the method is motivated convincingly. I also very much enjoyed the background section as it was helpful even as I am not an expert in Shapley values. Using only a local population density seems very natural. The experimental section seems convincing and appears to demonstrate the advantages of the method on several datasets (MNIST,  UCI) and problem settings (robustness, visual interpretiblity, prediction accuracy).\n\nA few suggestions for improving the manuscript are in order:\n\n-I would encourage the authors to use more consistent mathematical notation: In the manuscript, expectation values are labeled by their random variable (see e.g. L101-102 outer expectation), the corresponding density (see e.g. L104-105 left), or both (L101-102). It would be better to be consistent or denote dependence on random variable by S \\sim P(S). Another example of this is in L61 where \\bar{S} \\ j should be replaced by \\bar{S} \\{j} as j is not a set. \n\n-I also found it a bit hard to find the proofs for the propositions and theorems in the Supplement. It would be advisable to add a clear reference to the relevant section in the Supplement to the main text. For example, I was not able to locate a proof for Proposition 1 in the Supplemental. \n\n-Several pdf viewers that I tried did not correctly display some of the figures in the manuscript. This issue has not arisen with any other of the papers I have reviewed and could therefore be specific to the draft at hand. The authors may want to check this. For example, I had issues using Preview on an iPad. \n\nA few questions for the authors:\n\n(a)It would seem to me that the normalization constant n_c  of n in L97 can be given in closed form if the distance D of L96 is the standard Euclidean distance (as d is a Gaussian). In this case, estimating the expectation value in L101-102 by importance sampling would appear to be not necessary. Is IS used since other distances such as a Random Forest-based one is used?\n\n(b)Proposition 1 states that the Nadaraya-Watson estimator  is an unbiased estimator. Generically, self-normalized IS only leads to *asymptotically* unbiased estimators. Why is this not the case here?\n\n(c)Theorem 2 proves Libschitz continuity of the smoothed estimator. Is there any reason to expect that the Libschitz constant is small for reasonable choices of \\sigma_smtd?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are (tangentially) discussed in the Discussion section. Societal impact is hinted at in the introduction. Both aspects are therefore only discussed briefly but in my opinion sufficiently.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "pR4jesWu4WF", "reviewer": "Reviewer_6nZp", "summary": "Shapley value which is a game theory concept that assigns attribution to each player has been used in XAI, \nThey propose a method called Neighbourhood SHAP, which seems to perform better than Shapley's explanation when the local behavior of the model is of importance. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1) Notational style: This is minor, but still things like putting the random variable exactly underneath the expectation sign $\\mathbb{E}$ makes the notation a bit unappealing. Or writing $S\\setminus j$ rather than $S\\setminus \\{j\\}$. \n\n2) What do you mean by global population specifically? can it be defined more specifically before you start using it through out the paper?\n\n\n2) line 74, where are $L,C$ notation that are used in the equation come from?  \n\n\n\n\n\n\n3) I had a hard time finding the proof of theorems in the appendix, for instance theorem 2, is supplement G. Make sure to add the theorem statement in the supplement or change the title of section G to be more indicative of the theorem.\n\n4) In regard to theorem 2 proof, when you use the monotonicity of exponential and triangular inequality, $\\delta = ||z-y||^{2}$ so it should not be $\\delta^{2}$, you probably can fix it by defining $\\delta^{2} = ||z-y||^{2}$. In the denominator of the same equation, you are wrote things like $-||x_{j} - y||$ instead of $||x_{j} -y ||^{2}/\\sigma^{2}$. The 3rd line from below of the same page when you replace part of your equation with $\\hat{\\phi}^{\\textit{smooth}}$ should not the normalization part also be part of it based on your equation (2) in the main paper? \n\n\n\n5) Experiments: I like your experimental setup, the only concern that I have is running explanation methods on most of UCI data and MNIST sometimes is not enough to establish a convincing argument due to the simplicity of the data and as a consequence simplicity of prediction models. For instance, linear methods like lasso works perfectly fine (above 90 percent accuacy) on MNIST data set. It would've been nice to see result on movie review like IMDB, or CIFAR 10 which is a common datas for explanation. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IUegFuW8qn6", "reviewer": "Reviewer_xvf9", "summary": "This paper proposes a new method Neighbourhood Shapley value, which is able to interpret local behavior in attribution models and increase robustness against adversarial attacks, by using Nadaraya-Watson estimator as the importance sampling estimator, which is widely used for kernel regression. Results are supported by sound theoretical, analytical and real analysis. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Local interpretability of attribution models is an interesting topic and can be useful in multiple applications.\n2. This paper proposes a new method to interpret local model behavior using Nadaraya-Watson estimator. The approach is novel.\n3. This paper is well written and clearly conveys the idea and insights in great detail. \n4. Simulated examples are useful to demonstrate the idea while real data results show the generalizability to real data problems.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. TreeSHP is one of the latest developments in the line of SHAP when tree ensemble models are used for prediction. It also explains local model behavior via split of trees. It would be interesting to see a comparison between the proposed method and TreeSHAP. (TreeSHAP github: https://github.com/slundberg/shap )\n2. Computation burden of LIME and KernelSHAP is too high when sample size is large. It’s necessary to justify the computational burden of the proposed method in the main body and compare it with fast methods such as TreeSHAP. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "C_67w-aNDXA", "reviewer": "Reviewer_iiTd", "summary": "This paper focuses on the locality of explanation methods. Authors propose the Neighborhood SHAP and Smoothed SHAP, which are variants of the Shapley value. Authors also conduct various experiments to verify the effectiveness of the proposed explanation methods.", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\n1. This paper focuses on an important issue, i.e., the locality of explanation methods. The authors propose the Neighborhood SHAP and Smoothed SHAP to boost the locality of explanation results.\n\n[Weaknesses]\n\n1. The core strategy in this paper (i.e. sampling neighborhood samples to compute the baseline) is doubtful. In some cases, the neighborhood-based attribution may provide seemingly reasonable explanations, but theoretically, in other cases, the neighborhood-based attribution may have a significant risk of dramatically incorrect explanations.\n\nFor example, the experiment in Fig. 4 is a controversial example. For a certain perspective, the authors name some features as \"unrelated column\", because the DNN does not use these features for prediction when the input is not an OOD sample. Consequently, the explanation results also assign a weak attribution with such so-called \"unrelated\" features.\nHowever, we can also consider the exactly the same problem from another perspective. I.e., such so-called \"unrelated\" features have a strong correlation with whether a sample is OOD, which have been sophisticatedly encoded by the model. From this new perspective, a trustworthy explanation should reflect such correlation and assign these so-called \"unrelated\" features certain attributions.\n\nMore cucially, besides the above experiment, in real applicatoins, it is usually difficult for people to manually define \"unrelated\" features. In this case, the neighborhood-based explanation may have significant risk of incorrectly explaining an important feature as unrelated features. In particular, when there is no clear boundary between OOD samples and normal samples (i.e., in some applications when all samples are valid samples without OOD ones). In sum, in the general case, the logic of the DNN may be inconsistent with human recognition. I agree that in the special case of Fig. 4, the proposed method works well. However, theoretically, there is a great risk to use this method for other applications.\n\n2. In Line 49, the formula of P(S) is incorrect. It is supposed to be $P(S)=(|S|!(m-|S|-1)!)/m!$.\n\n3. Another key problem is as follows. In Line 117, authors claim that all Shapley axioms still hold for the Neighborhood SHAP. However, there is no detailed proof in the paper or in the supplementary material. Thus, it is still unknown whether the Neighborhood SHAP satisfies axioms, including the linear axiom, the symmetry axiom, the nullity axiom, and the efficiency axiom. Please formally prove this issue. Without such axioms, the trustworthiness of the explanation is doubtful.\n\n4. The contribution of this paper is incremental. Lee et al. [11] have proposed to use conditional distribution to define the feature importance. The proposed Neighborhood SHAP can be considered as a special case in [11]. Therefore, this paper only provides an incremental contribution.\n\n5. Authors prove that the Smoothed SHAP is captured by a Lipschitz constant. However, if a model is not bounded by Lipschitz and results of the Smoothed SHAP are captured by a Lipschitz constant, then is the explanation result still reliable?\n\n6. In experiments, authors evaluate the Neighborhood SHAP and previous methods mainly based on human intuition. However, it is not reasonable to assume that DNNs model knowledge in the same way as humans. Therefore, evaluation metrics are suggested to be applied in experiment.\n\n7. What do “feature=1” and “feature=2” mean in Figure 1? Should they be “Feature-1” and “Feature-2”? Besides, there is no label or number for the y-axis in the second sub-figure of Figure 1. Authors are suggested to correct this figure.\n\n8. Minor: \n(1) The paper is difficult to read due to the lack of rigorous and clear definitions of symbols. For example, in Line 55, what does f refer to?\n(2) In Line 225, the bandwidth $\\sigma$ can be misleading, since $\\sigma$ is also used to define the Neighborhood SHAP and the Smooth SHAP.\n(3) Some quotation marks in Lines 280-299 are mistakenly written.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses local feature importance and presents a limitation of current methods: they are, in a sense, not local “enough”. This is an important observation that should be studied. The authors present a possible solution to this problem by adding geometric and distribution-related components to the valuation function and show how this changes the feature importance score.

We had active discussion about this work. It was agreed that the problem presented is well justified. However, there were concerns about the proposed solution and therefore we conclude that it be more beneficial to reject the paper and encourage the authors to further study the problem. Here are some examples to questions that the authors may wish to consider when further exploring this domain:

-	How would the explanation behave for points that are relatively remote? Note that the reviewers have commented that the explanation could fail off the manifold and it is important to mention this in the final version of this work.
-	How would the explanation behave in high dimension when the Euclidean distance might be altered by many non-relevant features or features with different scale?
-	How should we measure how good is the explanation? I.e., if someone proposes a different way to provide local explanations, how would we know which one is better? In other words, how do we know that the feature importance score presented here is good?
-	Do the properties of Shapley Value match the requirements in the case? After all, Shapley Value was design to solve a fairness problem in a completely different domain and therefore it is not clear why the same properties are required in this setting.

This study is presenting an interesting direction but is not ready yet for publication in NeurIPS.



Decision: Accept (Poster)