Title: On Locality of Local Explanation Models
Authors: Sahra Ghalebikesabi
Lucile Ter-Minassian
Karla DiazOrdaz
Christopher C. Holmes
Venue: NeurIPS 2021
Keywords: Shapley values
model interpretability
reference distribution
neighbourhood sampling

Abstract: Shapley values provide model agnostic feature attributions for model outcome at a particular instance by simulating feature 
absence under a global population distribution. The use of a global population can lead to potentially misleading results when local model behaviour is of interest. Hence we consider the formulation of  neighbourhood reference distributions that improve the local interpretability of Shapley values. By doing so, we find that the Nadaraya-Watson estimator, a well-studied kernel regressor, can be expressed as a self-normalised importance sampling estimator. Empirically, we observe that Neighbourhood Shapley values identify meaningful sparse  feature relevance attributions that provide insight into local model behaviour, complimenting conventional Shapley analysis. They also increase on-manifold explainability and robustness to the construction of adversarial classifiers. 

TL;DR: We consider the formulation of  neighbourhood reference distributions that improve the local interpretability of Shapley values.

Review discussion:
[Strengths]
1. This paper focuses on an important issue, i.e., the locality of explanation methods. The authors propose the Neighborhood SHAP and Smoothed SHAP to boost the locality of explanation results.
[Weaknesses]
1. The core strategy in this paper (i.e. sampling neighborhood samples to compute the baseline) is doubtful. In some cases, the neighborhood-based attribution may provide seemingly reasonable explanations, but theoretically, in other cases, the neighborhood-based attribution may have a significant risk of dramatically incorrect explanations.
For example, the experiment in Fig. 4 is a controversial example. For a certain perspective, the authors name some features as "unrelated column", because the DNN does not use these features for prediction when the input is not an OOD sample.
Consequently, the explanation results also assign a weak attribution with such so-called "unrelated" features.
However, we can also consider the exactly the same problem from another perspective. I.e., such so-called "unrelated" features have a strong correlation with whether a sample is OOD, which have been sophisticatedly encoded by the model. From this new perspective, a trustworthy explanation should reflect such correlation and assign these so-called "unrelated" features certain attributions.
More cucially, besides the above experiment, in real applicatoins, it is usually difficult for people to manually define "unrelated" features. In this case, the neighborhood-based explanation may have significant risk of incorrectly explaining an important feature as unrelated features.
In particular, when there is no clear boundary between OOD samples and normal samples (i.e., in some applications when all samples are valid samples without OOD ones).
In sum, in the general case, the logic of the DNN may be inconsistent with human recognition. I agree that in the special case of Fig. 4, the proposed method works well.
However, theoretically, there is a great risk to use this method for other applications.
2. In Line 49, the formula of P(S) is incorrect. It is supposed to be $P(S)=(|S|!(m-|S|-1)!)/m!$.
3.
Another key problem is as follows. In Line 117, authors claim that all Shapley axioms still hold for the Neighborhood SHAP.
However, there is no detailed proof in the paper or in the supplementary material.
Thus, it is still unknown whether the Neighborhood SHAP satisfies axioms, including the linear axiom, the symmetry axiom, the nullity axiom, and the efficiency axiom. Please formally prove this issue. Without such axioms, the trustworthiness of the explanation is doubtful.
4. The contribution of this paper is incremental. Lee et al. [11] have proposed to use conditional distribution to define the feature importance. The proposed Neighborhood SHAP can be considered as a special case in [11].
Therefore, this paper only provides an incremental contribution.
5. Authors prove that the Smoothed SHAP is captured by a Lipschitz constant.
However, if a model is not bounded by Lipschitz and results of the Smoothed SHAP are captured by a Lipschitz constant, then is the explanation result still reliable?
6. In experiments, authors evaluate the Neighborhood SHAP and previous methods mainly based on human intuition.
However, it is not reasonable to assume that DNNs model knowledge in the same way as humans.
Therefore, evaluation metrics are suggested to be applied in experiment.
7. What do “feature=1” and “feature=2” mean in Figure 1? Should they be “Feature-1” and “Feature-2”?
Besides, there is no label or number for the y-axis in the second sub-figure of Figure 1. Authors are suggested to correct this figure.
8. Minor:
(1) The paper is difficult to read due to the lack of rigorous and clear definitions of symbols.
For example, in Line 55, what does f refer to?
(2) In Line 225, the bandwidth $\sigma$ can be misleading, since $\sigma$ is also used to define the Neighborhood SHAP and the Smooth SHAP.
(3) Some quotation marks in Lines 280-299 are mistakenly written.
No
2. **"In Line 49, the formula of P(S) is incorrect."**
Thank you for noting this typo; it has been amended.
3. **"Another key problem is as follows. In Line 117, authors claim that all Shapley axioms still hold for the Neighborhood SHAP.
However, there is no detailed proof in the paper or in the supplementary material.
Thus, it is still unknown whether the Neighborhood SHAP satisfies axioms, including the linear axiom, the symmetry axiom, the nullity axiom, and the efficiency axiom. Please formally prove this issue. Without such axioms, the trustworthiness of the explanation is doubtful."**
Neighborhood SHAP satisfies the SHAP axioms (see Appendix B, ll. 64-75), as these are formulated conditional on the value function.
As a result, they hold for any choice of value function, and in particular for expectations over neighbourhood distributions as explained in Appendix B, ll. 76-78. We refer to the proof in l. 117. We have now phrased the proof more explicitly in the revised version of the appendix.
4. **"The contribution of this paper is incremental. Lee et al. [11] have proposed to use conditional distribution to define the feature importance. The proposed Neighborhood SHAP can be considered as a special case in [11].
Therefore, this paper only provides an incremental contribution."**
We are assuming that the reviewer is referring to Covert et al, which is listed as paper [11] in our work. This paper uses conditional reference distributions to compute global feature attributions, which is outside of the scope of our paper, as we only study local feature attributions.
Furthermore, Neighbourhood SHAP extends both conditional as well as marginal SHAP values: Both conditional and marginal SHAP values impute the removed features by sampling from a global reference distribution that does not consider the distance to the test instance $x$.
When distances in the feature space are considered, marginal SHAP values become marginal Neighbourhood SHAP values, and conditional SHAP values become conditional Neighbourhood SHAP values.
As such Neighbourhood SHAP is not a special case of conditional SHAP values or the global model presented by [11].
Furthermore, marginal Neighbourhood SHAP values combine the benefits of conditional SHAP values (i.e. on-data-manifold model evaluation) and the benefits of marginal SHAP values (see [2]).
5. **"Authors prove that the Smoothed SHAP is captured by a Lipschitz constant.
However, if a model is not bounded by Lipschitz and results of the Smoothed SHAP are captured by a Lipschitz constant, then is the explanation result still reliable?"**
Lipschitz continuity has been established as a favorable characteristic of explainability tools [6]. Introducing smoothness can indeed lead to feature attributions that do not correctly reflect the model's behaviour. In certain circumstances, this disadvantage can be outweighed by the advantage of more "intuitive" explanations, i.e.
when used to "explain to justify" [14]. We included this clarification in the paper.
6. **"In experiments, authors evaluate the Neighborhood SHAP and previous methods mainly based on human intuition.
However, it is not reasonable to assume that DNNs model knowledge in the same way as humans.
Therefore, evaluation metrics are suggested to be applied in experiment."**
We especially thank Reviewer iiTd for this suggestion. The submitted version of the paper contained the following quantitative evaluation metrics:
* Local prediction MSE, adapted from [13], for performance measurement in Figure 5.
* Robustness to OOD detection in Figure 4a.
* Change in log-odds when masking with SHAP values as proposed by [8] in Figure 7b in Appendix K. As Yang et Kim [12] note, this kind of sensitivity measure is a useful evaluation metric.
* Local Lipschitz estimates as a robustness metric computed on 100 test points on the MNIST dataset as proposed by [3] in Figure 8a in Appendix K.
* Local Lipschitz estimates on maximally perturbed MNIST images as proposed by [3] in Figure 8b in Appendix K.
We note that it is common practice in the literature on SHAP values to mainly rely on human intuition, as there is no ground truth, and the ultimate goal is to provide explanations for humans. This is the case in the literature on local explanation models, including all methods we compare our approach with [3, 8-10].
As we appreciate the suggestion of the reviewer, we added an evaluation metric taken from the ROAR framework developed by Hooker et. al [1], which is a recently proposed deletion metric. For each observation, 20% of the most important features (measured by the magnitude of the absolute feature attributions) are imputed by their mean and the model is retrained.
Eventually, we compare the resulting changes in predictive accuracy. We distinguish between two settings: imputation with the (global) mean of the features and imputation with the local mean (i.e. the weighted mean of each feature where the weights were computed based on a distance metric to the observation).
As we see in the table below, Neighbourhood and Smoothed SHAP result in a higher absolute change in test performance than Marginal SHAP on the boston housing data set. These are preliminary results and further experiments (additional runs for error bars and additional data sets) are currently running. Results will be added to the appendix.
|   | Absolute change in test $R^2$ when imputing with local mean| Absolute change in Test $R^2$ when imputing with (global) mean
--- | --- | --- 
|SHAP|0.349|0.154|
|Neighbourhood SHAP|0.389|0.320|
|Smoothed SHAP|0.177|0.399|
7. **"What do “feature=1” and “feature=2” mean in Figure 1? Should they be “Feature-1” and “Feature-2”?
Besides, there is no label or number for the y-axis in the second sub-figure of Figure 1. Authors are suggested to correct this figure."**
Figure 1 was amended following the reviewer’s suggestions.
8. **"Minor: (1) The paper is difficult to read due to the lack of rigorous and clear definitions of symbols.
For example, in Line 55, what does $f$ refer to? (2) In Line 225, the bandwidth $\sigma$ can be misleading, since $\sigma$ is also used to define the Neighborhood SHAP and the Smooth SHAP. (3) Some quotation marks in Lines 280-299 are mistakenly written."**
The notation $f$ relates to the black-box function and is defined in l. 17. In l. 225, $\sigma$ refers to Neighborhood SHAP and has now been replaced with $\sigma_{nbrd}$ for clarity and consistency. We also added further clear definitions for other notations.
### References
[1] Hooker S, Erhan D, Kindermans PJ, Kim B. A benchmark for interpretability methods in deep neural networks. arXiv preprint arXiv:1806.10758. 2018 Jun 28.
[2] Janzing D, Minorics L, Blöbaum P. Feature relevance quantification in explainable AI: A causal problem. InInternational Conference on Artificial Intelligence and Statistics 2020 Jun 3 (pp. 2907-2916). PMLR.
[3] Merrick L, Taly A. The explanation game: Explaining machine learning models using shapley values. InInternational Cross-Domain Conference for Machine Learning and Knowledge Extraction 2020 Aug 25 (pp. 17-38). Springer, Cham.
[4] Aas, K., Jullum, M., and Løland, A. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. arXiv preprint arXiv:1903.10464, 2019.
[5] Chen, H., Janizek, J. D., Lundberg, S., and Lee, S.-I. (2020). True to the model or true to the data? arXiv preprint arXiv:2006.16234.
[6] Alvarez-Melis, D. and Jaakkola, T. S. (2018). On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049.
[7] Sundararajan, M. and Najmi, A. The many shapley values for model explanation. arXiv preprint arXiv:1908.08474, 2019.
[8] Lundberg, S. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874.
[9] Bhargava, V., Couceiro, M., and Napoli, A. (2020). Limeout: An ensemble approach to improve process fairness. arXiv preprint arXiv:2006.10531.
[10] Botari, T., Hvilshøj, F., Izbicki, R., and de Carvalho, A. C. (2020). Melime: Meaningful local explanation for machine learning models. arXiv preprint arXiv:2009.05818.
[11] Covert, I., Lundberg, S., and Lee, S.-I. (2020). Understanding global feature contributions with 344 additive importance measures. Advances in Neural Information Processing Systems, 33.
[12] Yang M, Kim B. Benchmarking attribution methods with relative feature importance. arXiv preprint arXiv:1907.09701. 2019 Jul 23.
[13] Plumb, Gregory, Denali Molitor, and Ameet Talwalkar. "Model agnostic supervised local explanations." arXiv preprint arXiv:1807.02910 (2018).
[14] Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE access, 6, 52138-52160.
[15] Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020, February). Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 180-186).
[16] Frye, C., de Mijolla, D., Begley, T., Cowton, L., Stanley, M., & Feige, I. (2020). Shapley explainability on the data manifold. arXiv preprint arXiv:2006.01272.
[17] Hsieh, C. Y., Yeh, C. K., Liu, X., Ravikumar, P., Kim, S., Kumar, S., & Hsieh, C. J. (2020). Evaluations and methods for explanation through robustness analysis. arXiv preprint arXiv:2006.00442
1. **"The core strategy in this paper (i.e. sampling neighborhood samples to compute the baseline) is doubtful. In some cases, the neighborhood-based attribution may provide seemingly reasonable explanations, but theoretically, in other cases, the neighborhood-based attribution may have a significant risk of dramatically incorrect explanations.
For example, the experiment in Fig. 4 is a controversial example. For a certain perspective, the authors name some features as "unrelated column", because the DNN does not use these features for prediction when the input is not an OOD sample.
Consequently, the explanation results also assign a weak attribution with such so-called "unrelated" features.
However, we can also consider the exactly the same problem from another perspective. I.e., such so-called "unrelated" features have a strong correlation with whether a sample is OOD, which have been sophisticatedly encoded by the model. From this new perspective, a trustworthy explanation should reflect such correlation and assign these so-called "unrelated" features certain attributions."**
Our approach is explicit, focussing SHAP with local reference neighbourhoods, and open to interpretation as such. We don't follow the notion that it can provide theoretically incorrect explanations.
The experimental set-up in Fig. 4 has been adapted from a well-established paper [15]. We note that the unrelated feature exhibits no correlation with the sensitive attribute, and is also not indicative of whether an observation is OOD (i.e. there is no correlation between the value of the unrelated feature and whether an observation is OOD).
However, the adversary's unbiased classifier depends on the unrelated feature when an observation is OOD.
[15] makes a strong argument why we do not want the model behaviour off the data manifold to be reflected in the feature attributions. They state that in a real-world setting, any observation that we may want to explain is on the data manifold. The adversary is in fact a biased model that takes race into account every time a criminal is judged. The user should not be distracted by the behaviour of the model on unrealistic observations. Please see [16] for more arguments on why this property is desirable. [17] explicitly mentions that imputation with values that "are too far" induces bias in the model explanations.
**"More cucially, besides the above experiment, in real applicatoins, it is usually difficult for people to manually define "unrelated" features. In this case, the neighborhood-based explanation may have significant risk of incorrectly explaining an important feature as unrelated features.
In particular, when there is no clear boundary between OOD samples and normal samples (i.e., in some applications when all samples are valid samples without OOD ones).
In sum, in the general case, the logic of the DNN may be inconsistent with human recognition. I agree that in the special case of Fig. 4, the proposed method works well.
However, theoretically, there is a great risk to use this method for other applications."**
"Unrelated" features are those features that are not used for any model evaluations of observations that lie on the data manifold. In a case where there are no OOD observations, the unrelated features are thus never used. Marginal Neighbourhood SHAP will then correctly not assign any attribution to the unrelated features. All other features, which are used by the model, will get a positive absolute attribution on average over multiple samples. Again, this property is desired, see e.g. [2]. In the case where the boundary between OOD samples and in-distribution samples is not well-separated, it might happen that also model behaviour off the data manifold is captured. The choice between marginal or conditional reference distribution for on- and off-data-manifold explainability is an area of active research [2-7], but it is not the central focus of our contribution. The proposed approach as neighbourhood distributions can be used with either marginal or conditional imputation.
The manuscript is well structured and the method is motivated convincingly. I also very much enjoyed the background section as it was helpful even as I am not an expert in Shapley values. Using only a local population density seems very natural. The experimental section seems convincing and appears to demonstrate the advantages of the method on several datasets (MNIST,  UCI) and problem settings (robustness, visual interpretiblity, prediction accuracy).
A few suggestions for improving the manuscript are in order:
-I would encourage the authors to use more consistent mathematical notation: In the manuscript, expectation values are labeled by their random variable (see e.g. L101-102 outer expectation), the corresponding density (see e.g. L104-105 left), or both (L101-102). It would be better to be consistent or denote dependence on random variable by S \sim P(S).
Another example of this is in L61 where \bar{S} \ j should be replaced by \bar{S} \{j} as j is not a set.
-I also found it a bit hard to find the proofs for the propositions and theorems in the Supplement. It would be advisable to add a clear reference to the relevant section in the Supplement to the main text.
For example, I was not able to locate a proof for Proposition 1 in the Supplemental.
-Several pdf viewers that I tried did not correctly display some of the figures in the manuscript. This issue has not arisen with any other of the papers I have reviewed and could therefore be specific to the draft at hand. The authors may want to check this.
For example, I had issues using Preview on an iPad.
A few questions for the authors:
(a)It would seem to me that the normalization constant n_c  of n in L97 can be given in closed form if the distance D of L96 is the standard Euclidean distance (as d is a Gaussian). In this case, estimating the expectation value in L101-102 by importance sampling would appear to be not necessary. Is IS used since other distances such as a Random Forest-based one is used?
(b)Proposition 1 states that the Nadaraya-Watson estimator  is an unbiased estimator. Generically, self-normalized IS only leads to *asymptotically* unbiased estimators. Why is this not the case here?
(c)Theorem 2 proves Libschitz continuity of the smoothed estimator. Is there any reason to expect that the Libschitz constant is small for reasonable choices of \sigma_smtd?
No
**"I would encourage the authors to use more consistent mathematical notation: In the manuscript, expectation values are labeled by their random variable (see e.g. L101-102 outer expectation), the corresponding density (see e.g. L104-105 left), or both (L101-102). It would be better to be consistent or denote dependence on random variable by S \sim P(S).
Another example of this is in L61 where $\bar{S} \setminus j$ should be replaced by $\bar{S} \setminus \{j\}$ as $j$ is not a set."**
We thank Reviewer dkVj for pointing out the notational inconsistencies which we have now fixed.
**"I also found it a bit hard to find the proofs for the propositions and theorems in the Supplement. It would be advisable to add a clear reference to the relevant section in the Supplement to the main text.
For example, I was not able to locate a proof for Proposition 1 in the Supplemental."**
Clear references to the proofs of all theorems and propositions were added in the main part, and theorem statements have been included in the appendix. The proof of Proposition 1 follows from the discussion in the paragraph right before its statement. We have now also included it explicitly in the appendix.
**"Several pdf viewers that I tried did not correctly display some of the figures in the manuscript. This issue has not arisen with any other of the papers I have reviewed and could therefore be specific to the draft at hand. The authors may want to check this.
For example, I had issues using Preview on an iPad."**
We apologise for the difficulties with viewing the figures. They were included as PDFs in the paper to minimise the loss in quality. The Firefox explorer on Linux machines seems to render the figures correctly. We have now decreased the size of the figures to allow them also to be viewed on iPads.
Below, we respond to the questions
**"a) It would seem to me that the normalization constant n_c of n in L97 can be given in closed form if the distance D of L96 is the standard Euclidean distance (as d is a Gaussian). In this case, estimating the expectation value in L101-102 by importance sampling would appear to be not necessary. Is IS used since other distances such as a Random Forest-based one is used?"**
IS is used to approximate the intractable integral over a target distribution $p(z)$ by sampling from a proposal distribution $q(z)$. In the case of Neighbourhood SHAP, the target distribution is defined by $p(x^*_{\bar{S}})=n(x^*_{\bar{S}}|x) = n_c \cdot d(x^*_{\bar{S}}|x_{\bar{S}}) \cdot r(x^*_{\bar{S}}|x)$ (l. 97). The normalization constant is here given by $n_c=1/\int d(x^*_{\bar{S}}|x_{\bar{S}}) \cdot r(x^*_{\bar{S}}|x) dx^*_{\bar{S}}$. As the reviewer correctly noted, this  fraction can be estimated empirically.
However, it does not make the use of importance sampling redundant, as its estimation does not faciliate sampling from the target distribution $n(x^*_{\bar{S}}|x)$ when we only have access to the empirical proposal distribution $q(x^*_{\bar{S}})=r(x^*_{\bar{S}}|x)$.
When the normalisation constant cannot be computed, it is necessary to opt for self-normalised IS, as proposed in the paper, instead of IS.
**"b) Proposition 1 states that the Nadaraya-Watson estimator is an unbiased estimator. Generically, self-normalized IS only leads to asymptotically unbiased estimators. Why is this not the case here?"**
We thank the reviever for pointing us to this mistake in the paper.
Indeed the NWE is only a consistent estimator, as is self-normalised IS.
**"c) Theorem 2 proves Libschitz continuity of the smoothed estimator. Is there any reason to expect that the Libschitz constant is small for reasonable choices of $\sigma_{smtd}$?"**
The Lipschitz constant is upper bounded by the maximal value of Smoothed SHAP. For $\sigma_{smtd}\rightarrow\infty$, it is thus upper bounded by the average of the SHAP values over all training observations.
While it is possible that this bound is not sufficiently low, it is lower than the corresponding bound for SHAP values if these are not constant across observations.
Thank you very much for the detailed reply.
I would encourage the authors to add a reference for the fact that self-normalized importance sampling leads to consistent estimator as this might not be a well-known fact in the explainability community.
Overall, the clarifications were helpful and are very much appreciated.
Thank you very much for the positive feedback. We have now added a reference to Theorem 9.2 of Owen (2013) in the revised version of the paper.
### References
Owen, A. B. (2013). Monte Carlo theory, methods and examples.
1) Notational style: This is minor, but still things like putting the random variable exactly underneath the expectation sign $\mathbb{E}$ makes the notation a bit unappealing. Or writing $S\setminus j$ rather than $S\setminus \{j\}$.
2) What do you mean by global population specifically? can it be defined more specifically before you start using it through out the paper?
2) line 74, where are $L,C$ notation that are used in the equation come from?
3) I had a hard time finding the proof of theorems in the appendix, for instance theorem 2, is supplement G. Make sure to add the theorem statement in the supplement or change the title of section G to be more indicative of the theorem.
4) In regard to theorem 2 proof, when you use the monotonicity of exponential and triangular inequality, $\delta = ||z-y||^{2}$ so it should not be $\delta^{2}$, you probably can fix it by defining $\delta^{2} = ||z-y||^{2}$. In the denominator of the same equation, you are wrote things like $-||x_{j} - y||$ instead of $||x_{j} -y ||^{2}/\sigma^{2}$. The 3rd line from below of the same page when you replace part of your equation with $\hat{\phi}^{\textit{smooth}}$ should not the normalization part also be part of it based on your equation (2) in the main paper?
5) Experiments: I like your experimental setup, the only concern that I have is running explanation methods on most of UCI data and MNIST sometimes is not enough to establish a convincing argument due to the simplicity of the data and as a consequence simplicity of prediction models.
For instance, linear methods like lasso works perfectly fine (above 90 percent accuacy) on MNIST data set. It would've been nice to see result on movie review like IMDB, or CIFAR 10 which is a common datas for explanation.
No
1. **"Notational style: This is minor, but still things like putting the random variable exactly underneath the expectation sign makes the notation a bit unappealing. Or writing $S\setminus j$ rather than $S\setminus j$."**
Thank you for the suggestions regarding the notational style which we have incorporated in the revised version of the paper.
2. **"What do you mean by global population specifically? can it be defined more specifically before you start using it through out the paper?"**
In line 65, we specify that global statistical populations refer to the marginal/conditional distributions which do not incorporate distance metrics, as used by marginal/conditional SHAP. We now made this definition explicit in the introduction of the revised version of the paper.
2. **"In line 74, where are $L, C$ notation that are used in the equation come from?"**
In line 75, $L$ is defined as the number of draws from the reference population. In line 76, $C$ is defined as the number of sampled coalitions. We stress the corresponding definitions explicitly in the revised version.
3. **"I had a hard time finding the proof of theorems in the appendix, for instance theorem 2, is supplement G. Make sure to add the theorem statement in the supplement or change the title of section G to be more indicative of the theorem."**
The title of section G has been changed and clear theorem statements have been added at the beginning to each proof. References for each proof have been added in the main part of the paper.
4. **"In regard to theorem 2 proof, when you use the monotonicity of exponential and triangular inequality, $\delta=||z-y||^2$ so it should not be $\delta^2$, you probably can fix it by defining $\delta^2=||z-y||^2$. In the denominator of the same equation, you wrote things like $-||x_j-y||$ instead of $||x_j-y||^2/\sigma^2$. The 3rd line from below of the same page when you replace part of your equation with $\hat{\phi}^{smooth}$ should not the normalization part also be part of it based on your equation (2) in the main paper?"**
We are grateful for this observation. The proof has been modified accordingly, and the Lipschitz constant has been updated.
5. **"Experiments: I like your experimental setup, the only concern that I have is running explanation methods on most of UCI data and MNIST sometimes is not enough to establish a convincing argument due to the simplicity of the data and as a consequence simplicity of prediction models.
For instance, linear methods like lasso works perfectly fine (above 90 percent accuacy) on MNIST data set. It would've been nice to see result on movie review like IMDB, or CIFAR 10 which is a common datas for explanation."**
We refrained from including experiments on advanced image data sets since the literature on local explanation models, including the methods we compare to, is most often illustrated on tabular data sets and MNIST images [1-6]. The paper includes an experiment on the COMPAS data set, besides the analysis of UCI data sets.
Additionally, we have now added an experiment on the IMDB data set in the Appendix to illustrate the performance of Neighbourhood SHAP on more complex data sets.
### References
[1] Merrick L, Taly A. The explanation game: Explaining machine learning models using shapley values. InInternational Cross-Domain Conference for Machine Learning and Knowledge Extraction 2020 Aug 25 (pp. 17-38). Springer, Cham.
[2] Alvarez-Melis, D. and Jaakkola, T. S. (2018). On the robustness of interpretability methods. arXiv preprint arXiv:1806.08049.
[3] Bhargava, V., Couceiro, M., and Napoli, A. (2020). Limeout: An ensemble approach to improve process fairness. arXiv preprint arXiv:2006.10531.
[4] Botari, T., Hvilshøj, F., Izbicki, R., and de Carvalho, A. C. (2020). Melime: Meaningful local explanation for machine learning models. arXiv preprint arXiv:2009.05818.
[5] Chen, H., Janizek, J. D., Lundberg, S., and Lee, S.-I. (2020). True to the model or true to the data? arXiv preprint arXiv:2006.16234.
[6] Lundberg, S. and Lee, S.-I. (2017). A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874.
1. Local interpretability of attribution models is an interesting topic and can be useful in multiple applications.
2. This paper proposes a new method to interpret local model behavior using Nadaraya-Watson estimator. The approach is novel.
3. This paper is well written and clearly conveys the idea and insights in great detail.
4. Simulated examples are useful to demonstrate the idea while real data results show the generalizability to real data problems.
No
1. **"TreeSHAP is one of the latest developments in the line of SHAP when tree ensemble models are used for prediction. It also explains local model behavior via split of trees. It would be interesting to see a comparison between the proposed method and TreeSHAP."**
We did not include TreeSHAP in our comparison,  as it focuses on tree-based ensembles, and our aim is to consider general, model-agnostic, explanation approaches.
Further, our method requires only black box (input-output) access to the model to be explained, while TreeSHAP requires access to the internal parameters, e.g. splits of the trees.
We agree that it is interesting to consider explainability methods for particular model classes, which is why we included an experiment on Deep SHAP (a fast computational approximation of SHAP for neural networks) in the original submission as an example. We have now also added the attributions from TreeSHAP to our plots for completion (Figure 6 in the main paper, and Figures 11, 17 and 23 in the Appendix).
2. **"Computation burden of LIME and KernelSHAP is too high when sample size is large. It’s necessary to justify the computational burden of the proposed method in the main body and compare it with fast methods such as TreeSHAP."**
As the reviewer noted, TreeSHAP is a computationally efficient approximation of SHAP for tree-based approaches.
As a result, it is indeed possible to define Neighbourhood TreeSHAP as an approximation of Neighbourhood SHAP to profit from the computational benefits of TreeSHAP. We leave its exploration for future research.
All in all, the use of neighbourhood distributions thus does not lead to increased computational complexity.

Meta-review:
role: PC
data: comment: The paper discusses local feature importance and presents a limitation of current methods: they are, in a sense, not local “enough”. This is an important observation that should be studied. The authors present a possible solution to this problem by adding geometric and distribution-related components to the valuation function and show how this changes the feature importance score.

We had active discussion about this work. It was agreed that the problem presented is well justified. However, there were concerns about the proposed solution and therefore we conclude that it be more beneficial to reject the paper and encourage the authors to further study the problem. Here are some examples to questions that the authors may wish to consider when further exploring this domain:

-	How would the explanation behave for points that are relatively remote? Note that the reviewers have commented that the explanation could fail off the manifold and it is important to mention this in the final version of this work.
-	How would the explanation behave in high dimension when the Euclidean distance might be altered by many non-relevant features or features with different scale?
-	How should we measure how good is the explanation? I.e., if someone proposes a different way to provide local explanations, how would we know which one is better? In other words, how do we know that the feature importance score presented here is good?
-	Do the properties of Shapley Value match the requirements in the case? After all, Shapley Value was design to solve a fairness problem in a completely different domain and therefore it is not clear why the same properties are required in this setting.

This study is presenting an interesting direction but is not ready yet for publication in NeurIPS.




Decision: Accept (Poster)