Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: SHAP computes feature attribution by estimating the expected change in model prediction when a feature is included into a random subset of the rest of features. To account for the "removed features", conventional SHAP employs a "global reference distribution" (typically the marginal or conditional global population) from which values are sampled for the missing (removed) features. This paper argues that such global reference distribution can be too coarse-grained as to understand the local behavior of model prediction. To increase local fidelity, the paper proposes Neighborhood SHAP which considers a more local reference distribution to sample from, which encompasses conventional SHAP as a special instance. The core idea of Neighborhood SHAP is the use of local reference distribution, which depends on not only the original marginal (or conditional) reference distribution considered in conventional SHAP but also a distance-based distribution that gives higher weights to values around the example to be explained. Such design allows Neighborhood SHAP to capture more fine-grained local behavior but also improves on-manifold explainability. The paper then discusses Smoothed SHAP (averaging SHAP around a local neighborhood) as a closely related variant. Experimental results demonstrate that Neighborhood SHAP increases local fidelity and possesses higher robustness to adversarial attacks, and Smoothed SHAP shows difference between attributions on the test instance and those on neighboring observations.
main_review: **Update**: Thank you to the authors for the well-thought responses, and conducting the suggested additional experiments. My questions have been addressed well, and I am happy to see the inclusion of additional discussions/experiments in the revision. I believe the paper cast an important notion on the "locality" of explanations, and has provided a comprehensive investigation on the topic.

Overall: The paper presents a clean framework to view current SHAP approaches, and the proposed Neighborhood SHAP is well-motivated. The formulation of Neighborhood SHAP naturally leads to a closely related variant, Smoothed SHAP. While the experimental section can be improved by more ablations and comparisons (see below), current presentation of the paper does provide insights on how Neighborhood/Smoothed SHAP capture different explanation characteristics compared to the original SHAP.

Strengths:
- First of all, the paper is clearly-written and the organization makes it easy to follow.
- The paper presents a comprehensive and clear overview of related work on SHAP. The overview provides a clean formulation on SHAP (line 62-63) and enables a clear viewpoint that current approaches to SHAP (marginal or conditional) are actually in a sense "global" reference distributions. By providing motivating examples illustrating the potential pitfalls of considering global reference distributions, the paper makes the problem of improving local fidelity well-posed.
- The proposed Neighborhood SHAP is well-motivated, and through importance sampling, it could be viewed as a generalization of the original SHAP. This makes Neighborhood SHAP a natural extension of SHAP.
- I appreciate that the authors have discussed how to choose the bandwidth for Neighborhood SHAP, as it could be critical to the success of Neighborhood SHAP.
- I appreciate that the authors have included discussion on Smoothed SHAP in the paper in addition to Neighborhood SHAP, as Smoothed SHAP appears to be a very closely related and natural variant to think of.
- Examples shown demonstrate the advantage of Neighborhood SHAP to the original SHAP: increased local fidelity and robustness to attacks. Interpretation on Neighborhood SHAP and Smoothed SHAP is also insightful.

Apart from the above, followings are some main question I have when reading the paper and suggestions on room to improve:
- In Figure 1, does decreasing the size of local neighborhood (e.g. N(0, 1) --> N(0, 0.1)) affect (improves) the results for LIME and SHAP? Would gradient-based methods be better at capturing this kind of local behavior? In general, how would Neighborhood SHAP compare to gradient-based methods like Integrated Gradient?
- In Figure 5, what is the exact metric (formal definition) used for the local prediction accuracy? How does the metric capture local fidelity better? How does Neighborhood SHAP compare to LIME on local fidelity (not shown in Figure 5)? How to explain why Neighborhood SHAP with small bandwidth can sometimes has better performance even when the neighborhood size of test data is large?
- I'd expect to see more discussion and comparison between NeighborhoodSHAP and Smoothed SHAP. Can SmoothedSHAP be formulated as a special instantiation of Neighborhood SHAP (or vice versa)? Empirically. how does Smoothed SHAP compare to NeighborhoodSHAP on robustness against attacks / on local fidelity / and on explanation stability (sensitivity)? These seem to be missing now, and they are important to understand the different characteristics between the two.
- Eq.2: Does Smoothed SHAP suffer from off-manifold problem?
- Figure 6, first column, NeighborSHAP and SmoothedSHAP has very different attribution (different sign), how to explain this discrepancy and which one to use?
- While the presented qualitative examples are illustrative, I believe more quantitative measurements on evaluating explanations (e.g., robustness to attacks or explanation sensitivity) could make the results much more convincing.
- Recent work has pointed out that explanations using reference distribution could subject to unintentional biases [1], does Neighborhood/Smoothed SHAP mitigate this problem?

Minor questions:
- In line 79-81, it is mentioned that the weighting of LIME does not ensure higher weights are assigned to model evaluations for observations closer to $x$. Why is this the case? Could the authors elaborate more on this?
- In line 85-87, why is SHAP not making any assumption on the form of $f$? SHAP does assume a linear additive model to fit the black-box model. Could the author discuss more on this?
- Equation between Line 104-105: why does $r(X|x)$ disappear in the last (rightmost) term?

[1] Evaluations and Methods for Explanation through Robustness Analysis. Hsieh et al. 2021.
limitations_and_societal_impact: The authors have pointed out some limitations of the work and how they managed to address them in the Discussion section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses an important problem in model-agnostic local explanation methods, i.e., using global distribution to simulate feature attributions for explaining local model behaviors. To this extent, the authors consider the formulation of neighborhood reference distributions that improve the local interpretability of Shapley values. Further, empirical evaluations show that Neighborhood Shapley values identify relevant feature attributions and improve on-manifold explainability and adversarial robustness against adversarial attacks.
main_review: Strengths
1. The paper provides a theoretical understanding of attribution-based explanation methods for analyzing local model behavior.
2. The paper is well-written with clear notations and motivations for NeighborhoodShap and SmoothedShap.
3. The paper shows that how smoothing Shapley values can identify unstable feature attributions and improve output explanations.

Open questions
1. One of the underlying assumptions to generate neighborhood samples using either the underlying data distribution process or from an estimator that approximates that distribution. In a black-box setting, we do not have any knowledge of the training data distribution, and for the estimator, there would be an error in approximating the underlying distribution. It would be great if the authors can comment on this.

2. How practical is the assumption that "models can detect the true data manifold?"

3. In the image classification results, Neighbourhood DeepSHAP looks at images in the neighborhood (i.e., images with similar stroke) which can be misleading. For instance, in Fig. 7, the neighborhood stroke for the top-left stroke of digit '8' can be similar to strokes of other digits (e.g., '2', '6', and '9').

4. The proposed methods are extensively evaluated using diverse metrics such as adversarial robustness, prediction accuracy, and visual inspections. It would be great if the authors can comment why they did not use more established metrics from the literature like deletion/insertion metric?
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors highlight on some of the flaws of SHAP for local explanations and propose a new variant, Neighbourhood SHAP, based on neighbourhood sampling and a kernel regressor.

Authors do a detailed theoretical and experimental analysis of the effects of the hyperparameters (i.g. kernel bandwith) of this technique and how they can be interpreted.
They consider several tabular datasets and one example on MNIST. They compare their insights with other common techniques (i.g. kernelSHAP).

Overall, authors further the theoretical results of interpretability tools.
main_review: Work is original, clear, and very extensive. Furthermore is provides theoretical insights to a relevant method in XAI. I think this work should be accepted and would improve with some minor edits.

Some points that authors should address and would make me consider to slightly improve their score (8 or 9):

* The title is pretty broad "On Locality of Local Explanation Models", when it actually only addressing a class of local explanation models ("SHAP").
* Authors might want to expand and contextualize their results within the broader family of XAI techniques like attribution/saliency techniques ( CAM/GradCAM, integrated gradients) or Layer-wise Relevance Propagation (LRP), especially since there are connections between these techniques and Shapley values [1].
* The MNIST example is mostly anecdotal, stronger results would be quantitative. There is already a wave of works that are attempting to evaluate the quality of explanations [2-7]. Adopting some of these ideas/frameworks would give stronger evidence for claims.

Authors should write their Societal Impact section.

Citations:
[1] https://arxiv.org/abs/1903.10992
[2] https://papers.nips.cc/paper/2020/hash/417fbbf2e9d5a28a855a11894b2e795a-Abstract.html
[3] https://arxiv.org/abs/1907.09701
[4] https://arxiv.org/abs/1904.11829
[5] https://arxiv.org/abs/2003.07258
[6] https://arxiv.org/abs/1806.10758
[7] https://arxiv.org/abs/1806.10758
limitations_and_societal_impact: Authors did not address it, they should be able to do it.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The main contributions of this paper are two variants of SHAP explanation method, Neighborhood SHAP and Smoothed SHAP. These two methods are motivated from the prior work in extending SHAP only captures the global behavior of the model but leaves the local behavior unexplored. The author demonstrates that the proposed explanations, neighborhood SHAP and Smoothed SHAP provide better local explanations over other SHAP methods in several data distributions. 
main_review: This is an interesting and complete work in general. The authors have provided detailed theoretical analysis in extending the SHAP method with a distribution that is more useful to capture the model’s local behavior. The technical part is sound. My main questions are about the motivation of Smoothed SHAP and some analysis in the experiment section. 

### Originality
The paper extends SHAP explantion with distributions that can capture the model’s local behaviors, which is the motivation for Neighborhood SHAP. However, the transition to Smoothed SHAP seems to be a little sudden such that even without Smoothed SHAP, Neighborhood SHAP seems to stand on its own. The only relation explained in the paper is they are related to the same technology, Nadaraya-Watson estimator, in different ways (line 161-162). It would be interesting to discuss how the way of treating neighborhood samplers as Nadaraya-Waston estimator provides the reader a better understanding behind Smoothed SHAP and Neighborhood SHAP. For example, is there some unified view of these two approaches? 

Other than that, the related work is well-cited and the novelty of this paper is significant. 

### Quality
The theoretical part is sound and the authors have provided very useful appendix content for the readers. I have several questions about the analysis of Neighborhood SHAP and Smoothed SHAP. 

- How do we choose between Neighborhood SHAP and Smoothed SHAP? When analyzing these two approaches at page 8, the baseline approach is mostly the previous SHAP methods, e.g. Marginal SHAP for both methods, respectively. But what should the human user trust more when represented with Neighborhood SHAP and Smoothed SHAP as there are still obvious gap between the results shown in Figure 6. When the relation between Neighborhood SHAP and Smoothed SHAP is not well-described in the previous section, it would be very useful to discuss which method is better in terms of local explainability and what is the suggested way in this paper to interpret result of the Neighborhood SHAP against Smoothed SHAP in an explicit way.

- Baseline choices in deep models. Comparing the neighborhood version and smoothed version of SHAP with LIME are reasonable choices for tree models or any models that are treated as black boxes. However, in the paragraph that shows Neighborhood DeepSHAP is more insightful than LIME seems to be less fair since DeepSHAP is based on DeepLIFT which has access to the model’s architecture and backpropagation while LIME is model-agnostic and therefore may not able to be faithfully capture the deep model’s behavior. The comparisons may be more fair if the baselines are also leveraging the gradient information, e.g. Integrated Gradient [1] and Smoothed Gradient [2]. 

### Clarity
The paper is well-organized. The writing is easy to follow even though there are some typos. The analysis in the experimental section is quite dense and there might be some ways to improve these paragraphs to make it easier to read. 
### Significance
This paper’s result is significant. Other researchers and especially the users of machine learning products may get benefit from the work. 


[1] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic Attribution for Deep Networks. ICML 2017
[2] Smilkov, D., Thorat, N., Kim, B., Viégas, F.B., & Wattenberg, M. (2017). SmoothGrad: removing noise by adding noise. ArXiv, abs/1706.03825.

limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: I thank the authors for a good back-and-forth discussion during the review process.  All reviewers recommend accepting this paper, appreciating the well-motivated method, the extensive complete original work, and the valuable the theoretical insights.  I encourage the authors to integrate all the suggestions from the reviewers into the camera ready.
Decision: Accept (Poster)