Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors create the Bayesian equivalents of LIME and SHAP that output credible intervals around explanations (of feature importance). By incorporating uncertainty, the authors create more stable explanations. Further, the credible intervals allow for a systematic way to set certain hyper-parameters (e.g., nr samples used to estimate feature importance). 

main_review: Strengths:
- The paper is generally well-written and clear 
- the proposed method is more stable (i.e., the explanation is less sensitive to small perturbations to the input) than its non-Bayesian counterpart

Weaknesses:
- limited novelty (see "Novelty" for a more in-depth comment)
- some aspects of the experiments could be improved (see "quality") 

--------------
Novelty/Related work 
- The paper proposes Bayesian equivalents of Lime and SHAP. These are constructed by using conjugate priors over the weights (and a noise term).  The novelty is somewhat limited as uncertainty-aware/Bayesian explanation methods already exist. 

--------------

Quality (methodology) 
- In general, the derivations and experiments seem sound. However, I have a couple of clarifying questions below.

Questions:
- Why is the hyperparameter sigma^2 the same for phi and eps in Equation 3? Given the two types of uncertainty modelled with these terms (feature and error), it's not directly clear to me why these two are the same.
- Table 1: Would it be possible to run this experiment on simulated data for which we can derive the "true value"? The estimate over 10,000 samples may be unable to find the true value (due to a methodological limitation). Further, could you give some qualitative examples of generated explanations (e.g., for MNIST)? 
- I would imagine that, e.g., error uncertainty should only be reducible to a certain extent (as our explanations cannot perfectly capture the local decision surface -- in particular if the neighbourhood of x cannot be modelled linearly). However, section 3.2 suggests that we can find an arbitrarily tight credible interval. How do you reconcile these two notions?

--------------
Related work:

Bykov et al. also investigate quantifying the uncertainty of explanations. It should be mentioned in the related work section.

S-LIME creates stabilized LIME explanations and may be of interest to include in the related work section. (Note: I believe this paper was posted after the NeuRIPS deadline, so this does not impact the score/evaluation.)

References:

Bykov, Kirill, et al. "How Much Can I Trust You?--Quantifying Uncertainties in Explaining Neural Networks." arXiv preprint arXiv:2006.09000 (2020).

Zhou, Zhengze, Giles Hooker, and Fei Wang. "S-LIME: Stabilized-LIME for Model Explanation." arXiv preprint arXiv:2106.07875 (2021).

--------------
Minor comments (did not affect evaluation):
- slight inconsistency between $\varepsilon$ and $\epsilon$ to denote the error term

------- 
Post-rebuttal update: 

The authors have addressed my main concerns in their rebuttal. Therefore, I've raised my score to a 7. 

limitations_and_societal_impact: Yes, the authors have discussed the impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper focuses on the development of a framework for generating explanations with associated estimates of uncertainty. The authors focus on two approaches from the XAI literature, LIME and KernelSHAP, in their evaluation. Their results are promising in terms of generating explanations with desirable attributes (consistent, stable, reliable, tractable).
main_review: This paper addresses the quite popular topic of explainable artificial intelligence (XAI); in particular, it focuses on deriving explanations from black box models, which are those that provide no access to their inner workings. The authors identify as challenges the fact that many approaches yield explanations that are of low quality in terms of correctness, reliability, and computational tractability. The main contributions are the development of a Bayesian framework for generating explanations in these black box settings that is capable of associating uncertainty with the results. In order to evaluate their framework, the authors instantiate the framework with two well-known approaches from the literature: LIME and KernelSHAP. An experimental evaluation shows that the approach yields promising results.

The paper is in general well written and well organized, and addresses an important topic that is very relevant to NeurIPS. As such, it should be of interest to a large audience. I have some fairly minor concerns that I would like the authors to comment on in the rebuttal phase:

-- What is the relationship to other approaches in XAI that don't necessarily suffer from all the failings identified for early approaches like LIME? For instance, some methods like BEEF (https://doi.org/10.1109/TCSS.2019.2902490) and some of the ones briefly discussed in the related work section don't seem to fall in this category. In short, could you please characterize more precisely when this approach is applicable? I think that the answer may lie at the distinction between local vs. global, which the authors briefly mention as part of future work.

-- Another, more minor, concern is that the development and evaluation of the approach is very centered on the two chosen approaches (LIME and KernelSHAP). Is the framework generalizable to others?

Minor errors and typos:

deployed in domain such as healthcare -->
deployed in domains such as healthcare

We carry out theoretical analysis -->
We carry out a theoretical analysis

A screen shot of the task is show -->
A screenshot of the task is shown
limitations_and_societal_impact: Not applicable
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Summary

Post-hoc explanations, such as from LIME or SHAP, can be unstable and it is unclear how reliable they are. This paper introduces a Bayesian framework to generating post-hoc explanations. The framework can provide information about how certain an explanation (e.g. a feature importance) is. 

Contributions

-	A Bayesian framework to estimate the certainty of post-hoc, local explanations
-	Instantiating the framework for LIME and SHAP and conducting experiments on various datasets
-	A method that samples perturbations more efficiently
-	A method that can estimate how many perturbations are needed to reach a user-specified level of confidence.

main_review: Strengths

-	The idea of adding uncertainty estimates to post-hoc explanation methods such as LIME and SHAP is interesting and an important consideration in order to improve the usability of such methods in practical scenarios.
-	The introduced Bayesian framework makes sense intuitively (Please note that I did not verify all equations/derivations) 

Weaknesses

Experimental results leave some questions open, i.e.:
-	One experiment to estimates the quality of uncertainty estimates measures how often the true feature importance lies within a 95% credible interval. However, the experiments uses pseudo feature importance because no true feature importance is available. The correctness of the pseudo feature importance relies on Prop 3.2 and a large enough perturbation value to be chosen. This makes it difficult to judge to what degree the experiment can be trusted because the difference between the tested method and the pseudo feature importance is only the number of perturbations. The experiment could be strengthened in two ways. 1.  set up a (toy) dataset where true feature importance is clearly defined. 2. What are the results when choosing BayesSHAP N=10k perturbations vs BayesLIME N=10k? These should be nearly identica; otherwise the assumption of the pseudo feature importance being (nearly) equal to true feature importance is compromised. Results could also be reported for BayesSHAP N=100 vs BayesLime N=10k and vice versa.
-	Correctness of Estimated Number of Perturbations: How can you be sure that G doesn’t simply overestimate? Maybe a value a lot less than G would have been sufficient?
-	Human Evaluation Experiment: Users were asked to guess a number from an image where the explanation was masked. Figure 8 indicates that each user was given30 such images and line 230 suggests that all 30 images where a masked version of number “4”. This seems like an unbalanced setup which make it difficult to determine the meaningfulness of this experiment.

Questions

-	User Study: Why did you decide to erase the explanations and measure failure to guess the correct image rather than keeping just the explanations and measuring success of guessing the correct image?
-	B.1: Why is it okay to make these assumptions?
-	Theorem 3.3: How do we know that S is large enough?

After Authors' Response
All my questions and weakness concerns were appropriately addressed, therefore I raised my score.

limitations_and_societal_impact: A "Broader Impacts and Limitations" section could be added that discusses potential dangers of trusting explanations. It could for example discuss that the choice of N is important or that explanations can still be wrong.
review_title: null

Meta-review: Reviewers unanimously recommend acceptance.  Authors are strongly encouraged to incorporate review comments and rebuttal discussion into their final revision.
Decision: Accept (Poster)