Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper is focused on the off-data manifold problem with Shapley values which is created by sampling data that is out of distribution. The goal is to develop efficient methods. Two main algorithms are proposed: Generative models to approximate conditional distributions and training supervised models for direct approximation. They show an advantage against original off-manifold Shapley values in experiments. The impression experiment is particularly interesting.
The overall idea of using generative models to solve the issue of off-manifold data in interpretability methods (not just SHAP) is generally a nice direction. Note that the problem is more prominent for the case of SHAP as the method is built on performance on all subsets of the input features and the paper makes a good case of showing the necessity of solving the off-manifold data problem for Shapley-based methods. The experimental results are also comprehensive and provide enough evidence for their usefulness.
 There seem to be novelty concerns "A performant method to estimate onmanifold Shapley values on general data is until-now lacking and a focus of this work." The problem of off-manifold data in interpretability is well studied and its specifics for the SHAP method have been discussed before https://proceedings.icml.cc/static/paper_files/icml/2020/334-Paper.pdf The authors make a very brief reference to this paper but do not actually mention how they are different. Unless the main differential contribution of the work to the existing literature is clear, I cannot change my score. If the contribution is ""experimental evidence ... in favour of the onmanifold approach", the contribution is not enough for this venue.

Questions and notes:
* The higher dimensional example is not actually high dimension and it does not give a sense of how the method would be applied to cases like ImageNet. Authors need to provide runtime results to make the case for the practical efficiency of their method. 
* There does not seem to be a discussion of the effect of the generative model's performance on the results. Having a good generative model that actually captures conditional probabilities for high dimensional data like images is still an open question and therefore, it will hinder the proposed method's performance.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper explores the drawbacks of existing explanation methods that implicitly use off-manifold samples while holding out groups of features. Most existing SHAP variants (e.g., KernelSHAP) use off-manifold samples, in part because using on-manifold samples from the conditional distribution is computationally challenging. This work proposes two methods to do so tractably (one "unsupervised," one "supervised") and demonstrates some advantages of the on-manifold approach.

I appreciated many results from this paper. It was valuable to explain that when using the on-manifold approach, two models that make identical predictions get identical SHAP values. It was also good to see that the on-manifold approach successfully detects dependencies on sensitive attributes where the off-manifold approach fails (Figure 2). When comparing the various approaches, showing that the on-manifold approach is roughly equivalent to retraining $2^n$ models is helpful, and showing that TreeSHAP does not perfectly model the conditional distribution corrects a potential misconception that this approach is equivalent to the on-manifold approach. 

A couple of questions and concerns about the methods and theory in this work. 

1. The perspective of the data distribution being controlled by unobserved latent variables and noise variables was not helpful, in my view. It was not clear what the authors meant by $O(\epsilon)$, it was not clear why we should assume that the relationship between $x$ and $z$ is invertible (see $\Psi_k$) and it was not clear when/why we should assume that each $x_i$ is a deterministic function of the other features (see $\tilde \Psi$). The math in this section (beginning of 3.1) seemed like another way of saying that data samples should come from dense parts of the data distribution, but without adding much additional insight. (I'm open to changing my opinion on this if others disagree.)
2. The metric for the on-manifold value function was not explained clearly. In what sense is this value function optimal? Is it necessary to use MSE to train a supervised classification model? I believe a more complete proof of this result is provided in Appendix A of Covert et al. (cited), and this result suggests how to train a supervised surrogate using cross entropy loss.
3. The comparison of the supervised and unsupervised approach is very important because it informs how future work should use the on-manifold method. I understand the space constraints, but if possible some of these results could be shown in the main text. Table 1 in the supplement seems to show that the supervised approach is vastly superior according to the proposed metric; the authors should also include a comparison with the off-manifold approach (using the marginal distribution) to give a sense of how significant the difference is. It would also be interesting to see the standard error comparison mentioned on p.8, which suggests that the supervised approach converges much faster.

A couple presentation concerns:

- Removing features using their conditional distribution is more specific than using on-manifold samples, yet this is what the authors seem to mean by "on-manifold." (It is possible to use samples from a different distribution that shares the same support as the conditional distribution.) The authors may wish to clarify this ambiguity and possibly explain more specifically why the conditional distribution is appropriate. 
- The presentation is often specific to classification models, but this seems unnecessary. Readers can probably translate the ideas to regression models on their own, but the authors may consider broadening their presentation.
- The "non-parametric limit" is an odd way of presenting the result in Eq. 9 (in my view, although others may disagree). In what sense is the model converging? To be more clear, the authors may consider explaining that this result requires the optimal model (i.e., the Bayes classifier) and that this should only be expected given unlimited data and a sufficiently flexible model. 
- When discussing the "unsupervised approach," it may be worth saying explicitly that this does not directly provide an estimate of the value function, but it provides the means to estimate it via a Monte Carlo approximation. The authors may also wish to point out that this means the unsupervised approach requires far more model evaluations than the supervised approach to evaluate any given coalition.
- What exactly is the Shapley distribution used in Eq. 16? I couldn't find where this was explained.
 
The paper has a couple missing citations, although they don't particularly diminish the work's novelty. Slack et al. [3] show that off-manifold methods can be fooled into missing dependencies on sensitive attributes; this could fit into the narrative in a couple parts of the paper. The "unsupervised approach" is just learning to sample from arbitrary conditional distributions, and there's more work on this topic than the Ivanov et al. paper. Douglas et al. [2] have a method called the "universal marginalizer" and Belghazi et al. [1] have a method called the "neural conditioner" (NC) that has been shown to outperform the AC-VAE. The authors may wish to comment why their approach is modeled off of the AC-VAE rather than the more recent NC.

Overall, I think that this work proposes a solution to an important problem (holding out features using their conditional distribution) and I believe it could be impactful. However, I also think the paper's presentation could be improved in the ways described above. 

[1] Belghazi et al., "Learning about an exponential amount of conditional distributions" (2019)

[2] Douglas et al., "A Universal Marginalizer for Amortized Inference in Generative Models" (2017)

[3] Slack et al., "Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods" (2019)

##########
Update
##########

The authors have addressed many of my concerns. Moving Table 1 into the main text and adding the new columns is an important change, and it seems that the authors have improved various aspects of their presentation. I'm raising my score to a 7.
review_title: New approach for calculating SHAP values while holding out features with the conditional distribution

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper gives thorough intuitions, theoretical arguments and synthetic examples that demonstrate the shortcomings of off-manifold Shapley values. It then proposes two approaches to overcome this limitation. Specifically, they enable scalable sampling from the on-manifold conditionals. The first approach is unsupervised based on VAEs and model-agnostic. The second approach builds upon the supervised model. The latter is shown to be more accurate and data efficient. Both approaches are shown to fix the limitations of off-manifold Shapley values on a real-world MNIST example. The authors provide implementation detail that will be useful for practitioners.
Thanks for this a well-written, well-motivated easy-to-follow paper.

Minor comments.
- add to the intro a defintion of "data splicing"
- formalize problem of off-manifold shapley values
- It would be great if the authors released code for their experiments.
review_title: In this well-designed and motivated study the authors highlight shortcoming of existing methods for computing Shapley values. They propose two methods for mitigation that enable sampling form estimate on-manifold conditionals. The methods are shown to be effective.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Paper Summary
In this paper, the authors pointed out that the explanation with Shapley value can be misleading due to inappropriate assumptions in the current estimation method. The authors pointed out that the "splicing" techniques used in the current estimation method assumes the variables to be independent, which is not always the case in practice.  Application of "splicing" to dependent variables can yield unrealistic data that is outside the data manifold. The authors demonstrated that the use of such unrealistic data outside the data manifold lead to inappropriate estimate of Shapley value (Fig. 2, 3, 4, 5). To mitigate the effect of such unrealistic data, the authors proposed using generative models to sample realistic data inside the data manifold, thus avoiding use of unrealistic data for estimating Shapley value.

### Quality & Clarity
I think the paper is written clearly, and the main claims are easy to follow.
The experiments are designed well to support the claim.

### Originality & Significance
#### Contribution of the paper
Essentially, this paper tackles the known problem (avoiding unrealistic data in Shapley value (*1)) by using a common technique (generative model to sample realistic data (*2)). Although the idea is straightforward, the method itself seems to be reasonable and would be useful in some extent.

(*1) The problem considered in this paper is not a new problem. The problem of using unrealistic data for estimating Shapley value is pointed out by several previous studies [Ref1, 2, 3, 4], as some of them are mentioned in the paper. 

(*2) The major contribution of this study is therefore on the methods for mitigating the effect of unrealistic data. In this study, the authors proposed using generative models to sample realistic data inside the data manifold. The authors also proposed training a surrogate model $g_y$ that approximates the conditional expectation used for computing Shapley value. To my knowledge, such techniques were not considered in the previous studies.  However, the use of generative models for sampling from conditional distribution itself is a common technique, e.g. [Ref5,6] used GAN and VAE for Model-X Knockoff.

#### Explanation and data manifold
The relationship between the data manifold and explanation is studied by [Ref7,8], which is missing in the current paper. It would be great if there are some discussions how the current paper and [Ref7, 8] are similar/different.
Below, I list some of them.
* Similar Points
  The similar problem, the explanation outside the data manifold, are studied in [Ref7, 8].
  The use of autoencoder is considered to mitigate the effect of data outside the data manifold.
* Different Points
  [Ref7, 8] focus on the gradient-based methods, while the current paper focuses on Shapley value.
In addition to similarity/difference, I am also interested in whether the theories of [Ref7, 8] are applicable to Shapley values, or whether there is a fundamental differences between the gradient-based methods and Shapley value.

[References]
* [Ref1] Problems with Shapley-value-based explanations as feature importance measures, ICML20
* [Ref2] The many Shapley values for model explanation, ICML20
* [Ref3] Explaining black box decisions by Shapley cohort refinement, arXiv19
* [Ref4] Interpretable Machine Learning (Sec 5.10.2), 2019
* [Ref5] KnockoffGAN: Generating Knockoffs for Feature Selection using Generative Adversarial Networks, ICLR19
* [Ref6] Deep latent variable models for generating knockoffs, 2020
* [Ref7] Explanations can be manipulated and geometry is to blame, NeurIPS19
* [Ref8] Fairwashing Explanations with Off-Manifold Detergent, ICML20

### Pros & Cons
[Pros]
* The use of generative models for sampling realistic data from conditional distribution seems to be reasonable to mitigate the effect of unrealistic data used in the current method "splicing."

[Cons]
* The use of generative models for sampling from conditional distribution itself is a common technique.
* The relationship between the data manifold and explanation is studied by [Ref7,8], which is missing in the current paper. It would be good to provide some discussions how the current paper and [Ref7, 8] are similar/different.

---
I read the rebuttal and the updated version of the paper. I think the authors did a good job resolving my concerns on novelty. I updated my score.
review_title: The use of conditional generative model seems to be reasonable.

Meta-review: A good paper with significant contribution on XAI and the on- vs off- data manifold explainability.
Reviewers have appreciated authors’ feedback and update of the paper (R1, R2, R4). I would like to personally thank the authors for a smooth, extensive and focused interaction w/ updates.

Decision: Accept (Poster)