Shapley explainability on the data manifold
Authors: Christopher Frye
Damien de Mijolla
Tom Begley
Laurence Cowton
Megan Stanley
Ilya Feige
Venue: ICLR 2021 Poster (2021)

Abstract: Explainability in AI is crucial for model development, compliance with regulation, and providing operational nuance to predictions. The Shapley framework for explainability attributes a model’s predictions to its input features in a mathematically principled and model-agnostic way. However, general implementations of Shapley explainability make an untenable assumption: that the model’s features are uncorrelated. In this work, we demonstrate unambiguous drawbacks of this assumption and develop two solutions to Shapley explainability that respect the data manifold. One solution, based on generative modelling, provides flexible access to data imputations; the other directly learns the Shapley value-function, providing performance and stability at the cost of flexibility. While “off-manifold” Shapley values can (i) give rise to incorrect explanations, (ii) hide implicit model dependence on sensitive attributes, and (iii) lead to unintelligible explanations in higher-dimensional data, on-manifold explainability overcomes these problems.


Published as a conference paper at ICLR 2021
SHAPLEY EXPLAINABILITY ON THE DATA MANIFOLD
Christopher Frye,
Damien de Mijolla, Megan Stanley,
Tom Begley, Ily Feige
Laurence Cowton,
Faculty, 54 Welbeck Street, London, UK
ABSTRACT
Explainability in Al crucial for model development, compliance with regula- tion, and providing operational nuance predictions. The Shapley framework for explainability attributes a model's predictions to its input features in a mathemat- ically principled and model-agnostic way. However, general implementations of Shapley explainability make an untenable assumption: that the model's features are uncorrelated. In this work, we demonstrate unambiguous drawbacks of this assumption and develop two solutions to Shapley explainability that respect the data manifold. One solution, based on generative modelling, provides flexible ac- cess to data imputations; the other directly learns the Shapley value-function, pro- viding performance and stability at the cost of flexibility. While "off-manifold" Shapley values can (i) give rise to incorrect explanations, (ii) hide implicit model dependence on sensitive attributes, and (iii) lead to unintelligible explanations in higher-dimensionn data, manifold explainability overcomes these problems.
INTRODUCTION
Explainability in Al is central the practical impact of Al on society, thus making it critical to get right. While many dichotomies exist within the field - between local and global explanations (Ribeiro et al., 2016), between post hoc and intrinsic interpretability (Rudin) 2011)) and between model-agnostic and model-specific methods (Shrikumar et al. (017) in this work we focus lo- cal, post-hoc, model-agnostic explainability as it provides insight into individual model predictions, does not limit model expressiveness, and is comparable across model types.
In this context, explainability can be treated as a problem of attribution. Shapley values (Shapley 1953) provide the unique attribution method satisfying a set of intuitive axioms, e.g. they capture all interactions between features and sum to the model prediction. The Shapley approach to explain-
ability has matured over the last two decades (Lipovetsky & Conklin 2001| Kononenko al. 2010 Strumbelj & Kononenko 2014
Datta et al. 2016 Lundberg & Lee 2017).
Implementations of Shapley explainability suffer from problem common across model-agnostic methods: they involve marginalisation over features, achieved by splicing data points together and evaluating the model on highly unrealistic inputs (e.g. Fig. [ Such splicing would only be justified if all features were independent; otherwise, spliced data lies off the data manifold.
Outside the Shapley paradigm, emerging explainability methods have begun to address this problem. See e.g. Anders et al. (2020) for general treatment of the off- manifoll problem in gradient-based based explainability. See also Chang et al. (2019) and Agarwal et al. (019) for image-specific explanations that respect the data distribution.
Within Shapley explainability, initial work towards remedying the off-manifold problem has emerged; e.g. Aas et al.) (2019) and (Sundararajan & Najmi (2019) explore empirical and kemel- based estimation techniques. but these methods do not scale to complex data. A satisfactorily gen- eral and performant solution to computing Shapley values on the data manifold has yet to appear and is focus of this work. Our main contributions are twofold:
Sec. compares on- and off manifold explainability, focusing novel and unambiguous short- comings of off-manifold Shapley values. In particular, we show that off manifold explanations are often incorrect, and that they can hide implicit model dependence sensitive features.

---

Published as a conference paper at ICLR 2021
7
Digit
Coalition
Splice 1
Splice
Splice
Splice
Splice 5
Figure 1: An MNIST digit, coalition pixels in a Shapley calculation, and 5 off off-manifold splices.
Sec. develops two methods to compute on-manifold Shapley values on general data sets: (i) flexible generative-modelligg technique to learn the data's conditional distributions, and (ii) simple supervised-learning technique that targets the Shapley value-function directly. We demonstrate the effectiveness of these methods on higher-dimensional data with experiments.
2 BACKGROUND ON SHAPLEY EXPLAINABILITY
The Shapley value (Shapley 1953) is a method from cooperative game theory that distributes credit for the total value ((N) earned by a team N = [,,,2 ...,,.., among its players:
|| (n |S| 1)! [o(SU v(S)
(1)
Σ
SCN(()
where the value function v(S) indicates the value that coalition of players S would earn without their other teammates. The Shapley value (i(1) represents player i's marginal value-added upon joining the team, averaged over all orderings in which the team can be constructed.
supervised learning, let fy (a)) be a model's predicted probability that data point æ belongs to class
y
To apply Shapley attribution to model explainability, one interprets the features (.i,...,5 F as players in game and the output fz( as their earned value. To compute Shapley values, one must define a value function representing the model's output on a coalition Tg S (m,
As the model is undefined on partial input æs, the standard implementation (Lundberg & Lee 2017) samples out-of-coalitoon features, z'' where S = \ S, unconditionally from the data distribution:
(x) (S) Exp(a') (fr L1zgg)]
(off)
We refer this value function, and the corresponding Shapley values, as lying off the data manifold since splices Is L generically lie far from the data distribution. Alternatively, conditioning out- of-coalition features x''s on in-coalition features 45 would result in on-manifold value function:
(3) The conditional distribution (()'(x))) is not empirically accessible in practical scenarios with high- dimensional data or many-valued (e.g. continuous) features. A performant method to compute on- manifold Shapley values on general data until-now lacking and a focus of this work.
( (S) = Ep(r(())) [x')
Shapley values (3), (i) provide local explainability for the model's prediction on data point æ. To understand the model's global behaviour, one aggregates the (x)( (i)'s into global Shapley values:
( Ep(z,y) (0),()) (i)
(4)
where p(z, y) is the labelled-data distribution. Global Shapley values be seen special case of the global explanation framework introduced by Covert et al. (020). As consequence of the axioms (Shapley 1953) satisfied by the (z) global Shapley values satisfy sum rule:
Σ क (+) Ep(asi [f)( ( [fg('')]
KEN
One interprets the global Shapley value 4 (i) as the portion of model accuracy attributable the (th feature. Indeed, the first term in Eq. (5) the accuracy one achieves by sampling labels from f's predicted probability distribution over classes. The offset term, which relates to class balance, is not attributable to any individual feature.
'The results of this paper can be applied regression problems by reinterpreting fy(z) the model's predicted value rather than predicted probability.
2

---

Published as a conference paper at ICLR 2021
3 EVIDENCE IN FAVOUR OF ON-MANIFOLD EXPLAINABILITY
The key differences between on- and off- aff-manifold Shapley values is subject of ongoing discussion; see Sundararajan & Najmi (019) or [Chen et al. (2020) for recent overviews. Here we focus on theoretical arguments and experimental evidence yet appear in the literature, in favour of the on- manifold approach. We begin with mathematically precise differences between on- and off-manifold methods in Sec. 3.1 and present unambiguous drawbacks of off-manifold Shapley values in Sec. 3.2
3.1 ON- VERSUS OFF- MANIFOLD DIFFERENCES MADE PRECISE
Suppose the model's 's input features 21, .1...... In are the result of data- generating process seeded by unobserved latent variables 21, ................................ Zdt Then there exist functional relationships
2; = (z 2d; C3) for i == 1, 1,...,,,
(6)
where 6 represents noise in Tt. In the limit of small <'', there are d directions in which a data point (I1,.. ................................ .n) can be perturbed while remaining consistent with the data distribution: these correspond to perturbations in 21,..., 2d in Eq. (6). The data thus lives on a d-dimensional manifold in ambient n-dimensional features space, and therefore satisfies n d constraints on the x; 's:
V A ( kn)
for k= L, ..... d
On- on-manifold Shapley values evaluate the model on inputs that satisfy these constraints, while the off-manifold approach uses spliced data that generically break them. For more detailed and math- ematically precise discussion of the data manifold in this context, see Anders et al. 2022).
ALGEBRAIC MODEL DEPENDENCE CAN BE MISLEADING
Any model (a) can be written in many algebraic forms that all evaluate identically on the data manifold. To show this, one can add any of the d constraints from Eq. 7 any of the model's n input slots. This changes the model's 's algebraic form but does not affect the model's output on the
data, since each constraint equals zero on-manifold. The model fy(x;) thus belongs an n(n d) dimensional equivalence class of functions that behave indistinguishably on the data.
On- ta-mmanioold Shapley values provide the same explanation for any two models that evaluate identi- cally on the data distribution, because on-manifold explanations do not involve evaluation anywhere else. Off- ifnifadi Shapley values provide different explanations for two models in the same equiv- alence class, as spliced data in the off- manifold value function break the constraints of Eq. 7
HIDDEN MODEL DEPENDENCE ON SENSITIVE ATTRIBUTES
This is not academic concern: it follows that off ffffmmnifold explanations vulnerable to adver- sarial model perturbations that hide dependence on select input features (Dombrowski et al.) 2019] (llckk et al.) 2020). Dimanovv et al. (2020) demonstrated that the off-manifold Shapley value for sensitive feature like gender could be reduced near zero via this vulnerability.
To see how this can happen, suppose that input feature ii represents gender and formally solve one of the constraints in Eq. 7 for x1. The result, say 21 v (x2...., can then be used to transform any model fy (x) into another
(A)))...,,T.)) = f, (12..... Fn); x2,
Fn
that has no algebraic dependence on gender æ but behaves identically to fy (x) on the data manifold. The off-manifold Shapley value for gender in f would vanish, since the off-manifoid value function of Eq. (2) depends I1 only through f (i.e. not at all). This result is problematic, since the two models behave cquivalently on the data and thus possess the same gender bias.
In contrast, the on- on-manifold Shapley values for f and would be identical, as 22 dependence enters the on-manifold value function of Eq. (3) through the conditional expectation value. In sense, on-manifold Shapley values represent the model's 's dependence on the information content of each feature, rather than the model's 's algebraic dependence.
3

---

Published as a conference paper at ICLR 2021
Census income data
Drug cousumption data
manitald On manitold
6.44
Original model
Globa Shapley value
Off manifoid
0.23
Suppressed model
0.14
Tree SHAP
0.12
menifold
Model retraining
0.44
0.33
0.06
0.04
0.02
Native country
Race
Hours per week
Workclass
Capiial loss
Sex
accupation
Age
Capital gain
Relatfonship
Education
Marial status
Nicotine
Crack
Meth
Katamine
Neroin
Canmabis
Coke
Amphet
Ecstasy
Mushrooms
Figure 2: (a) Vulnerability of off-manifold explanations to hidden model dependence. (b) Explana- tions of fixed model compared to a model that is retrained on each Shapley coalition of features.
We can demonstrate this on UCI Census Income data (Dua & Graff 2017) We trained a neural network predict whether an individual's income exceeds $50k based demographic features in the data. Coral bars in Fig. 2 a) display global Shapley values for this "Original model". (On- manifold values were computed with the unsupervised method developed in Sec.
We then trained an alternative model by fine-tuning the neural network above on a loss function that penalises model dependence on sex; see App. B for full details of this experiment. This resulted in a "Suppressed model" that makes identical predictions as the original model on 98.5% of the data. Teal bars Fig. aa display global Shapley values for this model. Note that the off-manifold Shapley value for sex is zero despite the similar behaviour exhibited by the original and suppressed models on the data. In contrast, on-manifold Shapley values explain both models similarly.
ON-MANIOOLD SHAPLEY VALUES IN THE OPTIMAL-MODEL LIMIT
Here we present result that strengthens the connection between on- on-manifold Shapley values and the data distribution: in the limit of an optimal model of the data, on-manifold Shapley values converge an explanation of how the information in the data associates with the labelled outcomes.
To show why this holds, suppose the predicted probability (xr converges to the true underlying
distribution p(ylæ)). In this optimal-mode limit (which is approached in the limit of abundant data and high model capacity) the on- en-manifold value function of Eq. 3 becomes
(on) f2(z) (S) + da's p(a's (x)) ply ag 1''g))) = p(y xs)
(9)
which shows that value is attributed 2; based on x,'s 's predictivity of the label y-
We can demonstrate this on UCI Drug Consumption data (Dua & Grafi 2017). Using the 10 binary features listed in Fig. 2 b) Mushrooms, Ecstasy, etc. we trained random forest f to predict whether individuals had consumed an I 1th drug: LSD. As the data contains just 10 binary features, were able empirically sample the conditional distributions in the on-manifold value function, Eq. (3). See Fig. bbbb) for the resulting off- and on-manifold global Shapley values.
Next fit separate random forest 9s each coalition S of features, 210 models in total, in the spirit e.g. of Strumbelj et al. (2009). We used the accuracy A(gs) of each model, in the sense of Eq. (5), as the value function for an additional Shapley computation:
|S| (n S 1)! [A(gsui) - A(gs)]
(1) = Σ
(10)
n!
SCNV
where (i) is directly the average gain in accuracy that results from adding feature to the set of inputs. These values are labelled "Model retraining" in Fig. bb.. Note their agreement with the on-manifold explanation of the fixed random forest f. On-manifold Shapley values thus indicate which features in the data most predictive of the label.
This consistency check allows us to show in passing that Tree SHAP (Lundberg et al, 2018 2020) does not provide method for on-manifold explainability. Observe in Fig. 2 (b) that Tree SHAP
4

---

Published as a conference paper at ICLR 2021
Feature value
Local Shopley velue
0.04
off maniiald
manitid
8.52
-0.5
Feature
Feature
Festure
(h)
mansiol
censity
30
Intler coalitions
Outlier coalitions
Off manifols
Off manifuld maniiits
On manits
0-
0.50
0.05
3.30
-0.2
-01
0.0
0.3
0.2
022
0.1
0.0
0.1 0.2
Nolse
Model
Model
(4)
(9)
Figure 3: An individual outlier (a), its off- and on-manifold explanations (b & c), the error rate in explanations (d), and the distribution of model outputs on Shapley coalitions (e & f).
roughly tracks the off manifold explanation, albeit larger on the most predictive feature and some- what smaller on the others. This occurs because trees tend to split on high-predictivity features first, and Tree SHAP privileges early-splitting features in otherwise off-manifoll calculation.
3.2 UNAMBIGUOUS SHORTCOMINGS OF
EXPLAINABILITY
Whereas above we clarified precise differences between on- and off-manifold Shapley values, this section focus unambiguous drawbacks of the off- mfaifold approach.
UNCONTROLLED MODEL BEHAVIOUR OAF-MANIFOLD
Sec. 3.1 might lead one to believe that off-manifold Shapley values provide insight into the algebraic dependence of model. However, the off-manifold approach of evaluating the model on spliced in- distribution data does not constitute controlled study of such dependence. Off- manifold Shapley values serve as a perilously uncontrolled technique, especially in complex nonlinear models such as neural networks. Indeed, it is widely known that deep-learning models are not robust to distributional shift (Nguyen et al. 2015 Goodfellow al. 2015). Still, off-manifold Shapley values evaluate the model outside its domain of validity, where is untrained and potentially wildly misbehaved. This garbage-ing iele--n-----ee inw-annggge-oouu problem is the clearest reason to avoid the off manifold approach.
Since this point has been documented in the literature (Hooker & Mentch, 2019). here simply provide an example: Fig. shows a binary MNIST digit (LeCun & Cortes 2010), coalition of pixels, and 5 random splices that would be used to compute off-manifold explanation.
OUTLIER DETECTION EXPLAINED INCORRECTLY MANIFOLDD
To demonstrate that off-manifold Shapley values frequently lead to incorrect explanations, here we offer an example on synthetic data where the ground -ruth explanation is known. We generated 10 synthetic data points, each consisting of 20 real-valued features, for the purpose of outlier detection. We split the dataset between 99% inliers and 1% outliers, with the classes generated according to:
20
Pmn (21...., F220) =
Σ
MT2. o (2xa))
(11)
2
==0,1
(1=1
II NT o( ) 112, (x):)
Poul [ (......, F20)
Σ 2=0,1
(12)
That is, there is a single binary latent variable 2. For inliers, each feature an independent noisy reading of the latent For outliers, the first 5 features An example outlier (with a = 0.05) is shown in Fig. Ba). We generated one such data set for each (0.01, 0.03. ... 0.15) in order study the effect of noise explanation errors.
centred instead around its opposite ž.
5

---

Published as a conference paper at ICLR 2021
Syuthetic data generation
Syuthetic data
A Abalone data
Global Shopphey valle
0.20
dll muntoll On mantoie
Off manifold
Fabe 50%
True 60%
On manifold (supervise))
0.15
On manttold (unsupervieed)
0.10
0.10
Fabes
N: m
Thue 10%
Fades
True 80%
0.25
0.2
0.1
0.00
0.05
y-1 koxi)
Kg
X1
Height
Sex
Length
Viscera weight
Diameter
Whote weight
Shucked weight
Stell weight
(a)
(b)
Figure 4: Negative global Shapley values arise off-manifold, in both (a) synthetic and (b) real data.
We fit an isolation forest (Liu et al., 2008 to perform outlier detection on each synthetic dataset, achieving 100% accuracy in every case. We computed the off- and on-manifold value functions of
Eqs. ( and (3) for each isolation forest by sampling the probability distributions directly, these
can be inferred from Eqs. (IT) and [2]. Figs. b) and 3 (c) show the resulting local Shapley values
for the example outlier from Fig. 3(a.
The ground-truth explanation of why Fig. a) represents an outlier is that its first 5 features break correlations that exist across 99% of the data. The on-manifold explanation of Fig. (]) correctly at- tributes the 5 largest Shapley values to features .I...,,,66 The off- explanation of Fig. 3b) unambiguously incorrect: feature I7 receives larger value than T2, I4, and æz. We consider an explanation to be erroncous if : 25 do not receive the 5 largest Shapley values.
To show the frequency of incorrect explanations, Fig. 3(d) displays the off- and on- on-manifold error rates as function of noise in the synthetic data set. Incorrect explanations off-manifold: one-quarter are in error the presence of minimal noise, and two-thirds are incorrect 0.15. The on-manifold error rate is dramatically lower across this range.
commonplace
Figs.3jes and 3.1) show the root cause of off manifold errors. These histograms display the distribu- tion of model outputs when evaluated on Shapley coalitions in the off- and on-manifold calculations for a = 0.05. In particular, Fig. Fe) shows the model evaluated on "inlier coalitions" which do not include 21,--..; Note that model outputs for on-manifold coalitions agree with the model evaluated on the actual data, while off- manifold coalitions follow a very different distribution. In particular, since a positive model output indicates a predicted outlier, Fig. Bee) shows that the off- manifold calculation itself fabricates outliers through its splicing procedure.
Similarly, Fig. fi) shows the model evaluated on "outlier coalitions" which do include 21....; 23- Note that model outputs are similar for on on-manifold coalitions and actual outliers, whereas off- manifold coalitions again differ dramatically. This is a manifestation of uncontrolled model be- haviour off the data manifold, and it ultimately leads to erroneous off-manifold explanations.
BREAKDOWN IN GLOBAL SHAPLEY VALUES MANIIFLD
To demonstrate that global Shapley values can be misleading off-manifold as well, we generated an additional synthetic data set according to the process in Fig. a). The data has two binary features and binary label. We fit decision tree to this data, resulting in precise match to Fig. (a). Note that the features Ii and # are positively correlated, both with each other and with label y. However, with zo fixed, the likelihood of y decreases slightly from 11 = 0 to 1. One might think of To as disease severity, Ø1 as treatment intensity, and y as mortality rate.
Fig. 4bb displays global Shapley values for this model. The global Shapley values positive on- manifold, consistent with their interpretation as the portion of model accuracy attributable to each feature. Off-manifold, however, negative value results from placing too much weight splices, e.g. with (Io, #1, y) = (0, 1, 1), that occur less frequently in the actual data. The negative value would erroneously indicate that 2 is detrimental to the model's 's overall performance.
We can demonstrate this on real data using UCI Abalone data (uua & Graff 2017). We trained a neural network to classify abalone as younger than older than the median age based on physical

---

Published as a conference paper at ICLR 2021
characteristics. Fig. c) displays global Shapley values for this model. (On-manifold values were computed using techniques developed in Sec. 4.1 see App. for details.)
Observe the drastic difference between the and off-manifold explanations in Fig. [c.. This due to the tight correlations between features in the data (4 weights and lengths) making the data manifold low-dimensional and important. Notice further the large negative off- Iof-manifold global Shapley value, negating its interpretation as the portion of model accuracy attributable to that feature.
4 SCALABLE APPROACHES TO ON-MANIFOLA SHAPLEY VALUES
In Sec. 3 we computed on-manifold Shapley values for simple data by estimating p(x' (rg) from the empirical data distribution or, for synthetic data, by knowing this distribution analytically. Here we introduce two performant methods to compute on-manifold Shapley values on general data. Sec. 4.1
develops the theory underlying our methods, and Sec. 4.2 presents additional experimental results.
4.1 THEORETICAL DEVELOPMENT OF ON- -N-IFOOOODD METHODS
Here we develop two methods to learn the on-manifold value function: (i) unsupervised learning the conditional distribution p(z'|=s) and (ii) a supervised technique to learn the value function directly.
UNSUPERVISED APPROACH
One can unsupervised learning learn the conditional distributions p(a' [x:)) that appear in the on-manifold value function. Here we take an approach similar to Ivanov et al. (2019) to learn these distributions with variational inference. See Douglas alternative techniques to learning conditional distributions that could be used here instead.
(2017) and Belghazi al. (2019) for
Our specific approach includes two model components. The first is variational autoencoder (Kingma & Welling 2014; Rezende et al. 2014), with encoder (o(2(z) and decoder pe (z|2). The second is a masked encoder, ryp (2( for which the goal is to map the coalition æy a distribution in latent space that agrees with the encoder qo (z(z) as well as possible. A model of p(æ" (rs) is then provided by the composition:
pa = dz pe (a(al12) (2las)
(13)
and good fit to the data should maximise (11(x))) A lower bound to its log-likelihoode given by
Co (xg()) log pe |2]] DEL (go(2<=)) (+<(()))))
(14)
While La could be used on its own as the objective function to learn p(x' (ıs), this would leave the variational distribution (2(xx= unconstrained, at odds with our goal of learning smooth-manifold structure in latent space. This concern can be mitigated by Creg -KLL. (gø(2|z) p(=)) which regularises (2(2)x) by penalising differences from smooth (e.g. unit normal) prior distribution p(z). We thus include Ligg as a regularisation term in our unsupervised objective: L = Lo + B Lieg-
METRIC FOR THE LEARNT VALUE FUNCTION
The unsupervised method presented above leads to a learnt estimate of the conditional distribution, and thus to an estimate of the on- on-manifold value function: ( (S) = [x(="(s)) I (=)).. With the goal of judging the performance of this estimate, consider the following formal quantity:
mse(#s, v) (p(()(ss) [5g(2²) ( (S))
(15)
This quantity is minimal with respect to (z) when , (e)()) [ in agree- ment with the definition, Eq. 33) of the on on-manifold value function. We can then quantitatively judge the performance of the unsupervised model ò(z'| by computing
MSE = Ex(x) ES--Shaply EniUUit (x)(x) (S))
(16)
7

---

Published as a conference paper at ICLR 2021
Drug consumption data
Global Shapley valuue
0.30
On manifols values
0.08
Supertsed
Empiicall
Unsupervised
0.04
0.20
Nicetine
0.00
Crack
Meth
Katamine
Heroin
Cannabis
Cake
Anprier
Ecsaayy
Mushrooms
Figure 5:
Validation of unsupervised and supervised techniques for computing on manifold Shapley values. Comparison against empirical ground truth, which appeared as "On manifold" in Fig. 2
Table I: Performance and stability, in terms of MSE, of supervised and unsupervised approaches. Performance is compared with off-manifold splicing and, where accessible, the empirical optimum.
DATA SET
OFF MANIFOLD
UNSUPERVISED
SUPERVISED
EMPIRICAL
DRUG
0.0634
0.0536 + 0.0007
0.0441 + 0.0002
0.0436
ABALONE
0.0647
0.0293 + 0.0009
0.0200 + 0.0001
CENSUS
0.0344
0.0300 + O. 0.0006
0.0250 + 0.0001
MNIST
0.0448
0.0257 0.0005
0.0121 0.0001
Note that this is precisely Eq. 15 averaged over coalitions S drawn from the Shapley sum features Is p(xs) drawn from the data, and labels y drawn uniformly over classes. Morcover, the mean- square-crror in Eq. (16) is easy to estimate using the empirical distribution p(z) and the learnt model ((s),, thus providing unambiguous metric judge the outcome of the unsupervised approach.
SUPERVISED APPROACH
The MSE metric of Eq. (6 supports a supervised approach to learning the on-manifold value function directly: one can define a surrogate model (xr) that operates on coalitions of features as (e.g. by masking out-of-coalition features) and that is trained to minimise the loss:
L= ==Ep(e) Egg-Shapley Ey--Unit [x(x)) (*s))
(17)
As discussed above Eq. (16). this loss is minimised as the surrogate model (z(2) approaches the on-manifold value function (fya")) of the model-to-be--eplaine..
4.2 ADDITIONAL EXPERIMENTS
Sec. above presented two experiments using the scalable on-manifold methods developed here. In particular, Fig. 2(a) applied the unsupervised method to Census Income data, showing that on- manifold Shapley values detect hidden model dependence sensitive features, and Fig. 4T)) applied both methods to Abalone data, showing that global Shapley values remain positive and interpretable on-manifold. In this section, perform additional experiments to study the performance and sta- bility of Sec. I methods, as well as their effectiveness on higher-dimensional data.
PERFORMANCE AND STABILITY
Our implementations of the unsupervised and supervised approaches to on-manifold Shapley values summarised in Apps. A and [ Both approaches lead to broadly similar results. Fig, coompares the two techniques on the Drug Consumption data, where explanations are compared against the ground-truth empirical computation from Fig. b).
The unsupervised approach is flexible but untargeted: p(ø' (as) ) is data-specific but -agnosic,,, accommodating explanations for many models trained the same data. The supervised approach
In more detail, here sample coalitions from Eq. where the probability assigned to each coalition is the combinatorial factor [S]!(n |S] 1)1/n!.
8

---

Published as a conference paper at ICLR 2021
Unymamtool Digi
2
3
4
of moneld Oemmnttii
0.1
0.08
0.04
0.00
URS-
(n)
(b)
Figure 6: (a) Randomly drawn MNIST digits explained on/oft oof manifold. Red/ blue pixels indi-
cate positive/ negative Shapley values, and the colour scale in each column is fixed. (b) Shapley summand as function of coalition size averaged over coalitions, pixels, and the MNIST test set.
must be retrained each model, but it entails direct minimisation of the MSE. The supervised method is thus expected to achieve higher accuracy. We confirmed this on all data sets studied in this paper, see Table I for numerical comparison of the MSEs.
In [able central values indicate the test-set MSE achieved by each method. The table compares the unsupervised and supervised methods against off-manifold splicing, showing significant improve- ment over this baseline. Note that an MSE of zero is not achievable, because (z) in Eq. (T6) or (17) not fully determined by partial imput Ty. For the Drug Consumption data where we can compute p(a1 empirically, the optimal MSE happens to be 0.0436.
Uncertainties Table represent the standard deviation test-set MSE upon repeating each method with fixed hyperparameters 10 times. (Uncertainties are absent for the off- manifold and empirical columns, as these do not involve training a separate model.) The table thus indicates that the super- vised method offers increased stability as compared to the unsupervised approach.
The supervised method more efficient as well: while the unsupervised technique estimates the value function by sampling from j(x'|xs) the supervised approach learns the value function di- rectly. The supervised method thus requires far fewer model evaluations to match the standard-error of the unsupervised method: roughly 10 times fewer in our experiments.
EXAMPLE ON MNIST
To demonstrate on-manifold explainability on higher-dimenssinal data, we trained a fully connected network on binary MNIST (LeCun Cortes 2010) and explained random digits in Fig. aa).
Despite having the same sum over pixels as controlled by the local version of Eq. 55 and explaining the same model prediction. each on-manifold explanation is more concentrated, with more interpretable structure, than its off-manifold counterpart. The handwritten strokes are clearly visible on-manifold, with key off-stroke regions highlighted as well. Off-manifold explanations generally display lower intensities spread less informatively across the digit-region.
These off off-manifold explanations are a result of splices as in Fig. I With such unrealistic input, the model's 's output is uncontrolled and less informative. In fact, is only on very large coalitions of pix- els, subject to minimal splicing. that the model can make intelligent predictions off-manifold. This is confirmed in Fig. [(b),, which shows the average Shapley summand a function of coalition size on MNIST. Note that primarily large coalitions underpin off-manifold explanations, whereas far fewer pixels are required on-manifold, consistent with the oiw-dimensional manifold underlying the data,
5 CONCLUSION
In this work, we took a careful study of the off aff-manifold problem in Al explainability. We presented important distinctions between on- and off-manifold explainability and provided experimental evi- dence for several novel shortcomings of the off-manifold approach. We then introduced two tech- niques to compute on- on-manifold Shapley values on general data: one technique learns to impute fea- tures the data manifold, while the other leams the Shapley malue-function directly. In-so-doing, we provided compelling evidence against the use of off manifold explainability, and demonstrated that on-manifold Shapley values offer viable approach Al explainability in real-world contexts.

---

Published as a conference paper at ICLR 2021
REFERENCES
J. Aas, M. Jullum, and A. Loland. Explaining individual predictions when features are dependent: More accurate approximations to Shapley values, 2019. [arXiv: 903. 14444..
C. Agarwal, D. Schonfeld, and A. Nguyen. Removing input features via a generative model to explain their attributions classifier's decisions, 2019. [arXiv: 910. 04256].
C. J. Anders, P. Pasliev, A.-K. Dombrowski, K.-R. Müller, and P. Kessel. Fairwashing explanations with off manifold detergent. In International Conference Machine Learning, 2020.
M. Belghazi, M. Oquab, and D. Lopez-Paz. Learning about an exponential amount of conditional distributions. In Advances in Neural Information Processing Systems, 2019.
C. Chang, E. Creager, A. Goldenberg, and D. Duvenaud. Explaining image classifiers by counter- factual generation. In International Conference on Learning Representations, 2019.
H. Chen, J. D. Janizek, S. Lundberg, and S.-I. Lee. True to the model or true to the data?, 2020. [arXiv:2006 1e..220066666222.. 1.234].
I. Covert, S. Lundberg, and S.-I. Lee. Understanding global feature contributions through additive importance measures, 2020. [arXiv:2004. 06668].
A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems, In IEEE Symposium on Security and Privacy, 2016.
B. Dimanov, U. Bhatt, M. Jamnik, and A. Weller. You shouldn't trust me: Learning models which conceal unfairness from multiple explanation methods. In SafeAl @ AAAI, 2020.
A.-K. Dombrowski, M. Alber, C. Anders, M. Ackermann, K.-R. Müller, and Kessel. Explanations can be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, 2019.
L. Douglas, I. Zarov, K. Gourgoulias, C. Lucas, C. Hart, A. Baker, M. Sahani, Y. Perov, and S. Johri. A universal marginalizer for amortized inference in generative models. In NIPS Workshop on Approximate Bayesian Inference, 2017.
D. Dua and C. Graff. UCI machine learning repository, 2017. [archive. i ics uuii edd/ml]]
I. Goodfellow, Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In International Conference Learning Representations, 2015.
G. Hooker and L. Mentch. Please stop permuting features: explanation and alternatives, 2019. [arXiv: 1905. 03151].
O. Ivanov, M. Figurnov, and D. Vetrov. Variational autoencoder with arbitrary conditioning. In International Conference on Learning Representations, 2019.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
I. Kononenko et al. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 2010.
Y. LeCun and C. Cortes. MNIST database, 2010. [yann. lecun. com/ eeweeexdbmmmmm xxdd/mmnsst)...
S. Lipovetsky and M. Conklin. Analysis of regression in game theory approach. Applied Stochastic Models in Business and Industry, 2001.
T. Liu, K. M. Ting, and Z.HH. Zhou. Isolation forest. IEEE International Conference on Data Mining, 2008.
10

---

Published as a conference paper at ICLR 2021
S. M. Lundberg and S.-1. Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, 2017.
S. M. Lundberg, G. G. Erion, and S.-I. Lee. Consistent individualized feature attribution for tree ensembles, 2018. farxiv:1800 03888]
M. Lundberg, G. Erion, H. Chen, DeGrave, J. Prutkin, Nair, Katz, Himmelfarb, N. Bansal, and S. Lee. From local explanations to global understanding with explainable Al for trees. Nature Machine Intelligence, 2020.
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence pre- dictions for unrecognizable images. In Conference Computer Vision and Pattern Recognition, 2015.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate infer- ence deep generative models. In International Conference on Machine Learning. 2014.
M. T. Ribeíro, Singh, and C. Guestrin. Why should trust you: Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining, 2016.
C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
L. S. Shapley. A value for n-person games. In Contribution to the theory of games, 1953.
A. Shrikumar, P. Greenside. and A. Kundaje. Learning important features through propagating activation differences. International Conference on Machine Learning. 2017.
D. Slack, Hilgard, E. Jia, S. Singh, and H. Lakkaraju. Fooling lime and shap: Adversarial attacks post hoc explanation methods. In AIES @ AAAI, 2020.
E. Strumbelj and I. Kononenko. Explaining prediction models and individual predictions with fea- ture contributions. Knowledge and information systems, 2014.
E. Strumbelj, I. Kononenko, and M. Robnik-Sikonja. Explaining instance classifications with inter- actions of subsets of feature values. Data Knowl. Eng.. 2009.
M. Sundararajan and A. Najmi. The many Shapley values for model explanation. In International Conference on Machine Learning, 2019.

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
For the unsupervised approach, we modelled the encoder go (z(22) as a diagonal normal distribution with mean and variance determined by a neural network:
(=(=)=)) (M(s(())
(18)
We modelled the decoder (=(2) as a product distribution:
po(x12) II (o(2)
(19)
where the distribution type (e.g. normal, categorical) of each H; is chosen per-data-set and each distribution's parameters are determined by a shared neural network. We modelled the masked encoder r (2(2s) as Gaussian mixture:
(=(=))))) = En En (x) (z)
(20)
To allow re (2 lns) to accept variable-size coalitions zg as input, we simply masked out-of-coalition features with special value (-1) that never appears in the data.
The unsupervised method has several hyperparameters: B which multiplies the regularisation term, the number of components in Eq. (20). as well the architecture and optimisation of the networks involved. For each experiment this paper, we tuned hyperparameters minimise the MSE of Eq. 416 held-out validation set; see App. for numerical details.
For the supervised approach, we modelled gy (xs) using a neural network, again masking out-of- coalition features (with -)) to accommodate variable-size coalitions Iş. This method's hyperpa- rameters, relating to architecture and optimisation, were similarly tuned to minimise the validation- set MSE; see App. [ for details.
B DETAILS OF EXPERIMENTS
Here we provide numerical details for the experiments presented in the paper.
B. DRUG CONSUMPTION EXPERIMENT
On the Drug Consumption data from the UCI repository (Dua & Graff 2017). used 10 binary features from the data set Mushrooms, Ecstasy, etc., as displayed in Fig. 5 to predict whether individuals had ever consumed an 11th drug: LSD. The explanations of Fig 2 b) and Fig. desscribe a random forest fit with default sklearn parameters and max mnx_feetures None, which achieves 82.2% test-set accuracy amidst 57 43 class balance.
In Fig. b)., global off-manifold Shapley values were computed using 10 Monte Carlo samples of Eq. For each labelled data point (x, y) sampled from the test set, single permutation was drawn to estimate Eq. (1) and single data point a' was drawn to estimate the off- off-manifold value function Eq. ( In all the figures of this paper, bar height represents the mean that resulted from Monte Carlo sampling, and error bars display the standard error of the mean.
Global on-manifold Shapley values in Fig. b) were computed similarly, but this case using the on-manifold value function of Eq. (3). For each sampled coalition æs, random data point x' was drawn from the test set, with the crucial requirement that I's = Ts. In the text, we refer to this empirically estimating the conditional distribution p(z'(xs). Such empirical estimation is only possible because this data set has a small number of all-binary features.
Tree SHAP values in Fig. 2 (b) were computed with the SHAP package (Lundberg & Lee 2017 with oouel = margin and feature feature-perture perturbation = tree meth.dependent.
The values labelled "Model retraining" in Fig. b were computed by fitting separate random forest 9s for each coalition S of features in the data set: 20 models in all. We used these models to compute the sum of Eq. (10), where A(gs) represents variant of model gs accuracy: it is the
12

---

Published as a conference paper at ICLR 2021
Table 2: Optimal hyperparameters found for computing on-manifold Shapley values.
DATA SET
METHOD
HIDDEN DIM.
LEARN, RATE
LATENT DIM.
MODES
B
DRUG
SUPERVISED
512
10-3
UNSUPERVISED
128
10-3
4
0.5
ABALONE
SUPERVISED UNSUPERVISED
512
10
256
10
2
0.05
CENSUS
SUPERVISED UNSUPERVISED
512
10-3
128
10
8
1
MNIST
SUPERVISED UNSUPERVISED
512
10
512
10-4
16
1
accuracy achieved if predicts labels by drawing stochastically from gs's predicted probability distribution (as opposed to deterministically drawing the maximum-probabile class).
The global on on-manifold Shapley values in Fig. b appear in Fig. 5 as well, labelled "Empirical", Fig. [5 also displays on-manifold Shapley values computed using the supervised and unsupervised
methods introduced in this paper. above, these are Monte Carlo estimates of Eq. (4). The supervised method involved training fully connected network on the MSE loss of Eq. (77) All
neural networks in this paper used 2 flat hidden layers, Adam (Kingma & Ba 2015) for optimisation, and batch size of 256. We scanned over grid with
hidden layer size {128, 256, 512}
(21)
learning rate [10 10
choosing the point with minimal MSE on held-out validation set after 10k epochs of training; see Table 2 Each supervised value Fig. corresponds 10 Monte Carlo samples.
The unsupervised method involved training variational autoencoder as described in Sec. 4.1 and App. A The encoder, decoder, and masked encoder were each modelled using fully connected networks, trained using early stopping with patience 100. We scanned over grid of hidden layer sizes and learning rates in Eq. (T as well
latent dimension (2, 4, 8, 16)
(22)
latent modes = [1,2]
regularisation (0.05, 0.1, 0.5, 1)
choosing the point with minimal validation-set MSE; Table Unsupervised values in Fig.
correspond to 10 Monte Carlo samples.
B.2 CENSUS INCOME EXPERIMENT
To produce the explanations of Fig. a) we used the Census Income data from the UCI repository (uua & Graff 2017). The data contains 49k individuals from the 1994 US Census, as well as 13 features which we used to predict whether annual income exceeded $50k. We trained fully connected network (hidden layer size 50, default skleam parameters, and early stopping). achieving a test-set accuracy of 85% amidst a 76 24 class balance.
The Shapley values for this model are labelled "Original model" in Fig. aa.. These puted exactly as described in App. B.I except that the supervised method used 5k epochs, and the unsupervised method used patience 50. Optimised hyperparameters are given in Table 2. The on- manifold values in Fig. a were computed using the unsupervised method. While the supervised method does not appear the figure, it performed to complete Table 1.
com-
We also fine-tuned the "Original model" to suppress the importance of sex. Motivated by Dimanov al. 2020) added term to the loss that penalises the finite difference in the model output with
13

---

Published as a conference paper at ICLR 2021
respect to sex (as this is discrete feature).
The modified loss function thus becomes
1 N Σ ((f(s)).))) + [a do(sex 1)) f2v do(sex 0))
(23)
where is the cross- eros-pyntrppy loss, f f2; do(sex j)) denotes evaluated on the data point z; with the value for replaced with j. and hyperparameter controlling the trade-off between optimising the accuracy and minimising the effect of sex. We fine-tuned the model for an additional 200 epochs with a 3. The resulting model agrees with the baseline on over 98.5% of the data, and has the same test-set accuracy. Shapley values for this model are labelled "Suppressed model" Fig. a).
B.3 ABALONE EXPERIMENT
The Abalone data set from the UCI repository (Dua & Graff 2017) to physical measurements (see Fig. 4 which we used to classify abalone as younger than or older than the median age. We trained neural network perform this task with hidden layer size 100, default sklearn parameters, and early stopping obtaining a test-set accuracy of 78%.
contains 8 features corresponding
Shapley values in Fig. Cc) were computed exactly described in App. B.1 except that the super- vised method involved training for 5k epochs. Optimised hyperparameters are given in Table 2
B.4 MNIST EXPERIMENT
For binary MNIST (LeCun & Cortes 2010), we trained fully connected network (hidden layer size 512, default sklearn parameters, and early stopping) achieving 98% test-set accuracy.
The digits in Fig. (()) were randomly drawn from the test set. Shapley values in Fig. a) were computed exactly as described in App. B.I] except that the supervised method involved training for 2k epochs, and the on-manifold explanations are based on 16k Monte Carlo samples per pixel. Optimised hyperparameters
given in Table 2 The on-manifold explanations in Fig, ([a) were computed using the supervised method. While the unsupervised method does not appear in the figure, it was performed to complete T
The average uncertainty, which is not shown Fig. ((a), is roughly 0.002 - stated as a fraction of the maximum Shapley value in each image.
14


Reviews:
[{"review_id": "ofSzxRbY6vo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is focused on the off-data manifold problem with Shapley values which is created by sampling data that is out of distribution. The goal is to develop efficient methods. Two main algorithms are proposed: Generative models to approximate conditional distributions and training supervised models for direct approximation. They show an advantage against original off-manifold Shapley values in experiments. The impression experiment is particularly interesting.\nThe overall idea of using generative models to solve the issue of off-manifold data in interpretability methods (not just SHAP) is generally a nice direction. Note that the problem is more prominent for the case of SHAP as the method is built on performance on all subsets of the input features and the paper makes a good case of showing the necessity of solving the off-manifold data problem for Shapley-based methods. The experimental results are also comprehensive and provide enough evidence for their usefulness.\n There seem to be novelty concerns \"A performant method to estimate onmanifold Shapley values on general data is until-now lacking and a focus of this work.\" The problem of off-manifold data in interpretability is well studied and its specifics for the SHAP method have been discussed before https://proceedings.icml.cc/static/paper_files/icml/2020/334-Paper.pdf The authors make a very brief reference to this paper but do not actually mention how they are different. Unless the main differential contribution of the work to the existing literature is clear, I cannot change my score. If the contribution is \"\"experimental evidence ... in favour of the onmanifold approach\", the contribution is not enough for this venue.\n\nQuestions and notes:\n* The higher dimensional example is not actually high dimension and it does not give a sense of how the method would be applied to cases like ImageNet. Authors need to provide runtime results to make the case for the practical efficiency of their method. \n* There does not seem to be a discussion of the effect of the generative model's performance on the results. Having a good generative model that actually captures conditional probabilities for high dimensional data like images is still an open question and therefore, it will hinder the proposed method's performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HBf5DGxrs1t", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the drawbacks of existing explanation methods that implicitly use off-manifold samples while holding out groups of features. Most existing SHAP variants (e.g., KernelSHAP) use off-manifold samples, in part because using on-manifold samples from the conditional distribution is computationally challenging. This work proposes two methods to do so tractably (one \"unsupervised,\" one \"supervised\") and demonstrates some advantages of the on-manifold approach.\n\nI appreciated many results from this paper. It was valuable to explain that when using the on-manifold approach, two models that make identical predictions get identical SHAP values. It was also good to see that the on-manifold approach successfully detects dependencies on sensitive attributes where the off-manifold approach fails (Figure 2). When comparing the various approaches, showing that the on-manifold approach is roughly equivalent to retraining $2^n$ models is helpful, and showing that TreeSHAP does not perfectly model the conditional distribution corrects a potential misconception that this approach is equivalent to the on-manifold approach. \n\nA couple of questions and concerns about the methods and theory in this work. \n\n1. The perspective of the data distribution being controlled by unobserved latent variables and noise variables was not helpful, in my view. It was not clear what the authors meant by $O(\\epsilon)$, it was not clear why we should assume that the relationship between $x$ and $z$ is invertible (see $\\Psi_k$) and it was not clear when/why we should assume that each $x_i$ is a deterministic function of the other features (see $\\tilde \\Psi$). The math in this section (beginning of 3.1) seemed like another way of saying that data samples should come from dense parts of the data distribution, but without adding much additional insight. (I'm open to changing my opinion on this if others disagree.)\n2. The metric for the on-manifold value function was not explained clearly. In what sense is this value function optimal? Is it necessary to use MSE to train a supervised classification model? I believe a more complete proof of this result is provided in Appendix A of Covert et al. (cited), and this result suggests how to train a supervised surrogate using cross entropy loss.\n3. The comparison of the supervised and unsupervised approach is very important because it informs how future work should use the on-manifold method. I understand the space constraints, but if possible some of these results could be shown in the main text. Table 1 in the supplement seems to show that the supervised approach is vastly superior according to the proposed metric; the authors should also include a comparison with the off-manifold approach (using the marginal distribution) to give a sense of how significant the difference is. It would also be interesting to see the standard error comparison mentioned on p.8, which suggests that the supervised approach converges much faster.\n\nA couple presentation concerns:\n\n- Removing features using their conditional distribution is more specific than using on-manifold samples, yet this is what the authors seem to mean by \"on-manifold.\" (It is possible to use samples from a different distribution that shares the same support as the conditional distribution.) The authors may wish to clarify this ambiguity and possibly explain more specifically why the conditional distribution is appropriate. \n- The presentation is often specific to classification models, but this seems unnecessary. Readers can probably translate the ideas to regression models on their own, but the authors may consider broadening their presentation.\n- The \"non-parametric limit\" is an odd way of presenting the result in Eq. 9 (in my view, although others may disagree). In what sense is the model converging? To be more clear, the authors may consider explaining that this result requires the optimal model (i.e., the Bayes classifier) and that this should only be expected given unlimited data and a sufficiently flexible model. \n- When discussing the \"unsupervised approach,\" it may be worth saying explicitly that this does not directly provide an estimate of the value function, but it provides the means to estimate it via a Monte Carlo approximation. The authors may also wish to point out that this means the unsupervised approach requires far more model evaluations than the supervised approach to evaluate any given coalition.\n- What exactly is the Shapley distribution used in Eq. 16? I couldn't find where this was explained.\n \nThe paper has a couple missing citations, although they don't particularly diminish the work's novelty. Slack et al. [3] show that off-manifold methods can be fooled into missing dependencies on sensitive attributes; this could fit into the narrative in a couple parts of the paper. The \"unsupervised approach\" is just learning to sample from arbitrary conditional distributions, and there's more work on this topic than the Ivanov et al. paper. Douglas et al. [2] have a method called the \"universal marginalizer\" and Belghazi et al. [1] have a method called the \"neural conditioner\" (NC) that has been shown to outperform the AC-VAE. The authors may wish to comment why their approach is modeled off of the AC-VAE rather than the more recent NC.\n\nOverall, I think that this work proposes a solution to an important problem (holding out features using their conditional distribution) and I believe it could be impactful. However, I also think the paper's presentation could be improved in the ways described above. \n\n[1] Belghazi et al., \"Learning about an exponential amount of conditional distributions\" (2019)\n\n[2] Douglas et al., \"A Universal Marginalizer for Amortized Inference in Generative Models\" (2017)\n\n[3] Slack et al., \"Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods\" (2019)\n\n##########\nUpdate\n##########\n\nThe authors have addressed many of my concerns. Moving Table 1 into the main text and adding the new columns is an important change, and it seems that the authors have improved various aspects of their presentation. I'm raising my score to a 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0qGylSCeg9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper gives thorough intuitions, theoretical arguments and synthetic examples that demonstrate the shortcomings of off-manifold Shapley values. It then proposes two approaches to overcome this limitation. Specifically, they enable scalable sampling from the on-manifold conditionals. The first approach is unsupervised based on VAEs and model-agnostic. The second approach builds upon the supervised model. The latter is shown to be more accurate and data efficient. Both approaches are shown to fix the limitations of off-manifold Shapley values on a real-world MNIST example. The authors provide implementation detail that will be useful for practitioners.\nThanks for this a well-written, well-motivated easy-to-follow paper.\n\nMinor comments.\n- add to the intro a defintion of \"data splicing\"\n- formalize problem of off-manifold shapley values\n- It would be great if the authors released code for their experiments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0RB6xDuKsj0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper Summary\nIn this paper, the authors pointed out that the explanation with Shapley value can be misleading due to inappropriate assumptions in the current estimation method. The authors pointed out that the \"splicing\" techniques used in the current estimation method assumes the variables to be independent, which is not always the case in practice.  Application of \"splicing\" to dependent variables can yield unrealistic data that is outside the data manifold. The authors demonstrated that the use of such unrealistic data outside the data manifold lead to inappropriate estimate of Shapley value (Fig. 2, 3, 4, 5). To mitigate the effect of such unrealistic data, the authors proposed using generative models to sample realistic data inside the data manifold, thus avoiding use of unrealistic data for estimating Shapley value.\n\n### Quality & Clarity\nI think the paper is written clearly, and the main claims are easy to follow.\nThe experiments are designed well to support the claim.\n\n### Originality & Significance\n#### Contribution of the paper\nEssentially, this paper tackles the known problem (avoiding unrealistic data in Shapley value (*1)) by using a common technique (generative model to sample realistic data (*2)). Although the idea is straightforward, the method itself seems to be reasonable and would be useful in some extent.\n\n(*1) The problem considered in this paper is not a new problem. The problem of using unrealistic data for estimating Shapley value is pointed out by several previous studies [Ref1, 2, 3, 4], as some of them are mentioned in the paper. \n\n(*2) The major contribution of this study is therefore on the methods for mitigating the effect of unrealistic data. In this study, the authors proposed using generative models to sample realistic data inside the data manifold. The authors also proposed training a surrogate model $g_y$ that approximates the conditional expectation used for computing Shapley value. To my knowledge, such techniques were not considered in the previous studies.  However, the use of generative models for sampling from conditional distribution itself is a common technique, e.g. [Ref5,6] used GAN and VAE for Model-X Knockoff.\n\n#### Explanation and data manifold\nThe relationship between the data manifold and explanation is studied by [Ref7,8], which is missing in the current paper. It would be great if there are some discussions how the current paper and [Ref7, 8] are similar/different.\nBelow, I list some of them.\n* Similar Points\n  The similar problem, the explanation outside the data manifold, are studied in [Ref7, 8].\n  The use of autoencoder is considered to mitigate the effect of data outside the data manifold.\n* Different Points\n  [Ref7, 8] focus on the gradient-based methods, while the current paper focuses on Shapley value.\nIn addition to similarity/difference, I am also interested in whether the theories of [Ref7, 8] are applicable to Shapley values, or whether there is a fundamental differences between the gradient-based methods and Shapley value.\n\n[References]\n* [Ref1] Problems with Shapley-value-based explanations as feature importance measures, ICML20\n* [Ref2] The many Shapley values for model explanation, ICML20\n* [Ref3] Explaining black box decisions by Shapley cohort refinement, arXiv19\n* [Ref4] Interpretable Machine Learning (Sec 5.10.2), 2019\n* [Ref5] KnockoffGAN: Generating Knockoffs for Feature Selection using Generative Adversarial Networks, ICLR19\n* [Ref6] Deep latent variable models for generating knockoffs, 2020\n* [Ref7] Explanations can be manipulated and geometry is to blame, NeurIPS19\n* [Ref8] Fairwashing Explanations with Off-Manifold Detergent, ICML20\n\n### Pros & Cons\n[Pros]\n* The use of generative models for sampling realistic data from conditional distribution seems to be reasonable to mitigate the effect of unrealistic data used in the current method \"splicing.\"\n\n[Cons]\n* The use of generative models for sampling from conditional distribution itself is a common technique.\n* The relationship between the data manifold and explanation is studied by [Ref7,8], which is missing in the current paper. It would be good to provide some discussions how the current paper and [Ref7, 8] are similar/different.\n\n---\nI read the rebuttal and the updated version of the paper. I think the authors did a good job resolving my concerns on novelty. I updated my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A good paper with significant contribution on XAI and the on- vs off- data manifold explainability.
Reviewers have appreciated authors’ feedback and update of the paper (R1, R2, R4). I would like to personally thank the authors for a smooth, extensive and focused interaction w/ updates.

Decision: Accept (Poster)