WeightedSHAP: analyzing and improving Shapley based feature attributions
Authors: Yongchan Kwon
James Zou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Attribution problem
Model interpretation
Shapley value

Abstract: Shapley value is a popular approach for measuring the influence of individual features. While Shapley feature attribution is built upon desiderata from game theory, some of its constraints may be less natural in certain machine learning settings, leading to unintuitive model interpretation. In particular, the Shapley value uses the same weight for all marginal contributions---i.e. it gives the same importance when a large number of other features are given versus when a small number of other features are given. This property can be problematic if larger feature sets are more or less informative than smaller feature sets. Our work performs a rigorous analysis of the potential limitations of Shapley feature attribution. We identify simple settings where the Shapley value is mathematically suboptimal by assigning larger attributions for less influential features. Motivated by this observation, we propose WeightedSHAP, which generalizes the Shapley value and learns which marginal contributions to focus directly from data. On several real-world datasets, we demonstrate that the influential features identified by WeightedSHAP are better able to recapitulate the model's predictions compared to the features identified by the Shapley value.

WeightedSHAP: analyzing and improving Shapley based feature attributions
Yongchan Kwon Columbia University New York, NY, 10027 yk301 1e20010ccolmmti edu
James Zou Stanford University and Amazon AWS Stanford, CA. 94305 jameszsstanf ord. edu
Abstract
Shapley value is popular approach for measuring the influence of individual fea- tures. While Shapley feature attribution is built upon desiderata from game theory, some of its constraints may be less natural in certain machine learning settings, leading to unintuitive model interpretation. In particular, the Shapley value uses the same weight for all marginal contributions- -iiei. it gives the same importance when large number of other features are given versus when small number other features given. This property can be problematic if larger feature sets are more or less informative than smaller feature sets. Our work performs rigorous analysis of the potential limitations of Shapley feature attribution. We identify simple settings where the Shapley value is mathematically suboptimal by assigning larger attributions for less influential features. Motivated by this observation, propose WeightedSHAP, which generalizes the Shapley value and learns which marginal contributions to foeus directly from data. On several real-world datasets, we demonstrate that the influential features identified by WeightedSHAP are better able recapitulate the model's predictions compared to the features identified by the Shapley value.
Introduction
Explaining how feature impacts model prediction crucial question in machine learning (ML) as it provides a deeper understanding of how the model behaves and what insights have been extracted from data. many real-world applications, it has been increasingly common to deploy complicated models such as deep neural network model or random forest to achieve high predictability. However, often comes with cost of unintuitive interpretations, and it naturally calls for principled and practical attribution method. The goal of this work is to quantify the contribution of individual features to a particular prediction, also known as the attribution problem.
Lundberg and Lee 2017 proposed a model-agnostic attribution method, SHapley Additive ex exPlana- tions (SHAP), based the Shapley value from economics. Supported by theoretical properties that the Shapley value satisfies, SHAP has been a popular method in the attribution literature [Janzing al. 2020 Sundararajan and Najmil 2020). For instance, Frye al. [2020) and Aas al. 2021 developed the SHAP algorithms for dependent features, and Heskes et al. 2020 and Wang et al.
[021] proposed a rigorous framework in causal inference settings. One of the practical problems to use SHAP is the heavy computational costs, and there have been works on improving computational
efficiency [Covert and Lee 202T Lundberg al.) 2020 Jethani et al. 202 In addition. SHAP has
been deployed to various applied scientific research [Lundberg et al. 2018 Janizek et al. 2021 Quu
al. 2022
The Shapley value, mathematical basis SHAP, is a simple average of the marginal contributions that quantify the average change in a coalition function when feature of interest is added from
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

subset of features with a given coalition size. There are different version of the marginal contributions by the coalition size, and the Shapley value takes uniform weight to summarize the influence of a feature. This uniform weight arises due to the efficiency axiom of the Shapley value, which requires the sum of attributions to equal the original model prediction, but it is often problematic because some may be more informative than others. As we will show in Section 3 the Shapley value is not optimal to sort features in order of influence on a model prediction.
Our contributions. While SHAP is widely used for feature attribution, its limitations are still not rigorously understood. We first show the suboptimality of the Shapley value through an analysis of the marginal contributions. We identify a key limitation of the Shapley value in that it assigns uniform weights to marginal contributions in computing the attribution score. We show that this can lead to attribution mistakes when different marginal contributions have different signal and noise. Motivated by this analysis, we propose WeightedSHAP, a generalization of the Shapley value which is more flexible. WeightedSHAP uses weighted average of marginal contributions where the weights can be learned from the data. On several real-world datasets, our experiments demonstrate that WeightedSHAP is better able to identify influential features that recapitulate model's prediction compared to a standard SHAP. WeightedSHAP is a simple modification of SHAP and is easy to adapt from existing SHAP implementations.
1.1 Related works
Model interpretation There are mainly two types of model interpretation depending on the quantity to be accounted for; global and local interpretations. The global interpretation is to explain the impact of feature on prediction model across the entire dataset [iioveesky and Conklin 200T Breiman 2001 Owen 2014 Broto et al.) 22200] Zhao and Hastie 202 Bénard al. [2022] For instance, for a decision tree model, Breiman et al. [017]] measures the total decrease of node impurity at node split by feature of interest an impact. In contrast. the local interpretation is to explain the impact of a feature on a particular prediction value [Lundberg and Leel 2017
Chen et al. 2018 Janzing 2021a
et al. 2020 Lundberg et al. 2020. Heskes al.] [020 Jethani al.
For deep neural network model, a gradient- based method uses the gradient evaluated at a particular input sample as an
impact (Simonyan et al. 2013] Sundararaaan al. 2017 Ancona et al. 2017 Selvaraju et al. 2017 Adebayo et al. 2018). Our work studies the local interpretation problem with focus on marginal contribution-bast based method, which we review in Section 2 The marginal contribution-based method is potentially advantageous over gradient-basee -bsed method as it does not require the differentiability of a prediction model.
Shapley value and its extension The Shapley value, introduced as a fair division method from economics [Shapley] 1953a], has been deployed in various ML problems. One leading application
is data valuation, where the main goal is to quantify the impact of individual data points in model training. Ghorbani and Zou [019 and Jia et al. [019] propose to use the Shapley value for measuring the individual data value, and this concept has been extended to handle the randomness of data [Ghorbani et al., 2020 Kwon et al. 202T As for the other applications of the Shapley value, model explainability [Ghorbani and Zou 2020 model valuation [Rozemberczki and Sarkar [00T]]. federated learning [Liu et al. 2021], and multi-agent reinforcement learning [Li et al.) |021 have been studied. We refer to RRozmberrzkii et al. [2022] for complementary literature review of MI. applications of the Shapley value.
The relaxation of the Shapley axioms has been one of the central topics in cooperative game theory (Shapley 1953b Banzhal III 1964 Kalai and Samet 1987 Weber 1988 Recently, Kwon and Zou 2021] propose to relax the efficiency axiom in the data valuation problem, showing promising results in the low-quality data detection task. Given that Shapley axioms are often not readily applicable to ML problems, relaxing them has the potential to capture a better notion of significance. In this work, we explore the benefits of relaxation of the efficiency axiom on the attribution problem.
2 Preliminaries
We review the marginal contribution and the Shapley value in the context of an attribution problem. We first introduce some notations. For d E N, let C Rd and y S R. be an input space and an output space, respectively. We use capital letter X = (XI, Xa) for an input random

---

variable defined on X, and lower case letter = (#1...., I2) for its realized value. We denote prediction model by f X' V. j N, we set [i] := {l...... _}} and denote a power set of [J] by 2b1, For vector by us := ("),.... "jja) ). We assume that X has joint distribution p (X) such that a conditional distribution iajsss Xs) is well-defined for any subset S $ [d]. With the notations, conditional
R and subset S = .............. st) S [d] we denote subvector
coalition function U (cand) 2/4 + R is defined as follows [undbbrg and Lee 2017
x.f
(1) where the first expectation taken with a conditional distribution (d)s Xs zs) and the second expectation is taken with a joint distribution p( (X)). Here,
(cond) z.f
:= = (laxs Xg = xs] - -[](
slight abuse of notation for f (z (s))ss to describe f(u) where 1t z, if i E S, and u, X, otherwise. By convention, we (cond) (0) 0. A conditional coalition function defined in (cond) ([d) f(x) [X)( and "f.
z.f
Equation
I is prediction recovered after observing partial information compared to the null information. For instance, if S = 0, the first term becomes the marginal expectation [[f( and nothing is recovered by S 0. For ease of notation, we write y (scond) (S) (ccon))()) (S) ) for remaining
x
part of the paper.
Marginal contribution- based attribution methods Given that the goal of the attribution problem assign the significance of an individual feature 24 on the prediction f(z), its primary challenge is how to measure the influence of the feature Zy. A leading approach is to quantify the difference in the conditional coalition function values ((cond) after adding one feature of interest. We formalize this concept below.
Definition 1 (Marginal contribution). For i,j E [d], define the marginal contribution of the i-th feature Fs with respect to j - 1 features as follows.
(x(2,)= = 10
Σ
e (cond) (S - (cond) (S).
(2)
The marginal contribution (x;); considers every possible subset S [d] (i) with the coalition size |S| j j -
and takes simple average of the difference u(cond)( {t)) - y (cond) (S). That is,
it measures the average contribution of the i-th feature z; when it is added to subset S.
Different marginal contributions A, (xx)) have been studied depending on the coalition size j in the literature. Zintgraf et al.
[017 considered j d and measured the leave-one-out marginal Xjdras
contribution Aa(x;)
#[d[\/s3] as an influence of feature. Guyon and Elisseefi [003 considered j = 1 and measured the coefficient determination as an influence of feature. Although they did not an individual prediction, their idea is essentially similar using A (m:) [ (()))) X; 2r]] [x))
f(x) E (x.
Another widely used marginal contribution- -based method is the Shapley value [Lundberg and Lee Covert and Lee, 22021) It summarizes the impact of one feature by taking simple average across all marginal contributions. To be more specific, the Shapley value is defined as follows.
Φshap = (..))
j=1
The Shapley value in (3) is known as the unique function that satisfies the four axioms of a fair division in cooperative game theory (Shapley 1953a] The four axioms and the uniqueness of the Shapley value are discussed in more detail in Appendix.
Although the Shapley value provides principled framework in game theory, one critical issue is that the economic notion of the Shapley axioms is not intuitively applicable to the attribution problem
Kumar et al.) 2020 Rozemberczki et al. 2022 In particular, the efficiency axiom, which requires the sum of the attributions to be equal to condy dd), is not necessarily essential because an order of
attributions is invariant to the constant multiplication. For instance, for any positive constant C' > 0.
3

---

an attribution (1²)) == C' x @shap(zz) will have the same order as the Shapley value Pshap, but the efficiency axiom is not required for dg. In Section 4 we will revisit this point and introduce a new attribution method that relaxes the efficiency axiom.
Evaluation metries for the attribution problem. In the literature, different notions of goodness have been proposed, for instance, the complete axiom (Sundararajan et al. 2017 Shrikumar et al. 2017]. the local Lipschitzness [Alvarez-Melis and Jaakkola 20183. and the explanation infidelity Yeh et al. 2019) with a focus on the total sum of attributions or the sensitivity of attributions. Recently, Jethani et al. [2021a] suggested using the Inclusion AUC to assess the goodness of an order of attributions. Specifically, the Inclusion AUC is measured as follows: Given an attribution method, features are first ranked based on their attribution values. Then the area under the receiver operating characteristic curve (AUC) is iteratively evaluated by adding features one by one from the most influential to the least influential. This procedure generates a AUC curve as function of the number features added, and the Inclusion AUC is defined as the area under this curve. Similar evaluation metrics have been used in Petsiuk et al. (2018) and Lundberg et al. 2022). Following the literature, we consider the area under the prediction recovery eтror curve (AUP) defined as follows.
Definition (Area under the prediction recovery error curve). For given attribution method @. an input E X, and & E [d], let I(k; é, x) S [d] be set of k integers that indicates k most inftuential features based on their absolute value (ob(2;)) For a prediction model f, we define the area under the prediction recovery error curve at x as follows.
d
(); : Σ - -B((x) B(f(X) XI(:)..) xI(k)
(4)
AUP is defined as the sum of the absolute differences between the original prediction f(x) and its conditional expectation B(((X) XI(k:o.>) (I(s)))) when the h most influential features given. Each term in Equation (4) measures the amount of prediction that is not recovered by the k most influential features, and thus this prediction recovery error is expected to decrease as & increases. The prediction recovery error can be described as a function of k; and the AUP measures the area under this function as in the Inclusion AUC.
3 The Shapley value is suboptimal
In this section, we show that the suboptimality of the Shapley value through rigorous analysis of the marginal contribution. We first derive useful closed-form expression of the marginal contribution when is linear and p(X) is Gaussian (Section 3.1). With this theoretical result, we present two simulation experiments where the Shapley value incorrectly reflects the influence of features, resulting in suboptimal order of attributions (Section 3.2).
3.1 A closed-form expression of the marginal contribution
To this end, assume that prediction model f is linear and an input distribution p(X) follows Gaussian distribution with zero mean and a block diagonal covariance matrix. To be more specific, we define some notations. For B E N, we set a vector d = (dI,...,de dg) E ND such that Σ do = d and a vector p := (PI...., PB) E 10, We denote a d x d block diagonal covariance matrix by (block) = diag (m)..))..... ((ggdg)) where (p = (1 La) + Pola, 17, Here, p.d for j E N, we denote the j x identity matrix by 1j. the j-dimensional vector of ones by 1, 1). E IR and 0, := x 1 Lastly, we denote a Gaussian distribution with mean vector and a covariance matrix Σ by ((,,, Σ). With the notations, we assume X ~ NO2,,, (bboock)) That is, every feature is normalized to have a unit variance and is included in one of B independent clusters. For j [B], the size of the j-th cluster is dj. and features are equally correlated to each other within a cluster. The correlation levels can vary from cluster to cluster.
In general, the marginal contribution in Equation 2 does not have closed-form expression, and it makes a rigorous analysis of the Shapley value difficult. In the following theorem, we derive a closed-form expression of the marginal contribution when f is linear and X A 2
(block))

---

Model fax) a:
Model fie) = 0.5
Mocell tu-
Modell 65
Corretation
Corretaiion
Snupley la
Stupley lalk
Shupley Sib
Shapky las
2
-2
(a) Illustrations of the suboptimality of Shapley-based based feature attributions on the four different situations when
d
2.
Prediction recovery emror curve
Prediction recovery entrr curve Correlation on: p 0.6
Correlation: 0.2
Correlation; a 02
Correlation: p 0.6
Ks mell
A Shapley
Au Shapley
B
Au
AUP
AUP
(fbo)
23
Shapley h2
Shapley
Ag
The number features added
The number festures added
(b) Illustrations a prediction recovery error curve and AUP comparison when 100.
Figure The Shapley value is suboptimal. (Top) a region described in black denotes the area the Shapley value fails to select the more influential feature. We encode {1; f) {(2 æ. f' as the background color visualize which feature is influential. Blue color describes region where the first feature æ1 is more influential, i.e., {11; f) 8 { f), and red describes region where the second feature 22 is more influential, i.e., {11 x; f) E(2 p f). The intensity for E1; ʃ) {2; æ, is described in color bar. (Bottom) we compare the three attribution methods A1, Oshap; and As the different situations by varying correlation p {0.2, 0.6). As for the prediction recovery error curve, we denote a 95% confidence band based on 100 samples. The lower AUP is, the better. In both settings, the Shapley value is suboptimal according AUP.
Theorem 1 (A closed-form expression for the marginal contribution). Suppose f(x) = Bo + xx for some (Bo, 8) R x Rd and X NO,,,, _p -(hlock)), Then, for i, [d]. the marginal contribution of the i-th feature x; with respect to samples is expressed
(=, x" (f Hi.
for some explicut matrix H(i.j) E Rddd
proof and the explicit term for H(i.j) are provided in Appendix. Theorem [ shows that the marginal contribution is a bilinear function of an input and the estimated regression coefficient
3. One direct consequence is that the Shapley value also has bilinear form Φshap (x;) = x (1)
for (ii) == E; Hii, We emphasize that this bilinear form greatly improves computational efficiency. Specifically, for all i E [d], since the term (i,j)
3 is not function of input æ, only need to compute the one-time in multiple attribution computations. Moreover, it also leads to a memory efficient algorithm there is need store the d x d matrix H(ijj)
3.2 Motivational examples
With the theoretical result introduced the previous subsection, we show that the Shapley- feature attribution is suboptimal and fails to assign larger values to more influential features.
When there are two features. When = 2, there are only two possible values for AUP. For any attribution method o,
E(1;2, if = {1}
AUP(O:
E(2;
otherwise
5

---

where E(k; x, f):= - Xx = xx] for k {1,2}. Therefore, the optimal order based on AUP is fully determined by [(1; x, f) and {(;; 2, f), for instance, the first feature œ1 is more influential than the second one 22 if (1: E1; f) { x, means that the original prediction f(x) is more accurately recovered by the first feature Ij than the second one 222.
E(2; f). is intuitively sensible because
Using the optimal order, demonstrate that the Shapley value does not necessarily assign large attribution to a more influential feature. We consider the four different scenarios with two different prediction models fx) E {1.5z, + 22, 0.5z, + z2) and two different Gaussian distributions, X~N (02, (0) for p E {0.2, 0.6). In these four scenarios, the terms E(1; xr, f) and {2; J) have a closed-form expression, and thus the optimal order is explicitly obtained. Moreover, due to Theorem more influential feature according to the Shapley value is explicitly obtained.
Figure ((a) illustrates the suboptimality of the Shapley value on the four different situations. In any situation, there is a non-negligible region (described in black) where the Shapley value fails to select more influential feature. In addition, this suboptimal area increases as the correlation gets larger, showing that the Shapley value-based explainability becomes poor when features are highly correlated.
When there are more than two features When d > 2, it is infeasible to find the exact optimal order because there are 2d-1 possible AUPs. For this reason, we compare the Shapley value with the two commonly used marginal contribution-based methods A and Aa. showing the Shapley value not optimal in terms of AUP. We assume the following setting: a trained model is linear f(x) = 3. + x² 3 for some (Bu, B) E R x Rd and an input vector X = (Xi,..., X2) follows a Gaussian distribution N (0z, That is, there are features and they are equally correlated to each other with the correlation p. We set d = 100 and consider two different situations by varying P E {0.2, 0.6]. Similar to the previous analysis, due to Theorem [ the three attribution methods are explicitly obtained. We evaluate the prediction recovery error and the AUP the 100 held-out test samples randomly drawn from the distribution / (Oa. M(p.4)) Detailed information is provided in Appendix.
Figure (bb) illustrates the suboptimality of the Shapley value when d = 100 and E {0.2, 0.6). In any situation, the prediction recovery curves for the Shapley value (described in green) have a steeper slope than A (described in red), but is not optimal as the Aa (described in blue) approaches to zero faster. When p = 0.6, the suboptimality becomes more severe in that the gap between Aa and the Shapley value gets larger.
4 Proposed method: WeightedSHAP
Our motivational examples the previous section suggest that the Shapley value does not necessarily assign larger attributions for more influential features, leading to suboptimal order of features. In fact, the last marginal contribution As outperforms other attribution methods Figure [(b) Although the use of A₂ is promising, we show that focusing one marginal contribution might lead to an more
unstable attribution method, suggesting a weighted mean of the marginal contributions.
concrete, we first examine the estimation error of the marginal contribution in the following.
Analysis of estimation error In practice, the Shapley value needs to be estimated, resulting in an estimation error. Given the mathematical form of the Shapley value in (3), this estimation error arises from the estimation error of the marginal contribution. In this reason, we investigate the estimation error of the marginal contribution. We consider the same setting used in Figure (bb) with p 0.6. As for the estimation of the marginal contribution, we follow a standard algorithm to estimate a conditional coalition function of (ccnnd) used in Jethani al. [202 and sampling- caaplnng-based based algorithm to approximate Aj. A detailed explanation for the estimation procedure and the additional result for p 0.2 are provided Appendix.
Figure 2 shows the relative difference between the true marginal contribution Aj obtained by Theorem 1 and its estimate Å as function of the coalition size j E [d]. Here, we use the relative

---

difference between A and B defined as |A-
- B/ max([A], [B) to avoid numerical instability that can be occurred by too small marginal contribution values. It shows the Ag. the most informative marginal contribution in Figure [(b)] has the largest relative difference from the true value. In other words, Aa has the largest signal to explain a model prediction, but at the same time, is the most unstable in terms of the estimation error. This finding motivates us to consider weighted mean of the marginal contributions that can reduce the estimation error while maintaining signals.
Relative difference btw 4 and 4 Correlation: p 0.6
Proposed method For a weighted vector (W1,-.., W2) such that Σj=1 Wy and wij 2 0 for all j E [d], we consider a weighted mean of the marginal contributions
9.0-
8.5
dw (1i) == Σ wy (
8.0
A weighted mean ow (1;) is expected to capture the in- fluence of features better than the Shapely value (3) by assigning a large weight to important marginal contribu- tions. As for the game theoretic interpretation, a math- ematical form of Equation (5) is known as a semivalue in cooperative game theory. It satisfies all the Shapley axioms but the efficiency axiom, which is not crucial in the attribution as we discussed in Section Dubey and Weber 1977 Ridaoui et al., 2018). Due to the relaxation of the efficiency axiom, semivalue not uniquely deter- mined, but it is known that the semivalue is almost unique up to weighted mean operation. A detailed explanation for the semivalue is provided in Appendix.
7.5-
5
20
35 50
65
80
95
Coalition size j
Figure 2: Illustrations of the relative dif- ference between the true marginal contri- bution Aj and its estimate Å, as func- tion of the coalition size j E [d]. We con- sider the same setting used in Figure [(b) The As is shown to have the largest rela- tive difference.
2
One natural and practical question that arises when using
a weighted mean @w (x:) is how to select the weight vector w. Since a weight vector to be selected is desired to have certain good property, this question can be rephrased to which concept of goodness should be optimized. However, it is difficult to have one universal desideratum by the intricate nature of model interpretations. There are different types of goodness and they often represent independent characteristics, as we discussed in Section In other words, a good attribution essentially depends on a practitioner's downstream task. To reflect this, we propose to learn a weight vector that optimizes
a user-defined utility. To be more specific, we let W S {w E Rd Σن= " = 1, wy 2 0} be a parametrized family of weights and T be a user-defined utility function that takes as input an attribution method and outputs its utility. Without loss of generality, we assume that the larger T is, the better it is (e.g., the negative value of AUP). Given and W, we propose WeightedSHAP as follows.
@WeightedSHAP (T, (T,W) == @w (T,W)
(6)
where w' user-defined utility.
"T, (T,W) := argmax ((o)). That is, we learn the optimal weight by optimizing a
When W includes the uniform weight (1/d,..., /dd e Rdd, then by its construction, we can guarantee that WeightedSHAP is always better than equal to the Shapley value according to the utility T. For instance, when the negative value of AUP is used for the utility T, AUP of WeightedSHAP is less than that of the Shapley value, i.e., AUP(dWaighnedHHAR Moreover, the more weight vectors are in W. the better the quality of WWeightedSHAP is guaranteed. WeightedSHHA dWeightedSHAPP depends on a set W. In our experiments, we parameterize an element w E W by the Beta distribution inspired by mathematical properties of the semivalue in Monderer and Samet 2002 Theorem 11]. Detailed information is provided in Appendix.
< AUP AUP((shaa))
Example 1 (WeightedSHAP and the Shapley value dshap on AUP). We revisit the motivational
example introduced in Section 3.2 With the negative AUP for T and some W 2 [Ad; (shap).
WeightedSHAP achieves significantly lowerAUP than both AA and @shap- Specifically, when (d, p) (100, 0.6), the AUPs of (Da, Dshaps (weeightedSHAP) are (1.49 + 0.06, 1.65 + 0.08, 0.77 + 0.03). respectively, where the numbers denote "mean + standard error" based on the 100 held-out test
7

---

Prediction
Prediction
Dotaset: boston
Dataze:: siroll
Dotazer: whitswine
Dataset: sbalonc
(060 -10000 X. mill
(0000) Me 2011
mightesii
[100 1500 XC
MC1
Shopky
(000 10000
21
Number of features added
Number of features acded
Number of features added
Number of featuras added
(a) Illustrations of the prediction recovery error curve on the four regression datasets. The lower, the better.
Inclusion MSe
Inclusio rese
Inckaiin KSF
Inc MSF acalene
Donasct:
Datasee: sirfol
whitwwine
PQ Shaply mayyreetior
MCI
Shepley
Shaup by
Shuske
HISE
0.9
635-
Number adoud
Number
(b) Illustrations of the Inclusion MSE curve on the four regression datascts. The lower, the better.
Figure 3: Regression tasks. Illutrations of the prediction recovery error curve and the Inclusion MSE curve function of the number of features added. We add features from most influential the least influential. We denote a 95% confidence interval based on 50 independent runs. WeightedSHAP achieves a significantly smaller MSE with fewer features than the MCI and the Shapley value.
samples. Meanwhile, the estimation errors of (Dat: Oshap @WeightedSHAP) are (9.23 + 0.02, 6.16 + 0.04, 7.90 + 0.11), respectively. In short, WeightedSSIAP achieves significantly lower estimation error than Aa while achieving the lowest AUP. Although @shap achieves the lowest estimation error, its AUP is significantly greater than both Aa and WWeightedSHAP. The uniform weight in Φshap helps reduce the estimation error, but loses signals too much. In contrast, WeightedSHAP well balances the signal and the estimation error, i.e., reducing the estimation error while taking more signals.
Implementation algorithm for WeightedSHAP Given a finite set W and an easy-to- co-ttut utility function T, the optimal weight w can be achieved by iteratively evaluating the utility T for each attribution method @w with w E W. In addition, &w is readily obtained as long as there are the marginal contribution estimates. Therefore, the key part of the implementation algorithm is to estimate a set of marginal contributions. The estimation of the marginal contributions consists of two parts, estimation of a conditional coalition function efrnnd) and approximation of the marginal contribution Aj- As for the first part, train surrogate model that takes as input a subset of input features and outputs a conditional expectation of a prediction value given the same subset [Frye] et al. 2020 Jethani al. 2021a b) It is known that this surrogate model unbiasedly estimates conditional expectation of a prediction value given a subset of features under mild conditions [Frye
let al., 2020 Covert al. 2020). Regarding the second part, a weighted mean is approximated by sampling-based algorithm [Ghorbani and Zou| 2019 Kwon and Zou| 2021]. We provide a pscudo algorithm in Appendix. In terms of the computational cost, our algorithm is comparable to standard the Shapley value estimation algorithm because both algorithms need to estimate the marginal contributions as a primary part [Lundberg and Lee 2017 Frye et al.) 2020]. For instance, with the classification dataset fraud, the marginal contribution estimation part takes 20.7 seconds per sample on average but the weight optimization part only takes 0.18 seconds, i.e. the weight optimization part is only 0.86% of the total compute.
5 Experimental results
We demonstrate the practical efficacy of WeightedSHAP on various regression and classification datasets. We compare WeightedSHAA with the marginal contribution feature importance (MCI) proposed by Catav et al. (202T) and the Shapley value on the prediction recovery error task and the inclusion performance task. Each task assesses the goodness of attribution order by iteratively
8

---

Dataser: fraud
Donaact phonome
Dataset wind
Datagot: coU
0212
PO
1000 Sr00 7C x000
MCI Weightesii
WiightddSigg upp
Widg #idghausSi0
7
X
+ 3:07
0.17
6.74
1000
0.00
15
Number featuris added
Number of festures added
Number of features added
Number of features added
(a) Illustrations of the prediction recovery error curve on the four binary classification datasets. The lower, the better.
Incuuton AUC
incusion Dalasell: plonerm
Inclusion AUC
Incluston
Deliovel: freidd
Daliisell: wind
Dar soet: upu
0.03
PRC)
6:20
PO
6.5
90
MC1 Shaathy
Shopphy
Stupky
Shapky
642-
0.00
35
V..
15
15 Numbbe of fechures wddted
25
10
35
Numbber fentures added
Number fnatures woded
Number featurees addled
(h) Illustrations of the Inclusion AUC curve on the four binary classification datasets. The higher, the better.
Figure 4: Classification tasks. Illustrations of the prediction recovery error curve and the Inclusion AUC curve function of the number of features added. Details are provided in Figure 3 Weighted- SHAP achieves a significantly higher AUC with fewer features than the MCI and the Shapley value.
measuring how much the original model prediction or its performance is recovered with given number of features. As for the model performance, we evaluate mean squared error (MSE) and AUC for regression and classification problems, respectively. We consider a gradient boosting model for a prediction model f. As for the surrogate model in coalition function estimation af g(cond), we use a multilayer perceptron model with the two hidden layers. and each layer has 128 neurons and the ELU activation function [Clevert al. 2015]. As for the WeightedSHAP in (6). we use the negative value of the AUP for T and set W that has 13 different weights including A1, Oshap, and Az. All the missing details about numerical experiments are provided in Appendix, and our Python-based implementations are available at htttttts//giiiiii wttp:/tttth......oooeeeeeeeeeeee cww wowo://ooo0000007 wwtp:/on0007//iiiggteeiiii Www.//eetitttttAape/tt
Figure ((a) compares the prediction recovery error curves for the WeightedSHAP (described in red) with the MCI (described in blue) and the Shapley value (described in green). WeightedSHAP shows always lower prediction recovery errors than the MCI and the Shapley value. Given that
WeightedSHAP minimizes the AUP, which is the sum of prediction recovery error If(x) (X)
Xs = =s|l WeightedSHAP does not necessarily have a smaller prediction recovery error for every number of features added. As for the MSE, Figure shows that WeightedSHAP has significantly smaller MSE than baseline methods with fewer features. In particular, on the airfoil dataset, WeightedSHAP achieves 0.53 MSE with 10 features, but the Shapley value never achieves this value because of the suboptimality of the attribution order.
We also evaluate the prediction recovery error and AUC for the four classification datasets. Similar to the regression cases, Figures (aa and [[b]sshww that WeightedSHAP effectively assigns larger values for more influential features and recovers the original prediction fxx) significantly faster than the MCI and the Shapley value. Specifically, on the fraud dataset, WeightedSHAP achieves 0.995 AUC with 14 features, but the Shapley value always has the lower AUC value. Our findings are consistently observed with different model for f or other datasets. Additional experimental results with different evaluation metrics and a qualitative assessment of WeightedSHAP are provided in Appendix.
5.1 Illustrative examples from MNIST
We present qualitative assessment of WeightedSHAP and examine how its top influential features differ from those from SHAP using the MNIST dataset. We train a convolutional neural network model using the same setting suggested in Jethani et al. [02 It achieves 98.6 % accuracy

---

WeightedSHAP
SHAP AUP
WeightedSHAP AUP-2252
SHAP AUP=7.45
AUP=
9
9
WeightedSHHP
SHAP AUP=5
WeightedSHAP AUP=1.25
SHAP 3.89
AUP=
Figure 5: Illustrative examples of WeightedSHAP and SHAP attributions MNIST images. We present the top 10% of influential features. Red (resp. blue) color indicates the corresponding feature positively (resp. negatively) affects the model prediction. (Top) WeightedSHAP clearly captures the last stroke of nine while SHAP fails to capture it. (Bottom) While SHAP has noisy negative feature attributions described by blue pixels, WeightedSHAP provides noiseless and intuitive explanations.
on the test dataset. We select illustrative images with significant difference in AUPs between WeightedSHAP and SHAP.
Figure compares the top 10% influential attributions for WeightedSHAP and the Shapley value. On the top images, while SHAP fails capture the last stroke of digit nine, which crucially important stroke to differentiate from the digit zero, WeightedSHAP clearly captures the strokes. On the bottom images, SHAP produces unintuitive negative attributions, providing noisy explanations. In contrast, WeightedSHAP provides noiseless and intuitive explanations.
Conclusion
In this paper, we provide analysis of the widely used SHAP attribution method. We discover that even in simple natural settings, SHAP can incorrectly identify important features. Mathematically, key limitation of SHAP in that it assigns uniform weights all marginal contributions. We propose WeightedSHAP which generalizes the Shapley value by relaxing the efficiency axiom. WeightedSHAP learns to pay more attention the marginal contributions that have more signal prediction, assigning larger attributions for more influential features. There are several limitations of WeightedSHAP that motivate interesting future works. Here use the AUP metric optimize the weights because AUP is commonly used in practice. However, there is no agreed-upon upon metric for evaluating feature attribution methods. Different users may care about different notions of attribution. Developing variants of marginal contribution weighting optimized for different applications is an important direction of future research. We believe that the core contribution of this paper- that the uniform weighting used by SHAP can be suboptimal-still provides useful insights for these investigations. Here we focus our experiments on directly comparing WeightedSHAPP with SHAP because our goal is to characterize the limitations of SHAP. There is large body of works comparing SHAP with other attribution methods that are complementary to our work (Ych et al. 2019 Jethani al. 2021a].
Acknowledgment
The authors would like to thank all anonymous reviewers for their constructive comments. We also would like to thank Ruishan Liu for the helpful discussion.
References
Scott Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Pro-
ceedings of the 31st international conference neural information processing systems, pages
10

---

4768 2017.
Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explain- able ai: causal problem. In International Conference on Artificial Intelligence and Statisties, pages 2907-2916. PMLR, 2020.
Mukund Sundararajan and Amir Najmi, The many shapley values for model explanation. In International conference on machine learning, pages
PMLR, 2020.
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and llya Feige. Shapley explainability on the data manifold. arXiv preprint arXiv:2006.0127. 2020.
Kjersti Aas, Martin Jullum, and Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations shapley values. Artificial Intelligence, 298:103502, 2021.
Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. Advances neural information processing systems, 33:4778 4789,, 2020.
Jiaxuan Wang, Jenna Wiens, and Scott Lundberg. Shapley flow: A graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics, pages 721-729. PMLR, 2021.
Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pages 3457-3465.
PMLR, 2021.
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence, (1)):56667, 2020.
Neil Jethani, Mukund Sudarshan, Ian Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. arXiv preprint (eXXi:2107.0744: 2021a.
Scott Lundberg, Bala Nair, Monica S Vavilala, Mayumi Horibe, Michael Eisses, Trevor Adams, Wai Low, Shu-Fang Newman, Jerry Kim, et al. Explainable machine-
David E Liston, Daniel King-Wai learning predictions for the prevention of hypoxaemia during surgery. Nature biomedical engineer- ing, 2(10)7749-760, 2018.
Joseph D) Janizek, Ayse Berceste Dincer, Safyy Celik, Hugh Chen, William Chen, Kamila Naxerova, and Su-In Lec. Uncovering expression signatures of synergistic drug response using an ensemble of explainable ai models. bioRxiv, 2021.
Wei Qiu, Hugh Chen, Ayse Berceste Dincer, Scott Lundberg, Matt Kaeberlein, and Su-In Lee. Interpretable machine learning prediction of all-cause mortality. medRxiv, pages 2021-01, 2022.
Stan Lipovetsky and Michael Couklin. Analysis of regression game theory approach. Applied Stochastic Models in Business and Industry, 17(4):319-330 2001.
Leo Breiman. Random forests. Machine learning. 45(1):5-32, 2001.
Art B Owen. Sobol' indices and shapley value. SIAM/ASA Journal on Uncertainty Quantificutio, 2 (1):245-251, 2014.
Baptiste Broto, François Bachoc, and Marine Depecker. Variance reduction for estimation of shapley effects and adaptation unknown input distribution. SIAM/ASA Journal Uncertainty Quantification, 8(2))693-716. 2020.
Qingyuan Zhao and Trevor Hastie. Causal interpretations of black-box models. Journal of Business & Economic Statistics, 39(1)272--281 1021.
Clément Bénard, Gérard Biau, Sébastien Da Veiga, and Erwan Scornet. Shaff: Fast and consistent shapley effect estimates via random forests. In International Conference on Artificial Intelligence and Statistics, pages 5563-5582. PMLR, 2022.

---

Leo Breiman, Jerome H Friedman, Richard A Olshen, and Charles J Stone. Classification and regression trees. Routledge, 2017.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information- theoretic perspective model interpretation. In International Conference on Machine Learning, pages 883-892. PMLR, 2018.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.603: 2013.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328, PMLR, 2017.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-basee based attribution methods for deep neural networks. arXiv preprint arXiv: 111///1111111 1.06104, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In Proceedings of the IEEE international conference on computer vision, pages 6186626, 2017.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28):307-317, 1953a.
Amirata Ghorbani and James Zou, Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. Proceedings the VLDB Endowment, 12(11):1610-1623, 2019.
Amirata Ghorbani, Michael Kim, and James Zou. distributional framework for data valuation. In International Conference Machine Learning, pages 3535-3544. PMLR, 2020.
Yongchan Kwon, Manuel A Rivas, and James Zou. Efficient computation and analysis of distributional shapley values. In International Conference on Artificial Intelligence and Statistics, pages 793-801. PMLR, 2021.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Advances in Neural Information Processing Systems, 01-5922--9332 2020.
Benedek Rozemberezki and Rik Sarkar. The shapley value of classifiers in ensemble games. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1558-1567, 2021.
Zelei Liu, Yuanyuan Chen, Han Yu, Yang Liu, and Lizhen Cui. Gtg-shapley: Efficient and accurate participant contribution evaluation in federated learning. arXiv preprint arXiv: 10:iv:2109.02000 109.02053, 2021.
Jiahui Li, Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agem reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 934-942, 2021.
Benedek Rozemberezki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. The shapley value in machine learning. arXiv preprint arXiv:2202.05554, 2022.
Lloyd S Shapley, Additive and non-additive set functions. Princeton University, 1953b.
John Banzhaf III. Weighted voting doesn't work: A mathematical analysis. Rutgers Rev., 19:
317, 1964.
12

---

Ehud Kalai and Dov Samet. On weighted shapley values. International journal of game theory, 16 (3):205-222, 1987.
Robert J Weber. Probabilistic values for games. The Shapley Value. Essays in Honor of Lloyd S. Shapley, pages 101-119, 1988.
Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:21 10.449,, 2021.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint IXX::1702.04595, 2017.
Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. Journal of machine learning research, 3(Mar): 115--1182 2003.
Elizabeth Kumar, Suresh Venkatasuurammanian, Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value-based explanations as feature importance measures. In International Conference on Machine Learning, pages 5491-5500. PMLR, 2020.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pages 3145- 3153. PMLR, 2017.
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.0800 2018
Chih-Kuan Yeh, Cheng-Yo Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, 32, 2019.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:/806.07440 2018.
Pradeep Dubey and Robert James Weber. Probabilistic values for games. Technical report, YALE UNIV NEW HAVEN CONN COWLES FOUNDATION FOR RESEARCH IN ECONOMICS, 1977.
Mustapha Ridaoui, Michel Grabisch, and Christophe Labreuche. An axiomatisation of the banzhaf value and interaction index for multichoice games. In International Conference on Modeling Decisions for Artificial Intelligence, pages 143-155. Springer, 2018.
Dov Monderer and Dov Samet. Variations on the shapley value. Handbook of game theory with economic applications, 3:2055-2076, 2002.
Neil Jethani, Mukund Sudarshan, Yindalon Aphinyanaphongs, and Rajesh Ranganath. Have we learned to explain?: How interpretability methods can learn to encode predictions in their inter- pretations. International Conference on Artificial Intelligence and Statistics, pages 1469-1467. PMLR, 2021b.
Ian Covert, Scott M Lundberg, and Su-In Lee. Understanding global feature contributions with additive importance measures. Advances in Neural Information Processing Systems, 33:17212- 17223. 2020.
Amnon Catav, Boyang Fu, Yazeed Zoabi, Ahuva Libi Weiss Meilik, Noam Shomron, Jason Ernst, Sriram Sankararaman, and Ran Gilad-Bachrach. Marginal contribution feature importance an axiomatic approach for explaining data. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine URL http::://prcceeddiiiiii
Learning Research, pages 1324-1335. PMLR, 18-24 Jul 2021.
mlr.
html
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (clus). arXiv preprint arXiv:1511.07289 2015.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We describe the limitations the conclusion section.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] We provide proof in Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We provide used codes in the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We provide implementation details in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [No] It not the main focus of our paper.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yhVJX4k3QGo", "reviewer": "Reviewer_AA8w", "summary": "The paper considers the task of feature attribution (specifically, from the perspective of correctly capturing the *order* of attributions). It demonstrates that a common feature attribution method based on the Shapley value (SHAP) is sub-optimal in the sense that there exist instances in which it does not correctly identify the order of important features. It proposes a modification of SHAP that relaxes the efficiency axiom, where the marginal contributions can be weighted arbitrarily, and the optimal weighing is learned from data.\n\n", "questions": "1. The current paper seems to suggest that ideally (if estimation weren't an issue), we would assign higher importance to $\\Delta_d$ than the importance assigned by the Shapley value. If I understand correctly, this seems to contradict part of the motivation for removing the efficiency axiom in Kwon & Zou (they show $\\Delta_j$ becomes less informative as $j$ approaches $d$, mostly because the marginal contribution of a single element to a large training data diminishes). Can you clarify this point?\n\n2. Not only does the assumption that the set $W$ is finite seems limiting, but I was surprised to see that the cardinality of $W$ in the experiments was very small (13!). Several questions: Was $W$ chosen to be so small for computational reasons, i.e. because the algorithm is basically a for loop over the elements in $W$? How did you select the set $W$? Are the results sensitive to this choice? (the paper points to the appendix for details but I could not locate them).\n\n3. Can you demonstrate how weightedSHAP behaves on the examples from Section 3? (see above)", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is written clearly and related work is discussed properly. Understanding the limitations of current feature attribution methods is an important area of research. \n\nWeaknesses:\n\n1. One general weakness of this line of work is that by questioning whether the assumptions made by the Shapley value are relevant to the objective of correctly identifying the order of feature attributions (\"given that Shapley axioms are often not readily applicable to ML problems\"), the choice to stay within the general framework of Shapley seems more questionable. I understand that the efficiency axiom doesn't seem important for correctly identifying the order of important features - is the linearity axiom any more relevant than that? If one could write a similar paper proposing to relax the linearity axiom, then isn't just the takeaway here that the entire approach of using the Shapley value is not optimal? To be clear, this isn't an argument specifically against the current paper, but I do think it's important to keep in mind when considering the contributions & impact of the paper.\n\n2. I think the connection between the observations regarding sub-optimality of SHAP (Section 3) and the proposed method (Section 4) could be articulated better. E.g. the main takeaway from the first negative example in Section 3 is that SHAP fails when the features are highly correlated. How does allowing the weights of the marginal contributions to be non-uniform address this issue?  I'm not sure I understand how the explanation regarding the issue of estimation error in the beginning of Section 4 is technically related to the negative results of Section 3. Ideally, it would be good to revisit the motivational examples from Section 3 and demonstrate how weightedSHAP improves on them (e.g. reduces the region of failure).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Gvrvzr7sqlU", "reviewer": "Reviewer_SvbQ", "summary": "The paper investigates a limitation in the feature attribution method SHAP. To ameliorate this limitation, the authors propose a method WeightedSHAP, which generalizes SHAP to use a weighted average over coalition sizes. The optimal weighting is chosen based on a given utility function and set of possible weights. Experiment results show that the attributions generated WeightedSHAP have better performance than Shapley values on the AUP metric.", "questions": "* The experiment in Figure 2 shows the relative difference between $\\hat \\Delta_j$ and $\\Delta_j$, but how is this difference reduced by using the weighted mean? Is there a corresponding figure plotting the relative difference after applying the weighted mean?", "limitations": "Limitations have been adequately discussed in the main text.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths:**\n* The investigated problem is important; SHAP is a popular method for approximating feature attributions. The proposed method shows improvement over the baseline method on a key metric, and has a nice game-theoretic interpretation.\n* The paper is well-written and well-motivated, with clear and concise notation.\n\n**Weaknesses:**\n* Limited comparisons. The only comparison is to the unweighted Shapley Value on AUP. The paper could be improved by including other metrics and/or value functions. For example, it would be interesting to see if improved performance on AUP generalizes to performance on other metrics (e.g. ROAR [1]). There are also other methods that similarly aim to improve the Shapley value's limitations w.r.t. feature correlation. For example, [2] proposes a modified value function. [3] proposes a feature attribution ranking that incorporates feature interactions. Comparisons to such methods would improve the paper.\n* No qualitative analysis. The proposed method shows improvement on AUP, however it would be beneficial to see how the actual feature importance ordering changes. Ideally there would be some user study, but even qualitative examples would improve the paper\n* No time comparison or time complexity analysis. The main text states that the \"[computation cost] is comparable to the Shapley value estimation algorithm\", however the WeightedSHAP algorithm must add some nonzero computation expense which likely increases as size of $\\mathcal{W}$ and / or number of features increases. The experiments also use relatively small datasets and |\\mathcal{W}| seems relatively small.\n\n**Miscellaneous Minor Issues:**\n* Typo in Line 132: \"efficiency axiom does is not required\"\n* Line 279: \"WeightedSHAP is always better than the Shapley value\". Minor nitpick, however this statement could be misleading, given that it's possible for WeightedSHAP to equal performance of Shapley value (as stated in the next sentence). \n\n**Summary:**\nThe paper is well-motivated, and the proposed method shows improvement over the unweighted Shapley value on the AUP metric. I have concerns with the limited experiments (see above); including additional qualitative comparisons and comparisons with other metrics would improve on this point.\n\n\n[1] Hooker et al, A Benchmark for Interpretability Methods in Deep Neural Networks  \n[2] Catav et al, Marginal Contribution Feature Importance - an Axiomatic Approach for   Explaining Data  \n[3] Masoomi et al, Explanations of Black-Box Models based on Directional Feature Interactions  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G34PxQ8bZpJ", "reviewer": "Reviewer_hhZc", "summary": "The paper illustrates that regular SHAP does yield a good AUP. They provide examples of why this is the case and provide arguments that regular SHAP is suboptimal. They propose WegihtedSHAP as a solution to improve the AUP, which parametrizes the marginal contributions with a weight that is optimized on the user-specified utility function $\\mathcal{T}$. Experiments the demonstrate the advantage of WeightedSHAP.", "questions": "1. It's not clear what $\\mathcal{T}$ exactly means, can you give an example?\n2. I've understood that suboptimal refers to that AUP \"converges\" slower when you add the top k important features. Would it be possible to characterize this in terms of a convergence rate?\n\n------\n\nPost-rebuttal comments:\n\nI'd like to thank the authors for their response and clarification. Happy to keep my score at 6, I think the image explanations are a nice addition.\n\n", "limitations": "The paper considers tabular data for regression and classification. It would be interesting to understand if this could be extended to consider image data as well.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positive points:\n1. Clearly defined problem - It's well-formulated through empirical evidence\n2. Solution is quite simple\n3. Experiments convincing \n\nNegative points:\n1. Writing could be clearer\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2_-hRaLSRY6", "reviewer": "Reviewer_E2Dv", "summary": "This paper found that the conditional expectation-based Shapley value might fail in some cases, and they thought the problem was caused by the fact that the Shapley value uses the same weight for all marginal contributions. They proposed WeightedSHAP where they assigned a learnable weight to the marginal contributions and make the WeightedSHAP equal to the weighted average of the marginal contributions. ", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strength]\n1. This paper showed a significant problem of the conditional expectation-based Shapley value and provided detailed theoretical discussion about it.\n\n[Weakness]\n1. This paper only discussed about the conditional expectation-based Shapley (CES) value, while baseline value-based Shapley (BShap) value was not discussed. [cite 1] has already showed that the CES didn't satisfy the dummy and linearity axioms, while BShap could satisfy all four axioms. Therefore, it is not surprising that the CES might fail in some special cases. \n\n[cite 1] Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International conference on machine learning, pages 9269–9278. PMLR, 2020\n\n2. The WeightedSHAP lacks of theorectical soundness. In addition to efficiency axiom, due to the nature of CES, the weightedSHAP also does not satisfy the dummy and linearity axioms, thus this method does not satisfy three of the four axioms of the Shapley value. And, the weight vector of weightedSHAP are learned from some utility function which might cause bias towards the model explanation and influence the objectivity of the explanation. Therefore, the WeightedSHAP lacks of theorectical soundness.\n3. The experimental results in this paper may not be objective. The weight vector of the WeightedSHAP is optimized w.r.t the utility function AUP, while some similar metric AUC or MSE curve are used to compare the Shapley and WeightedSHAP. \n4. The experiment was not sufficient. Other than the method proposed in [cite 2], some other Shapley-value based explaination methods (like DeepSHAP [cite 3], etc.) should be also compared with WeightedSHAP.\n\n[cite 2] Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. arXiv preprint arXiv:2006.01272, 2020.\n\n[cite 3] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pages 4765–4774, 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept