Joint Shapley values: a measure of joint feature importance
Authors: Chris Harris
Richard Pymar
Colin Rowat
Venue: ICLR 2022 Poster (2022)
Keywords: explainable AI
Shapley value
interaction index
cooperative game theory

Abstract: The Shapley value is one of the most widely used measures of feature importance partly as it measures a feature's average effect on a model's prediction.  We introduce joint Shapley values, which directly extend Shapley's axioms and intuitions: joint Shapley values measure a set of features' average effect on a model's prediction.  We prove the uniqueness of joint Shapley values, for any order of explanation.  Results for games show that joint Shapley values present different insights from existing interaction indices, which assess the effect of a feature within a set of features.  The joint Shapley values seem to provide sensible results in ML attribution problems.  With binary features, we present a presence-adjusted global value that is more consistent with local intuitions than the usual approach.

Published as a conference paper at ICLR 2022
JOINT SHAPLEY VALUES: A MEASURE OF JOINT FEA-
TURE IMPORTANCE
Chris Harris
Richard Pymar
Tokyo. Japan
Economics, Mathematics & Statistics Birkbeck College University of London, UK r p-ppyarr @ubb..rbbbb.... ac. uk
Raptor Financial Technologies chris wwwrrsaarrrhheeooe com
Colin Rowat
Economics
University of Birmingham, UK
C. rowat @bham. ac uk
ABSTRACT
The Shapley value is one of the most widely used measures of feature importance partly it measures feature's 's average effect model's prediction. We intro- duce joint Shapley values, which directly extend Shapleyys axioms and intuitions: joint Shapley values measure set of features' average contribution to model's prediction. We prove the uniqueness of joint Shapley values, for any order explanation. Results for games show that joint Shapley values present different insights from existing interaction indices, which assess the effect of a feature within set of features. The joint Shapley values provide intuitive results in ML attribution problems, With binary features, we present a presence-adjusted global value that is more consistent with local intuitions than the usual approach.
INTRODUCTION
Game theory's Shapley value partitions the value arising from joint efforts among individual agents
(Shapley 1953). Specifically, denote by N = {1, fl...., n. set of agents, and by G.V the set of games on N, where a game is a set function from 2 to with v(2) = 0. Then uSS) the worth created by coalition S C N. When there is no risk of confusion, we omit braces to indicate singletons (e.g. i rather than (i)) and denote a set's cardinality by the corresponding lower case letter (e.g. |S))
For any agent i. Shapleyys value is then
s! (n 1)! n!
v. (v)
Σ
(SU)) - (S)]
SCNVO
This is the average worth that i adds to possible coalitions S, weighted follows: if the set of agents, S. has already 'arrived', draw the next agent, i, to arrive uniformly over the remaining N agents.
Shapley|s ss value is widely used in explainable Al's attribution problem, partitioning model predictions among individual features (q.v. Strumbeli & Kononenko (2014); Lundberg & Lee (2177)))) after the model has been trained. Evaluating the prediction function a specific feature value corresponds to an agent's presence; evaluating it reference (or baseline) feature value corresponds to the agent's absence. A feature's s Shapley value
value is its average marginal contribution to the model's predictions.
When features are correlated, individual measures of importance may mislead (Bhatt et al. 2020
Patel et al. 2021 I) ) as measures of individual significance such as the f-test do. Thus, Shapley S value has been extended to sets of features (Grabisch & Roubens 1999a; [Marichll et al. 2007: Alshebli et al. 2019:] Dhamdhere et al. (020)) As these extensions introduce axioms not present in Shapley they do not preserve the (Shapley value's intuition.

---

Published as a conference paper at ICLR 2022
We extend [Shapley|s is axioms to sets of features, randomizing over sets rather than individual features. The resulting joint Shapley value thus directly extends Shapley|s value to measure of sets of feature importance: the average marginal contribution of a set of features to model's predictions.
Our approach's ' novelty is seen in our extension of the null axiom: in Shapley (153), a null agent contributes nothing any set of agents which it may belong: here, null set contributes nothing to any set to which may belong. By contrast, interaction indices (Grabisch & Roubens 1999a Alshebli al. 2019 Dhamdhere al. 2020) recursively decompose
into individual elements, retaining the original Shapley null axiom. Thus, these indices measure sets' contributions relative to their constituent elements so are complementary to the joint Shapley value.
The generalised Shapley value (Marichal et al. 2007) is closer to our work, but differs in a number of respects: our probabilities are independent of the size of the set of features under considcration" our efficiency axiom is fully joint, while theirs are based on singletons and pairs; our symmetry axiom provides uniqueness without reliance on recursion or partnership axioms.
To illustrate, consider a null coalition, T, whose individual members contribute positively to coalitions that they join. Interaction indices assign negative value to T, as the individual members act discordantly together. However, from joint feature importance viewpoint, 7 should be assigned value O. The joint Shapley value matches this intuition.
In the movie review application presented below, the joint Shapley values reveal contributions of collections of words, including grammatical features such as negation ( (!disappointed) versus ( (won't, disappointed)), and adjectives (leffort) versus (terrific, )ffort)).
Like the Shapley-Taylor interaction index (Dhamdhere et al. 2020), our efficiency axiom depends on positive integer k, the order of explanation, which limits the number of joint Shapley values to those for subsets up to cardinality k. As there are 2 - non-empty subsets of N, the full set of joint Shapley values rapidly becomes unmanageable otherwise. In practice, i should be set to trade off insight (favouring higher k) against computational cost (favouring lower k))
For each k, the extended axioms yield unique joint Shapley value. Unlike in Shapley (953), the joint anonymity and symmetry axioms are not interchangeable: each imposes distinct restrictions.
Section presents and extends the original Shapllayaaaaaaaa Shapley axioms. Section Jintroduces joint Shapley values, deriving them as the unique solution to the extended axioms. Section 4 illustrates joint Shapley values in the game theoretical environment and applies them the Boston housing (Harrison & Rubinfeld 1978) and Rotten Tomatoes movie review (Pang & L 2005) datasets, comparing them to interaction indices and presenting a sampling technique to facilitate calculation. Section concludes. The Appendix collects proofs and other supplemental material.
2 EXTENDING SHAPLEY'S AXIOMS
For a game v E G.N, and a permutation on N, denote a permuted game by ou E GN such that (s)) v(S), V C N, where a(S) = {a(i) i S). An index Φ(v) of the game v E GN is any real-valued function on 2
The original axioms uniquely satisfied by the (standard) Shapley value th are:
LI linearity: & is a linear function on g.V. i.e. v0 + +w) (vv) + (ww and ((av) av)(() for any w E G and E IR.
NU null: agent that adds no worth to any coalition has no value, i.e. if o(S U (i)) v(S) for all S C N \ {i}, then (v) = 0. This axiom is sometimes called dunmny.
EF efficiency:
The sum of the values of all agents is equal to the worth of the entire set, i.e. for all u GN. Ei= (v())) vNN..
AN anonymity: For any a on N and any u G., (r(()) = vora (ov), for all i N.
Measures of joint significance (e.g. F-tests) and model selection (e.g. BIC or AIC) penalize larger feature avoid overfitting. As joint Shapley values are calculated after model training, this prohlem does not arise.
2

---

Published as a conference paper at ICLR 2022
SY symmettry: If two agents add equal worth to all coalitions that they can both join then they receive equal value: if (S (i}) = (J)ss S {i.z} then Va(o) = (t). This
is strictly weaker than anonymity.
Now extend each of these axioms in natural ways to conditions on sets rather than singletons. Below, (v) denotes index for coalition S on game v.
JLI joint linearity: & is linear function on G i.c. o(v + w)) = (v)) + d(w) and o(av) = aι(v) for any v, G and a E R. (This axiom has not been modified.)
JNU joint null: A coalition that adds no worth to any coalition has value, i.c. if (S =(S) for all _ \ T, then dr(v) = O.
JEF joint efficiency: The sum of the values of all coalitions up to cardinality k equal the worth of the entire set, i.e. for all E GN
Φr(v) = v(N).
TCN: TISK
JAN joint anonymity: For any on N and any u E CGN. (T((") = (o(T) (ov), for all T S N.
JSY joint symmetry: If two coalitions perform equally when joining coalitions that they both join and for other coalitions they add no worth then they receive an equal value, i.e. if
UT) = vIS U T") for all S N N (TT),,
(TT = v(S) for all S C N \ T such that S 20,
3. uS) = v(S) for all S C N \ T' such that SnT Ø,
then (v) (r)
Axiom JSY only equates the joint Shapley values for coalitions T and 7" if they contribute identically to coalitions that they may both join, and contribute nothing to the other coalitions. Axioms JLI JEF and JAN are all also used in Dhamdhere et al. (2020). Our joint null and joint symmetry notions appear to be new: they reflect our interest in a set of features' contribution to a model's 's predictions, so that the set's cardinality should not play role in determining its value.
JOINT SHAPLEY VALUES
Our main result is that there is unique solution to axioms [L]
INU
JEF
JAN
and JSY the joint
Shapley value. The uniqueness up to the 1hth order of explanation; we say nothing about [T] > k.
Theorem 1. For each order of explanation & f1...... n). there is unique (up the hthh order of explanation) index &' which satisfies axioms JLI
JNU
JEF
JAN
and JSY It has the form
(1(1)= = Σ (s( u(S)
SCNYT
for each Ø + N with TT < k, where (40
4 uniquely solves the recursive system
>==(r=kjvo AjvO (2)
90
Σ iill
4r
Σ kA(n-r) ("-")
for all r E El....,
When &- = 1, the joint Shapley values coincide with Shapley's values When & = n, we have
(-2)a-3 2 Vr (0...... 10.....7. 1}.
gr Σ j=0
For each kt the constants 4, are non-negative and satisfy Σe-1 (C) = 1. Further, as the joint Shapley value is similar in form equation I 's standard Shapley value, the value of coalition T
The Shapley-Tayle interaction index, ST also has this property Dhamdhere
2020
3

---

Published as a conference paper at ICLR 2022
depends on its marginal contribution to other coalitions; unlike interaction indices, it does not depend on the worth of its constituent agents.
As with the Shapley value, the joint Shapley value can be seen the worth brought by 'arriving' agents but, rather than arriving one at a time, they can now also arrive coalitions. We develop this interpretation in Appendix
We show here the implication of each of the joint axioms introduced above and prove Theorem [
It is already known that joint linearity restricts measures to be linear combinations of worths: Lemma 1 (Grabisch & Roubens (1999)), Proposition 1). If & satisfies JL then every @ #TCN there exists a family of real constants taftseN such that for every GN,,
ΦT(()= Σ ds
SCN
Axiom JNU then constrains the values of the constants {a²}:
a be the constants from Lemma ] Then for every @ + T N and Ø # S N T. ag = -asur. Further, for every Ø HT C N, C.SSNNAZ T, and ø + H S T, HUS = O.
Lemma Suppose & satisfies JLL and JNU and let sCN,, Ø#TCN
SCombining these two lemmas yields: Proposition 1. Suppose é satisfies IL and JNU Then there exist constants (p² that depend on and S such that for every Ø + S N and gN
ΦT(U)= = Σ -((S)((SUI) - v(S)
(3)
SCMVT
Now establish a condition on the (S)} values which must be satisfied under JEF
Proposition 2. For each order of explanation k. & satisfies axioms JL JNU and JEF if and only if for every Ø T S N with [T] < k and v E EN,, (r(()) Escrvr (S) UT) - v(S) with {xT (S)} satisfying
8N(S)=
Σ (S
E
DF(S).
ØYTCS: ITIS
DTTCNYS:
ITISK
for all + USC. C N, where 8 equals I if N and otherwise.
Recall that symmetry axiom SY is strictly weaker than anonymity axiom AN
MMalawski 2020).
This is not the case for their joint counterparts: each imposes a different constraint on the constants (S)) as we shall
here. First, consider the effect of imposing JAN
Proposition 3. For each order of explanation k á satisfies axioms for every every
JLI
JNU
JEF
and JAN
if and only
Ø + C N and E GN,, (T( = =SSENTT' (8)
0 UT)
(S)) with {o" (S)}
satisfying 44 and
pT ( CMM
C,SCNN, S' CNN 8
(5)
The analogous result for JSY
Proposition 4. For each order of explanation k, d satisfies axioms IL every every satisfying (4 and
JNU
JEF
and
JSY
if and only
# C and E GN (T( ESENNT (S) ((UUT)
v ()) with {or (S)
p" ((S)= (NN
(6)
Combining Propositions
2 completes the proof of Theorem
The computational complexity of deriving the (go...., 4--1) is O(kk2). Once these have been determined, the joint Shapley values can be calculated; the complexity of doing is O A n2",
4

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
4.1 GAME THEORETICAL.
We present two game theoretical models from Dhamdhere al. (2020) with known 'ground truths'. For each, we compare the joint Shapley value, the Shapley interaction index (Grabisch & Roubens 1999b), the generalised Shapley value (Marichal et al. 2007). the added-vales index (Alshebli et 2019 and the Haapley-Taylor interaction index (Dhamdhere et al. 2020). respectively:
n ( 8
pT (v)
Σ SCNVP
Σ LCT
(-1) uSU VI S
n1
(n 8 - 1)1s!
@8 (0) = Σ
G8
(v (SUT) - u (S)];
(n t + 1}!
SCN(T
c! (8-c--1) 1)!
oT (v) = v (T)
Σ
Σ
- -
2n1
s!
iET
SENiES Ccsli
(() EsgNT -1
< u ( (SUL)
Se (v; =
ifl= k.
Table 1: Joint and interaction measures in n = game theory examples
Shapley (k: = 1)
pCs
AV
9T
(Umm k.)
8' (Umih) k)
T
2
1:
0=
==
The majority game: Um (T)
when t 2, and is equal 0 otherwise
0
1/3
1/3
1/3
1/3
1/9
2/21
1,5
1/2
1/3
1/3
2/9
1/21
N
0
2
3/21
A linear model with crosses: (T")
Σ
Ligr + cmax {0, 2} for E R
1
1/a +c)
(3 c)
1/+3++))
1/12
1
5/18 (2 +c)
8/21 (2 + c)
1.
1/sc 142(4+0,
1/sc
1/sc
1/18 (8 +c)
1/21 (8 + c)
N
4+0
3+0
3/1c
3/21 (3 + c)
In the n = 3 majority game, Table Table[]show; shows that, while Shapley-Taye assigns 0 to singletons, the joint Shapley value recognises that singletons contribute positively when joining existing coalitions. The Shapley-Taylor and Shapley Interaction indices assign negative values to N (for discordant interaction between members); the joint Shapley rewards N for adding worth in its own right.
In the linear model with crosses, joint Shapley values' signs can again differ from those of interaction indices: let e = 22 which sets of = 0: if
joins the empty coalition or either singleton, it adds unit worth; however, when i = joins {2, 3). it subtracts unit worth. Averaging (according to the arrival order), gives a net contribution of O: coalition does not contribute any worth in expectation. By contrast, the Shapley-Taylee value always (for k > 1) assigns value to singletons equal to their worth, as it not (by design) capturing information about expected contributions of features.
4.2 THE AI/ML ATTRIBUTION PROBLEM
Following Strumbelj & Kononenko 2010), let be prediction function, and x (z1, ......... In) E A an instance from the feature space. For a set of features S S N. define the prediction difference Pr (5) when only features in S are known, as
Te (S) = LA
Σ 1f(T(I,2,S)) f(=)),
AEA
where Txx,2,S) the difference between the expected prediction when only the feature values of in S are known,
2. E A is defined as T(x, x, S); equal to x; for E S and e, otherwise. Thus U (S)
5

---

Published as a conference paper at ICLR 2022
and the expected prediction when feature values are known. Then for each T S N, (f(()) is the contribution of features 7' to the prediction [(x). To estimate (fx), let X be random variable whose law coincides with the law the number of agents already present when T arrives in the
arrival interpretation (so that the law of X depends on the size of T):
P(X i)
la
for each E {0; -0.....71) n and S be random variable uniform the set of subsets of N \ of size X. Once X is sampled, we can generate the set of agents already arrived, called S, by choosing it uniformly from all subsets of N of size precisely X. Then
(vx) = Σ j=0
n -
q A - Use (S)].
Thus, by the law of large numbers, we can estimate this expectation by taking an average of (;(T' U S.) (S,) where S, has the same distribution as S and (s()) f[r((, 2,, S)) - f(z)), where 2; is chosen uniformly from the feature space. Refer to (f(2)) as the local joint Shapley We can combine local values
obtain, for each feature, global joint Shapley value. We do so in two ways. The first is the standard methodology (q.v. Lundberg & Lee (2077)), which averages the absolute values of locals. We introduce the second for models with exclusively binary variables by considering presence/absence of features T in æ. This presence-adjuste global joint Shapley value is defined
value of features T at instance
1
H
E ΦEA
1x(T)== 1))x(()))
LA
where ==(T)= one if all features T are present in æ; and O otherwise. Covert et al. (2020) introduced SAGE (Shapley Additive Global Importance), more sophisticated treatment of global influence measures that maintains efficiency at the global level.
All experiments on a single Intel(R) Core(TM) i7-6820HQ CPU. Training details and tuning parameters are provided in the accompanying code.
4.2.1 SIMULATED DATA
Consider three features, F1,42 and 33, each uniformly drawn from (0, 1) and a dataset of 50 observations. We first consider independently drawn features; then investigate correlation by fixing 22 = Fi. We use simplified 'ML models', (x), to obtain exact global joint Shapley values, averaging the absolute values of local joint Shapley values derived from equation 77. Table displays the results.
Table 2: Uniform random variables; 1e
311
.22
33
T1, F2
21,
22,
JT, 32,
independent features
f1 (z) 21
0.122
0.049
0.049
0.037
fa (w) 1
+ 22
0.122
0.114
0
0.061
0.049
0.045
0.046
fa (x: = 21 x2
0.122
0.114
0
0.062
0.049
0.045
0.047
correlated features: W2
(z)
0.122
0.122
0
0
0.049
0.049
fa (a:)
0.122
0.122
0
0.098
0.049
0.049
0.073
Independent variables As only influences fi, only coalitions including it receive value; the more diluted its role, the lower the value; 2y and Øy are symmetrically irrelevant. For fz. I1 and Iz play equal roles, and receive equal values (up variance due to the sample size). For f2 and fs. {21, x2 receives similar value, and more than in fi, where only 2; influenced model predictions. Comparing f2 and fa shows that values only differ for coalitions containing both 21 and 22.

---

Published as a conference paper at ICLR 2022
Correlated features f2 now exhibits what term cancellation effect between 21 and zz- assigning a value of zero to coalitions containing both. Similarly, fa reveals an enhancement effect: values assigned to coalitions with both 2 and 22 larger than in the independent case.
Enhancement and cancellation effeets do not uniquely identify underlying phenomena. To illustrate, let 2 22 and 23 be independent Bernoulli(0.5) random variables. Setting i = 3 and letting oc, we obtain the exact presence-adjusted global joint Shapleys in Table
Table 3: Bernouli((O5)) random variables; k = 3
f (œ)
F1,
I1, 23
22,22
21,22,43 43
f1 (z) x1
0/21
0
0
1/21
1/21
0
1/56
fa (z) = +32
6/21
5/21
0
2/21
1/21
1/21
1/28
(x) 2132
6/42
5/42
0
1/14
1/42
1/42
3/112
The joint Shapley values containing æ1 and 22 are larger for models f2 and fa than for model fi, different sort of enhancement effect from that above. However, the joint Shapley value for 21 is smaller in fa than it is in f1 or f2, again different type cancellation effect.
Finally, joint Shapley values can provide insight into the black-box model's structure. If, for example, f can be decomposed as fx) = g (21) + h (zz..... z..) with independent Bernoulli0..5) random variables, then the presence-adjusted global joint Shapley value of æ1 is 1/2 (g (1) g (0)) Σn= ("=") = 5/21, as shown in the table. If the joint Shapley value deviates from this, we reject the decomposition, as in the case of f4.
4.2.2 BOSTON HOUSING DATA
For comparability, we follow Dhamdhere et al. (2020) by training a random forest on the Boston hous- ing dataset (Harrison & Rubinfeld] 1978), computing global values using the first notion discussed above. presents the largest and smallest global joint Shapley and Shapley-Tayy Taylor values.
Table 4: Joint Shapley values for the Boston dataset
Shapley (k 1)
G (f:A) 3
d (f:R) k
k: ==2
k
RM: 2.57
RM: 3.12
RM: 3.12
LSTAT: 0.63
LSTAT: 0.42
LSTAT: 2.47
LSTAT: 2.04
LSTAT: 2.04
RM: 0.56
RM: 0.34
AGE: 0.81
DIS: 1.55
DIS: 1.55
LSTAT, RM: 0.30
AGE: 0.11
DIS: 0.63
CRIM: 1.37
CRIM: 1.37
AGE, LSTAT: 0,21
LSTAT, RM: 0.11
CRIM:
DIS, LSTAT:
AGE, RM: 0.20
NOX: 0.10
NOX: 0.44
NOX: 1.15
B: 1.31
DIS, RM: 0.19
DIS: 0.08
PTRATIO: 0.27
B: 0.24
CHAS. RM: 0.15
AGE, CRIM. RM: 0.21
CHAS, TAX: 0.02
RAD, TAX, ZN: 0.00
TAX: 0:22
LSTAT, TAX: 0.15
AGE, DIS. PTRATIO: 0.21
RAD, ZN: 0.01
CHAS: 0.00
INDUS: 0.19
INDUS, RAD: 0.15
DIS. LSTAT, PTRATIO: 0.21
CHAS, RAD: 0.01
CHAS, RAD. TAX: 0.00
RAD: 0.13
NOX, TAX: 0.14
AGE, LSTAT, PTRATIO: 0.20
ZN: 0.01
0.01
CHAS, RAD, ZN: 0.00
0.07
DIS. INDUS: 0.13
AGE, NOX, RM: 0.20
CHAS: 0.01
CHAS, TAX, ZN: 0.00
CHAS: 0.07
DIS, PTRATIO: 0.12
B, CRIM, LSTAT: 0.19
CHAS, ZN: 0.01
CHAS, ZN: 0.00
For k = 1, the joint Shapley values are the classical Shapley values. The unimportance of CHAS seems to reflect its low variance (only of 506 units back onto the Charles).
For l: = 2, LSTAT and RM still have the largest joint Shapley values, with (LSTAT, RM) having third largest. The top pairs all involve LSTAT or RM, indicating that these variables also contribute to explaining house prices jointly with other variables. By contrast, (LSTAT, RM) is the 16th largest Shapley- Taylor interaction value. The distributed throughout the pairs; by contrast, the singleton & = 2 Shapley- Taylor values outrank all the pairs. This is consistent with our extension of axiom axiom|NU]to not favour singletons. The largest joint Shapley value involving NOX is (NOX, LSTAT), which is about five times large as (NOX, DIS)
singleton joint Shapley values are fairly evenly
7

---

Published as a conference paper at ICLR 2022
and (NOX, RAD). This is consistent with Harrison and Rubinfeld's observation that NOX offset RAD and DIS, but reinforced LSTAT: knowing the values of NOX and LSTAT adds a lot of predictive power; knowing the values of NOX and RAD DIS tends to wash out additional predictive power.
For = 3, singletons again dominate: the largest triple is {AGE, LSTAT, RM). the three individually most important features. Continuing the above analysis, the joint Shapley value of (DIS, RAD) is about 0.015, while that of (DIS, NOX, RAD) is about 0.008, and that of (DIS, LSTAT, RAD) is about 0.022. We understand this to mean (DIS, NOX, RAD] contains the same sort of information as (DIS, RAD), but with NOX offsetting the other variables, while LSTAT brings novel socio-economic information the distance variables DIS and RAD, which continues to partially offset. Similarly. the least important joint features include the individually unimportant CHAS and its variants echoing the role played by RM and LSTAT at the top of the list. While (DIS, LSTAT) has a Shapley- Taylor value of 1.33 (indicating a large interaction between the pair but not its overall importance), its joint Shapley is 0.06 (ranked 18th) directly assigning a value of importance.
Table 4s values are exact. Figure I demonstrates the estimation procedure above, showing con- vergence of the sampled values as the iterations increase. As with other sampling procedures (e.g. Strumbelj & Kononenko (2014), (2114)), this cuts the complexity to order n to a linear power k; as k + n, the complexity approaches 2 times polynomial in (compared to 3 for the exact (b)).
000022
000155
122
144
000000
150
300
250
330
Sutber of Lumain
Figure 2: Difference between consecutive /-' samples averages converges to zero; k = 2 movie review #2
Figure| I: Sampled d' converges to exact d with 2 norm & 1.45; Boston
4.2.3 MOVIE REVIEWS
We train a fully connected neural network (two hidden layers, 16 units per layer, ReLU activations) on the binary movie review classifications in Pang & Lee (2005). From the full set of reviews, we remove a test block of 100 (picked to include those in Table of Dhamdhere et al. (2020) for analysis. We encode reviews as the 1000 most common words in the corpus, augmented by key words in Table of Dhamdhere et al. (2020) to aid comparison of measures, for a total of 1004. (The DSA features are drawn from positive reviews: {won'th von't). (disappointed), (both), (inspiring), (a), (crisp), (excellent), (youthful), (John), (terrific).) Binary accuracy is typically c. 76% after four epochs.
Table 5 local joint Shapley values are tiny each of the 2000 joint features has tiny effect on the probability a positive review. In the vein of Dhamdhere et al. we identify intuitive effects:
1. negation: the negative local joint Shapley values for ( (disappointed) and [be, disappointed] become positive when negated by adding [won't] to the coalition.
2. enhancement: the positive local joint Shapley values for (both) and (and) are enhanced by the positive (and, both).
3. context: the sign of local joint Shapley values involving (well) depend on the context; while [you, well) is positive, [left, well) is negative.
4. lost potential: the positive local joint Shapley values involving (fascinating) become negative when (been) is added.
5. adjectival: while ( (director) and (effort) are individually negative, they become positive when qualified with the adjectives (winning) and (terrific), respectively.
8

---

Published as a conference paper at ICLR 2022
Table 5: Examples of local joint Shapley values in the Pang & Lee 2005 movie reviews
Review
joint Shapleys
(disappointed -2 10-5
[won't]: x 10
I: negation: aficionados of the whodumit won't be disappointed
{be, disappointed]: -9 10-8
(won't, disappointed]: 6 10-5
(won't, be, disappoinied]: 5 10-9
[both]: 2 x 10-4
2: enhancement: both inspiring and pure joy
{and): 6 10
{and. both): 10-6
{you, well): x 10-7
context: you wish Jacquot had left well enough alone
{left, well]: -3 10-7
[would): - 10-4
(fascinating] J: 2 x 10"4
4: lost potential: fascinating little thriller that would have been perfect
[would, fascinating):
10-7
(would, heen, fascinating]:
10
(effort): -1 x 10-5
[director]: ): -9 x 10-6
adjectival: director ... award ... make terrific effort
(terrific, effort): 8 10-7
(winning, director): 5 x 10
For global values, as the features are binary, we compute the presence-adjusted global joint Shapleys, 31. Comparing the DSA features' global 0² across the test reviews for k = and k shows [terrific] to be the largest, followed by [both] then (excellent). This reflects prevalence in the training set, where (terrific) appears four times as often in positive reviews. Similarly, on the negative @ side, [John} is followed by [disappointed): (John) appears thrice as often in negative reviews.
The largest pairs are (both, inspiring) followed by fexcellent, youthful). All of the pairs, except (John, terrific), have positive sign: these pairs typically do not appear outside the positive reviews from which they were drawn. As our encoding discards sequential information, the pairs are smaller than the singletons: coalition merely indicates co-occurrence in review, rather than bigram.
As the 10 DSA features are a very small subset of all 1004, we cannot asses whether JEF holds. However, health check note: the average presence-adjusted local joint Shapleys over the positive reviews about 20% larger than that over the negative reviews. They are both negative: as these features are drawn from positive reviews, but largely absent in any given review, their effect is negative. Further, Table 6 indicates that the ranking of single features is largely preserved as increases from 2: the exception is (a, crisp]: the corpus, (crisp) typically preceded by (a).
Figure shows the estimated values' convergence.
Table 6: Largest presence-adjusted global joint Shapley values on DSA features
terrific
both excellent won't crisp
youthful youthful
inspiring crisp
disappointed disappointed
John John
terrific
both
excellent
won't
inspiring
CONCLUSIONS
The joint Shapley value directly extends Shapley|s value to measure the effect a set of features on a model's predictions. Further work is needed to maintain properties like global efficiency (Covert et al.) 2020) and to make sampling more efficient (Williamson & Feng 2020 Mitchell et al. (021). We believe that understanding complex models is labor intensive. Nevertheless,
envisage common workflow that computes = Shapley values, followed by = to identify strong pairwise effects; analyses for l > then respond to the analyst's evolving questions about the model's functioning.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This paper contributes to the literature on explainable AI, an important component of the Fairness, Accountability and Transparency research agenda for ethical AL. It does not use human subjects; only draws publicly available datasets; the work has not been sponsored, and does not seek to promote any third organisations; none the authors face any conflit of interest.
7 REPRODUCIBILITY STATEMENT
Our proofs and source code are available in the accompanying supplemental material; all data are taken from the public domain.
REFERENCES
Bedoor K Alshebli, Tomasz P Michalak, Oskar Skibski, Michael Wooldridge, and Talal Rahwan. A measure of added value in groups. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 13(4):1-46, 2019.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. Explainable machine learning in deployment. Proceedings the the 2020 Conference on Fairness, Accountability, and Transparency,
pp. 648 1188-67,222 2020.
lan Covert, Scott Lundberg. and Su-In Lee. Understanding global feature contributions with additive importance measures. arXiv preprint arXiv:2004. 00666,, 2020.
Kedar Dhamdhere, Ashish Agarwal, and Mukund Sundararajan. The Shapley Taylor interaction index. In International Conference on Machine Learning, pp. 9259-92666 PMLR, 2020.
Michel Grabisch and Mare Roubens. An axiomatic approach to the concept interaction among players in cooperative games. International Journal of Game Theory, 28(4):547-565, 1999a.
Michel Grabisch and Mare Roubens. Probabilistic interactions among players cooperative game. In Beliefs, Interactions and Preferences in Decision Making, pp. 205-216. Springer, 1999b.
David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Managemeni, 5(1):81-102, 1978.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Marcin Malawski. A note on equal treatment and symmetry of values. Transactions on Computa- lional Collective Intelligence XXXV, pp. 76-84. Springer, 2020.
Jean-Luc Marichal, Ivan Kojadinovic, and Katsushige Fujimoto. Axiomatic characterizations of generalized values. Discrete Applied Mathematics, 155(1):26-43, 2007.
Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for Shapley value estimation. arXiv preprint arXiv:2104.12199, 2021.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL, 2005.
Neel Patel, Martin Strobel, and Yair Zick. High dimensional model explanations: an axiomatic approach. In Proceedings of the 2021 ACM Conference Fairnsss, Accountability, and Trans- parency. pp. 401-411, 2021.
Lloyd S. Shapley. A value for n-person games. In Harold William Kuhn and Albert William Tucker (eds.), Contributions to the theory of games, volume II of Annals of Mathematical Studies, chapter 17, pp. 307-317. Princeton University Press, Princeton, 1953.
10

---

Published as a conference paper at ICLR 2022
Brian Williamson and Jean Feng. Efficient nonparametric statistical inference population feature importance using Shapley values. In International Conference on Machine Learning, pp. 10282- 10291. PMLR, 2020.
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18, 2010.
Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41(3):647-60 644-666,,, 2014.
In this supplementary material we discuss the arrival order interpretation, present the proofs of all results, and discuss notion of joint symmetry obtained by removing conditions 2 and 3 from JSY
A ARRIVAL ORDER INTERPRETATION
As discussed in the paper, the joint Shapley value can be viewed in terms of the worth brought by "arriving' agents but, rather than arriving one at time, they can now also arrive in coalitions. To be precise, consider this procedure: at time O, no agents have arrived; at each next set of agents to arrive is chosen uniformly from the set of non-empty subsets of size at most of the remaining (yet arrive) agents. Then of is the expected worth brought by coalition T when it arrives (a coalition is assigned zero worth if it does not arrive at any time). To see this, denote by A, the coalition to arrive at time i, by B₁ the union of all coalitions that have arrived up to time i: B, Uزs Aj. and by pr the probability that some time coalition T arrives, pr P(3i: B; = T). We have the recursive relationship:
f1.2....} the
Pr
P(B:11 S)P(A) = T\8 B S)
SCT: |
(n-1S))A
TI-1
(n (n-s)Ak
Σ
Σ
P(B-1==S)
Σ
n
|S|
Σ
''ss
1
Ps
Σ
8
SCT: -
where the last line any set size s. Thus we see that pr only depends on T through its cardinality, and defining
P: = Pr
Σ
for any with [T the expected worth brought by coalition T under this procedure is
Σ Σ
A A v(S)]
Σ P(s(o(SSU)) v(8)).
1= SCN
SCNYT
Further, we have the relationship
Σ (x= Cpps
P.
(m-O)A (n-9)
and since we know that Po = (we start with no agents having arrived) and Pa. (we finish with all agents having arrived) also have the identities:
n-
Ps:
1
Po
S=N-
By comparing with the identities defining (gn...., 9n 1). we deduce that p.. = q: for all t e
(0. 10....,7.. 1}, which verifies this arrival interpretation.

---

Published as a conference paper at ICLR 2022
B PROOFS
Proof of Lemma
For the first statement, for each 2 + $ N \ T consider the game
us(R) = [1 = S or R SUT,
otherwise.
Then for such S, by JNU Φr (us) = 0. By this equality, Lemma and the definition of us.
0 = Φr(Us)= =
Σ
( = = + asu--
RCN
For the second statement, for each the game
+ Ç let or be constant and for S S N \ T, consider
on 0
if R SUR H for some Ø H ç T.
x zg
otherwise.
By JNU () = 0 for every S S N
T. Thus by Lemma
0 = (2()5))
Σ f
Σ
T
2 = Σ
T auu
R
Ø+HST
2#HºT
Since this holds for every choice of constants an, it follows that aSuh 0 for all S S N 7 and
* H & T, as required.
Proof of Proposition 1 By Lemma ! there exist constants {a5)scN ##TCN 24NNN such that for every v and Ø N.
ΦT(U)= Σ asu(S)= Σ a5o(S)+ Σ
oSSHu(SUU)) +
T
SCN
SCNVT
T = Σ (a5v(S) + asuro(SUTT) = Σ
((UT) - x(5)];
SCNAT
SCNYT
where the last two equalities owe to Lemma
2 The proof is complete by setting (S) abur-
Proof of Proposition Suppose o satisfies axioms JLI
JNU
and JEF
Then by Proposition the
constants {p" (S)} exist, such that for every v and Ø + C
(T(U)=
Σ (T(S)((SSUT) (S)) SCNVT
Now for each Ø + R C consider the identity game
S = R {
"
otherwise.
Then for every + T N with [T] <
(r(wn)
Σ (S)) SCNVT
Note that the term wn(S U UT) wn(S) in the above sum is equal only when S R and = R \ S. i.e. only when 5 R T and Ø + T S R. Further note that this term is equal to -1 only when R and T' o i.e. when S = R and Ø of T C N R (as must have SnT Ø). In all other cases, this term is O. Hence we deduce from JEF that
n R)) wR(N)-
Σ (T
Σ
(RR).
BATER: ITiSK
BETENAR TiSk
12

---

Published as a conference paper at ICLR 2022
Now we show the implication in the other direction. If (v) Σ Escry p" (()))((S)))) c(S) then it is immediate that JLI
and JNU are satisfied. FEr wish to show that for every v,
are
Σ
Σ (((S)(((SUT)))) - x(S) = v(N).
SENN
PiSk
Note that for each Ø + R S N, the coefficient of (R) on the left-hand side in the above equation is
Σ p (RA)
E (R))..
TCR
CNTCNAR:
ITiSK
ITiS
But by equation 4 this is equal to 8x (R).
Proof of Proposition In light of Proposition]
2 we just have to consider JAN
Only if: Suppose o satisfies JLL JNU JEF and JAN First, we shall establish that
(S)
S, EN s-s.
Fix such T, S and S². Consider again the identity game, ws and let a be self- permutation
such that + S S + S. and ({i}) = {i} for all & SUS. A S N \ SS"S') and (T) T, we have by JAN
Φr (ws) Φ0-1(T) (ws) or (ows)
where
ows (R) = ws (a- (R)) =f 1 if R (S) S'
}= Wes (R).
0 otherwise
Hence we obtain @r(ws) = rer (ows)) = +r (wss) ). Next, from Proposition | we have
Φr (ws) = Σ pf (Q) [ws (QU ws (Q)) = -p"(())
QENYT
and similarly Φr (ws²) -p" (S'). Hence we obtain p" (S)= =p" (S"), showing 9
Using induction on now establish that (5) holds. Fix T' and T" of the same size. For the base case, suppose = = n - t. For S C N VT T and S' C N \ T', this forces S = N T and S' N T'. Now consider the game
i = otherwise,
In (R)
so that x,, (R) if and only if R N. Define self-inverse permutation that (T) =T', a (T') = T and (ft))= = (i) for all & (TU Then by JAN and as Or, = Zn.
(T())) = ( (.n) = Φr (ax,)) = (xn).
Next, from Proposition 1
Φr (Ix) = Σ p (Q) [ (TU (N)T),,
QENVT
and similarly ΦT, (wn) TT (N T'). Hence we obtain of (N D" (N (N)T)) which establishes the base case.
We now suppose that p" (S) = p" (S') for s = c where S N \ S C (T" and e a positive integer. We shall show that pT (S) = pT' (S') for all s s 2 where where
S C N W and S' NT'. To this end, consider the game
x (R) = {1
if 2 n 1+
otherwise
13

---

Published as a conference paper at ICLR 2022
Thus, before, may write
0r (2) = E
p" (Q) (QUT)-
-
Σ
(Q).
QENTT
QCNT
1-0-1595h-c-14
Similarly,
Φr (x)
Σ
p" (Q).
Q'CNNT" 159 Cn-c-1+t
Again, by JAN we prove that or (x) ΦT (x). Define a self-inverse permutation a so that (T) =
T", (T") T and a ({i) { for all i & (TUT). As worth in game depends only on coalition's 's cardinality, we have or = z. Thus, by JAN Φr (a) = Po(T) (ax) = (r(T) (2) = (x).
However,
Ør (a)
Σ (Q)+
E
E
(Q)+
E
pf (Q))
QENTT v=n-c-1
QENiT
QENIT
Q'ENNT'
R 1-9
q=n-c-1
a n-c- <n-c-1+
and
OT (2) E
D (Q)++
E
DT (Q)
Q'ENTT' =n=-o-1
Q'ENIT'
R-c-1<d en-o-1+e
which gives, by the inductive hypothesis and ΦT(I) = ΦT
Σ p (Q)= = Σ p- (Q')
QCNTT g
D'CNYT' gnnnce1
But by 99. ()) (Q)
(")=
= = (Q)) if = q' Thus the above equation becomes
(9) (n-=_))
n n-t
- (0000(")))000' pr (Q))
for any Q S T and Q' C NNCCN N with = of c-c 1. Thus, p" (Q)=p²' (Q), completing the inductive step, and the 'only if' statement.
If: Suppose (5) is satisfied. Fix permutation N and game E G. Then for any Ø + T N,
or (ov) = Σ D" (O) - ou (S)]= Σ (S
SCNVT
SCNVT
Σ ( SCNVT
- (
Defining the set S' 0 (S) allows us rewrite the above
or (ov) = =
Σ
of (a (a) [ (S' (S - (S')
S'CN
1(T)
Σ
a (S) - v(S)] ()-1
S'C
with the penultimate step due condition
Proof of Proposition 4 In light of Proposition
2 we just have to consider JSY
Only if: Suppose satisfies JLI JNU JEF and JSY fix + T,T" C N, and consider again the
identity game wR. Then for any Ø MARCNNTUU
w U T) wn(S UT') for all SC (NUUT)
14

---

Published as a conference paper at ICLR 2022
"x((SU) = 0 wx(S) for all S S N \ such that S * 2,
"u(SSU)) = 0 (S( for all S _ N T" such that S 5n T Ø.
Hence by JSY Φr (wR) = or (wR). But or = (R) and ør (WR) = = (R(R.. This shows
that p² (R) = p" (R) for all Ø o
To show that (²))) = (0) we consider
the game
{1 2 w" w²(s) to otherwise.
Then
w'US = 1- w*(ST") for all S S "NNN(TUT)
""(SUT) (8) for all S S N NXT such that S S'T" of Ø (since then S / Ø),
"=(SUUT) = w (S) for all S C T" such that S Ø (since then S * Ø))
thus follows by JSY that Φr(w") = (w*)))). However, Φr(w*) = p" (Ø) and Φr (w")
p" (0), which gives the required identity and shows that (6) holds.
If: Now show the implication in the other direction. Suppose (6) holds and G satisfies the three conditions in USY Then
Φr(v)=
Σ (S())((STT)
E (S() v(S)]
SCNYT
SENV(TUT))
Σ (CM)(TUM)
p" (S))(SUUT))
Σ (S(()((SS)T) - u(S)] prr(v)
SCNYT'
Hence ISY is satisfied.
Proof of Theorem 7 We have to show that there exists exactly choice of constants (S))
which satisfy equations (4). Notice that satisfying (5) and (6) is equivalent to satisfying
p - N CNATT s.t. 8=
Thus (S) does not depend on at all, and only depends on the cardinality of S. Let Is denote p" (S) for any S CN\T. Then can re-write equation 4 in terms q,
n-1 Σ
(10)
Di=(s-Kjvo k)yo (7 Σ D k) ("T")
UsE (1
(11)
Note that for any 4o, equation fully determines all other gi, for i E {1...., 1} and go is then determined by (T).. Thus there is at most one solution. However, we have already identified (see the arrival-order discussion in Appendix A that solution to this recurrence is given by
(go. .... .00. (po..... ... . for which po D (9)
C STRONG JOINT SYMMETRY
We examine the effect of removing conditions 2 and 3 from JSY As turns out, this leads to the non-existence an index. To be precise, we consider replacing axioms JAN and JSY with:
SJS strong joint symmelry: fix Ø + T,T" C N. N. Then
v = (
0T (v) (v).
15

---

Published as a conference paper at ICLR 2022
Proposition 5. There is no index & satisfying axioms exist for all games.
JLI
JNU
JEF
and
SJS
that is guaranteed to
Proof. Since @ satisfies JLI
JNU
and
JEF
by Proposition
ΦT(1)
D (S) x(S)
SCNYT
with {p" (S)} satisfying 44) for any game v E G. We consider two games vi, 02 E G(1.2), As
N
{1,2). f gives p (Ø) = p(2) (2) p( ({2)), and p (2) + (2)=
p(2) (Ø) 1.
Suppose 2n ({1)) = v1 2}} = 1, ei ((2)) = 0. SJS thus gives that (11)(11) = 01122)(01); i.e.
(2) (0) + (2) which implies (0) 1/2.
Suppose also that u2 ((1)) 0>((1, 2}) = ({2)) = SJS gives that d1 (v2) = (12) =
(12)) i.c. (2) = (2) = p22) (2) which implies pfl.2 (0) = 1/3, giving a contra- diction.
Thus,
SJS is too strong notion of symmetry, imposing linear restrictions sets of unequal sizes.
16


Reviews:
[{"review_id": "tV9OVFQxhub", "reviewer": "Reviewer_VFxY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The code looks quite good. Has walkthroughs, examples, etc. Implementation looks clean.\n\n- The theoretical results look clean, intuitive, and well-presented.\n\n- The work inlcudes some discussions on practical questions, including the employed hardware resources, and algorithm complexity estimates\n\n- The work gives an overview of a good set of other SOTA approaches for Shapley\n\n\n\n\nWeaknesses/Considerations:\n- For XAI-based approaches, it is very important to consider their practical advantages, ease-of-use, and intuitiveness. Whilst this work gives a good treatment of the mathematical properties of the proposed approach (and related approaches), these properties are not necessarily tied to practical advantages. For example, emphasis is placed on the fact that \"interaction indices are computed with discrete derivatives\" (page 1), and yet, it is non-obvious what this implies from a practical standpoint.\n\n- Furthermore, it feels as if this work is lacking a general summary of how to apply this approach for gaining insights on model behaviour. Currently, evaluation is done by running the approach on several values of \"k\", comparing the different outcomes, and tying it to properties, such as \"negation effects\". However, such an approach feels quite manually-intensive, involving significant efforts of post-hoc interpretation of the results. This will especially be the case with a larger number of features, and larger values of \"k\". Including a short description of how to apply this proposed approach in order to gain insights from a model with minimal manual effort involved would be helpful.\n\n- Whilst this work lists numerous related SOTA approaches to feature importance (and Shapley values in particular), a comparison is only made on the toy benchmark datasets. Giving concrete examples of the differences in explanations provided by these approaches on the more complex cases (e.g. Boston Housing), would provide a much clearer view of their relative strengths and weaknesses. At the very least, a justification for why such a comparison is impossible would be helpful.\n\n- There are quite a few explanation properties introduced in the Applications section, such as the \"negation effect\", or the \"cancellation effect\". If this is novel terminology introduced in this work, it would be helpful to discuss this earlier on in the methodology section (or in the appendices). If these properties are based on prior work - it would be helpful to include references. If not - giving more intuition on these properties (perhaps with more illustrative toy examples) would help.\n\n\n\nOther Remarks:\n- \"evaluating it at a reference or baseline feature\" (page 1) - I think there's a typo in that sentence\n\n- The Introduction section is essentially merged with the Related Work section, making it a little hard to follow. I suggest splitting this into two separate sections for clarity.\n\n- Is there any particular reason for why the \"Applications\" section is not called \"Experiments\"?\n\n- Table 2: not sure if so many decimal places are required. Removing some of them could make the table more compact, and improve readability.\n", "summary_of_the_paper": "This work introduces \"joint Shapley values\", which directly extend Shapley’s axioms and intuitions: joint Shapley values measure a set of features’ average effect on a model’s prediction. This work naturally extends Shapley's axioms from a single feature, to sets of features. In a nutshell: joint Shapley values measure the average marginal contribution of a set of features to a model’s predictions. This work presents rigorous mathematical results for the joint Shapley values approach. This approach is then evaluated on several datasets, including: (i) simulated data, (ii) Boston Housing data, (iii) Movie Review data.", "summary_of_the_review": "The potential impact of this work is not clear as not enough is said about practical advantages.\n\nGaining some insights into the model behaviour would make the content of the paper stronger.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ecyLypaHmoV", "reviewer": "Reviewer_QfeR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths. The paper is well written and the findings are well explained. The experimental section illustrates that the newly introduced joint Shapley values lead to reasonable results in real applications. Some examples of the joint Shapley values are provided in Table 5 to clarify the obtained results.  \n\nWeaknesses. The method is computationally heavy (but it is not easy to do it less computationally expensive). Indeed, the sampling can be done more efficiently, however, it is difficult to criticise this point much, since the authors mentioned it as their future work.\n", "summary_of_the_paper": "The paper proposes an extension of the Shapley values, namely, the joint Shapley values which measure a set of features' average effect of a models prediction. The uniqueness of the joint Shapley values is proved. Moreover, training details and tuning parameters are provided in the accompanying code.", "summary_of_the_review": "The topic is important and challenging. The results are novel, and the experimental section provides a nice illustration how the joint Shapley values can be used.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c_VrO3JjMgj", "reviewer": "Reviewer_Djge", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I agree with the theoretical significance of this work, which generalizes the Shapley value for assigning a contribution of *a set of agents*.\n\nHere are some comments:\n\n1. In practice, how can we choose k? It would be great if the authors could provide some rules. \n\n2. In the simulation, what is the explicit form of the set function? For example, in Table 2, even if there is an explicit form of a function f, it's nontrivial to derive the corresponding set function. \n\n3. Continuing from 2, I recommend adding discussion on the choice of the set function.\n\n\n\n\n\n", "summary_of_the_paper": "This paper introduces joint Shapley values, which extend Shapley values to measure contributions of *sets*.\n", "summary_of_the_review": "- I agree with the theoretical significance of this work. \n\n- A few questions/comments on the perspective of practitioners are listed above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers think the topic is important and challenging. The results are novel, and the experimental section provides a nice illustration how the joint Shapley values can be used. However, the paper can be improved by including more real world applications and experiments.
Decision: Accept (Poster)