Title: A Refutation of Shapley Values for Explainability
Venue: NeurIPS 2023
Accepted: no
Keywords: Shapley values

Explainability
Abstract: Recent work demonstrated the existence of Boolean functions for which Shapley values provide misleading information about the
relative importance of features in rule-based explanations. Such misleading information was broadly categorized into a number of
possible issues. Each of those issues relates with features being relevant or irrelevant for a prediction, and all are significant
regarding the inadequacy of Shapley values for rule-based explainability. This earlier work devised a brute-force approach to identify Boolean functions, defined on small numbers of features, and also associated instances, which displayed such inadequacy-revealing issues, and so served as evidence to the inadequacy of Shapley values for rule-based explainability. However, an outstanding question is how frequently such inadequacy-revealing issues can occur for Boolean functions with arbitrary large numbers of features. It is plain that a brute-force approach would be unlikely to provide insights on how to tackle this question. This paper answers the above question by proving that, for any number of features, there exist Boolean functions that exhibit one or more inadequacy-revealing issues, thereby contributing decisive arguments against the use of Shapley values as the theoretical underpinning of feature-attribution methods in explainability.

Abstract

Recent work demonstrated the existence of Boolean functions for which Shapley values provide misleading information about the relative importance of features in rule-based explanations. Such misleading information was broadly categorized into a number of possible issues. Each of those issues relates with features being relevant or irrelevant for a prediction, and all are significant regarding the inadequacy of Shapley values for rule-based explainability. This earlier work devised a brute-force approach to identify Boolean functions, defined on small numbers of features, and also associated instances, which displayed such inadequacy-revealing issues, and so served as evidence to the inadequacy of Shapley values for rule-based explainability. However, an outstanding question is how frequently such inadequacy-revealing issues can occur for Boolean functions with arbitrary large numbers of features. It is plain that a brute-force approach would be unlikely to provide insights on how to tackle this question. This paper answers the above question by proving that, for any number of features, there exist Boolean functions that exhibit one or more inadequacy-revealing issues, thereby contributing decisive arguments against the use of Shapley values as the theoretical underpinning of feature-attribution methods in explainability.

1 Introduction

Feature attribution is one of the most widely used approaches in machine learning (ML) explainability, 19 begin implemented with a variety of different methods [64, 56, 57]. Moreover, the use of Shapley 20 values [60] for feature attribution ranks among the most popular solutions [64, 65, 48, 17, 47], 21 offering a widely accepted theoretical justification on how to assign importance to features in machine 22 learning (ML) model predictions. Despite the success of using Shapley values for explainability, 23 it is also the case that their exact computation is in general intractable [8, 21, 22], with tractability 24 results for some families of boolean circuits [8]. As a result, a detailed assessment of the rigor of 25 feature attribution methods based on Shapley values, when compared with exactly computed Shapley 26 values has not been investigated. Furthermore, the definition Shapley values (as well as its use in 27 explainability) is purely axiomatic, i.e. there exists no formal proof that Shapley values capture any 28 specific properties related with explainability (even if defining such properties might prove elusive). 29

Feature selection represents a different alternative to feature attribution. The goal of feature selection 30

is to select a set of features as representing the reason for a prediction, i.e. if the selected features take 31

their assigned values, then the prediction cannot be changed. There are rigorous and non-rigorous 32

approaches for selecting the features that explain a prediction. This paper considers rigorous (or 33

model-precise) approaches for selecting such features. Furthermore, it should be plain that feature 34

selection must aim for irredundancy, since otherwise it would suffice to report all features as the 35

explanation. Given the universe of possible irreducible sets of feature selections that explain a 36 prediction, the features that do not occur in any such set are deemed irrelevant for a prediction; 37 otherwise features that occur in one or more feature selections are deemed relevant . 38

Since both feature attribution and feature selection measure contributions of features to explanations, 39

one would expect that the two approaches were related. However, this is not the case. Recent 40

Submitted to 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Do not distribute.

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

work [35] observed that feature attribution based on Shapley values could produce misleading information about features, in that irrelevant features (for feature selection) could be deemed more important (in terms of feature attribution) than relevant features (also for feature selection). Clearly, misleading information about the relative importance of features can easily induce human decision makers in error, by suggesting the wrong features as those to analyze in greater detail. Furthermore, situations where human decision makers can be misled are inadmissible in high-risk or safety-critical uses of ML. Furthermore, a number of possible misleading issues of Shapley values for explainability were identified [35], and empirically demonstrated to occur for some boolean functions. The existence in practice of those misleading issues with Shapley values for explainability is evidently problematic for their use as the theoretical underpinning of feature attribution methods.

However, earlier work [35] used a brute-force method to identify boolean functions, defined on a very small number of variables, where the misleading issues could be observed. A limitation of this earlier work [35] is that it offered no insights on how general the issues with Shapley values for explainability are. For example, it could be the case that the identified misleading issues might only occur for functions defined on a very small number of variables, or in a negligible number of functions, among the universe of functions defined on a given number of variables. If that were to be the case, then the issues with Shapley values for explainability might not be that problematic.

This paper proves that the identified misleading issues with Shapley values for explainability are much more general that what was reported in earlier work [35]. Concretely, the paper proves that, for any number of features larger than a small k (either 2 or 3), one can easily construct functions which exhibit the identified misleading issues. The main implication of our results is clear: the use of Shapley values for explainability can, for an arbitrary large number of boolean (classification) functions, produce misleading information about the relative importance of features .

Organization. The paper is organized as follows. Section 2 introduces the notation and definitions used throughout the paper. Section 3 revisits and extends the issues with Shapley values for explainability reported in earlier work [35], and illustrates the existence of those issues in a number of motivating example boolean functions. Section 4 presents the paper's main results, proving that all the issues with Shapley values for explainability reported in earlier work [35] occur for boolean functions with arbitrarily larger number of variables. (Due to lack of space, the detailed proofs are all included in Appendix A, and the paper includes only brief insights into those proofs.) Also, the proposed constructions offer ample confidence that the number of functions displaying one or more of the issues is significant. Section 5 concludes the paper.

2 Preliminaries

Boolean functions. Let B = { 0 , 1 } . The results in the paper consider boolean functions, defined on m boolean variables, i.e. κ : B m → B . (The fact that we consider only boolean functions does not restrict in the significance of the results.)

In the rest of the paper, we will use the boolean functions shown in Figure 1, which are represented by truth tables. The highlighted rows will serve as concrete examples throughout.

Classification in ML. A classification problem is defined on a set of features F = { 1 , . . . , m } , each with domain D i , and a set of classes K = { c 1 , c 2 , . . . , c K } . (As noted above, we will assume D i = B for 1 ≤ i ≤ m , but domains could be categorical or ordinal. Also, we will assume K = B .) Feature space F is defined as the cartesian product of the domains of the features, in order: F = D 1 ×···× D m , which will be B m throughout the paper. A classification function is a non-constant map from feature space into the set of classes, κ : F →K . (Clearly, a classifier would be useless if the classification function were constant.) Throughout the paper, we will not distinguish between classifiers and boolean functions. An instance is a pair ( v , c ) representing a point v = ( v 1 , . . . , v m ) in feature space, and the classifier's prediction, i.e. κ ( v ) = c . Moreover, we let x = ( x 1 , . . . , x m ) denote an arbitrary point in the feature space. Abusing notation, we will also use x a..b to denote x a , . . . , x b , and v a..b to denote v a , . . . , v b . Finally, a classifier M is a tuple ( F , F , K , κ ) . In addition, an explanation problem E is a tuple ( M , ( v , c )) , where M = ( F , F , K , κ ) is a classifier.

Shapley values for explainability. Shapley values were first introduced by L. Shapley [60] in the context of game theory. Shapley values have been extensively used for explaining the predictions of ML models, e.g. [64, 65, 20, 48, 15, 52, 62, 69], among a vast number of recent examples. The complexity of computing Shapley values (as proposed in SHAP [48]) has been studied in recent

x

1

x

2

x

3

κ

I

1

(

x

)

0

0

0

0

0

0

1

0

0

1

0

0

0

1

1

0

1

0

0

0

1

0

1

1

1

1

0

1

1

1

1

1

(a) Function

κ

I

1

x

1

x

2

x

3

κ

I

3

(

x

)

0

0

0

0

0

0

1

0

0

1

0

0

0

1

1

1

1

0

0

1

1

0

1

0

1

1

0

1

1

1

1

1

(b) Function

κ

I

3

x

1

x

2

x

3

x

4

κ

I

4

(

x

)

0

0

0

0

0

0

0

0

1

0

0

0

1

0

0

0

0

1

1

0

0

1

0

0

0

0

1

0

1

0

0

1

1

0

0

0

1

1

1

1

1

0

0

0

0

1

0

0

1

0

1

0

1

0

1

1

0

1

1

0

1

1

0

0

1

1

1

0

1

1

1

1

1

0

1

1

1

1

1

1

(c) Function

κ

I

4

x

1

x

2

x

3

x

4

κ

I

5

(

x

)

0

0

0

0

0

0

0

0

1

0

0

0

1

0

0

0

0

1

1

0

0

1

0

0

0

0

1

0

1

0

0

1

1

0

0

0

1

1

1

1

1

0

0

0

0

1

0

0

1

0

1

0

1

0

0

1

0

1

1

1

1

1

0

0

0

1

1

0

1

1

1

1

1

0

0

1

1

1

1

0

(d) Function

κ

I

5

years [8, 21, 7, 22]. This section provides a brief overview of Shapley values. Throughout the section, 95 we adapt the notation used in recent work [8, 7], which builds on the work of [48]. 96

Let Υ : 2 F → 2 F be defined by 1 , 97

i.e. for a given set S of features, and parameterized by the point v in feature space, Υ( S ; v ) denotes 98 all the points in feature space that have in common with v the values of the features specified by S . 99

Also, let ϕ : 2 F → R be defined by, 100

For the purposes of this paper, we consider solely a uniform input distribution, and so the dependency 101 on the input distribution is not accounted for. A more general formulation is considered in related 102 work [8, 7]. However, assuming a uniform distribution suffices for the purposes of this paper. As a 103 result, given a set S of features, ϕ ( S ; M , v ) represents the average value of the classifier over the 104 points of feature space represented by Υ( S ; v ) . 105

106

107

108

109

110

111

Finally, let Sv : F → R be defined by 2 ,

Given an instance ( v , c ) , the Shapley value assigned to each feature measures the contribution of that feature with respect to the prediction. A positive/negative value indicates that the feature can contribute to changing the prediction, whereas a value of 0 indicates no contribution.

Example 1. We consider the example boolean functions of Figure 1. If the functions are represented by a truth table, then the Shapley values can be computed in polynomial time on the size of the

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

truth table, since the number of subsets considered in (3) is also polynomial on the size of the truth table [35]. (Observe that for each subset used in (3), we can use the truth table for computing the average values in (2).) For example, for κ I 1 and for the point in feature space (0 , 0 , 1) , one can compute the following Shapley values: Sv (1) = -0 . 417 , Sv (2) = -0 . 042 , and Sv (3) = 0 . 083 .

Logic-based explanations. There has been recent work on developing formal definitions of explanations. One type of explanations are abductive explanations [37] (AXp), which corresponds to a PI-explanations [61] in the case of boolean classifiers. AXp's represent prime implicants of the discrete-valued classifier function (which computes the predicted class). AXp's can also be viewed as an instantiation of logic-based abduction [24, 59, 13, 23]. Throughout this paper we will opt to use the acronym AXp to refer to abductive explanations.

Let us consider a given classifier, computing a classification function κ on feature space F , a point v ∈ F , with prediction c = κ ( v ) , and let X denote a subset of the set of features F , X ⊆ F . X is a weak AXp for the instance ( v , c ) if,

where c = κ ( v ) . Thus, given an instance ( v , c ) , a (weak) AXp is a subset of features which, if fixed to the values dictated by v , then the prediction is guaranteed to be c , independently of the values assigned to the other features.

Moreover, X ⊆ F is an AXp if, besides being a weak AXp, it is also subset-minimal, i.e.

Observe that an AXp can be viewed as a possible irreducible answer to a ' Why? ' question, i.e. why is the classifier's prediction c ? It should be plain in this work, but also in earlier work, that the representation of AXp's using subsets of features aims at simplicity. The sufficient condition for the prediction is evidently the conjunction of literals associated with the features contained in the AXp.

Example 2. Similar to the computation of Shapley values, given a truth table representation of a function, and for a given instance, there is a polynomial-time algorithm for computing the AXp's [35]. For example, for function κ I 4 (see Figure 1c), and for the instance ((0 , 0 , 1 , 1) , 0) , it can be observed that, if features 3 and 4 are allowed to take other values, the prediction remains at 0. Hence, { 1 , 2 } is an WAXp, which is easy to conclude that it is also an AXp. When interpreted as a rule, the AXp would yield the rule:

In a similar way, if features 1 and 3 are allowed to take other values, the prediction remains at 0. Hence, { 2 , 4 } is another WAXp (which can easily be shown to be an AXp). Furthermore, considering all other possible subsets of fixed features, allows us to conclude that there are no more AXp's.

Similarly to the case of AXp's, one can define (weak) contrastive explanations (CXp's) [53, 36]. Y ⊆ F is a weak CXp for the instance ( v , c ) if,

̸

(As before, for simplicity we keep the parameterization of WCXp on κ , v and c implicit.) Thus, given an instance ( v , c ) , a (weak) CXp is a subset of features which, if allowed to take any value from their domain, then there is an assignment to the features that changes the prediction to a class other than c , this while the features not in the explanation are kept to their values.

Furthermore, a set Y ⊆ F is a CXp if, besides being a weak CXp, it is also subset-minimal, i.e.

A CXp can be viewed as a possible irreducible answer to a ' Why Not? ' question, i.e. why isn't the classifier's prediction a class other than c ?

Example 3. For the example function κ I 4 (see Figure 1c), and instance ((0 , 0 , 1 , 1) , 0) , if we fix features 1, 3 and 4, respectively to 0, 1 1, then by allowing feature 2 to change value, we see that the prediction changes, e.g. by considering the point (0 , 1 , 1 , 1) with prediction 1. Thus, { 2 } is a CXp. In a similar way, by fixing the features 2 and 3, respectively to 0 and 1, then by allowing features 1 and 4 to change value, we conclude that the prediction changes. Hence, { 1 , 4 } is also a CXp.

The sets of AXp's and CXp's are defined as follows:

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

(The parameterization on M and v is unnecessary, since the explanation problem E already accounts for those.) Moreover, let F A ( E ) = ∪ X∈ A ( E ) X and F C ( E ) = ∪ Y∈ C ( E ) Y . F A ( E ) aggregates the features occurring in any abductive explanation, whereas F C ( E ) aggregates the features occurring in any contrastive explanation. In addition, minimal hitting set duality between AXp's and CXp's [36] yields the following result 3 .

Proposition 1. F A ( E ) = F C ( E ) .

Feature (ir)relevancy in explainability. Given the definitions above, we have the following characterization of features [33, 34, 32]:

By Proposition 1, the definitions of necessary and relevant feature could instead use C ( E ) . Throughout the paper, we will use the predicate Irrelevant ( i ) which holds true if feature i is irrelevant, and predicate Relevant ( i ) which holds true if feature i is relevant. Furthermore, it should be noted that feature irrelevancy is a fairly demanding condition in that, a feature i is irrelevant if it is not included in any subset-minimal set of features that is sufficient for the prediction.

Example 4. For the example function κ I 4 (see Figure 1c), and from Example 2, and instance ((0 , 0 , 1 , 1) , 0) , it becomes clear that feature 3 is irrelevant. Similarly, it is easy to conclude that features 1, 2 and 4 are relevant.

How irrelevant are irrelevant features? The fact that a feature is declared irrelevant for an explanation problem E = ( M , ( v , c )) is significant. Given the minimal hitting set duality between abductive and contrastive explanations, then an irrelevant features does not occur neither in any abductive explanation, nor in any contrastive explanation. Furthermore, from the definition of AXp, each abductive explanation for E can be represented as a logic rule. Let R denote the set of all irreducible logic rules which can be used to predict c , given the literals dictated by v . Then, an irrelevant feature does not occur in any of those rules. Example 4 illustrates the irrelevancy of feature 3, in that feature 3 would not occur in any irreducible rule for κ I 4 when predicting 0 using literals consistent with (0 , 0 , 1 , 1) .

To further strengthen the above discussion, let us consider a (feature selection based) explanation X ⊆ F such that WAXp ( X ) holds (i.e. (4) is true, and so X is sufficient for the prediction). Moreover, let i ∈ F be an irrelevant feature, such that i ∈ X . Then, by definition of irrelevant feature, there must exist some Z ⊆ ( X \{ i } ) , such that WAXp ( Z ) also holds (i.e. Z is also sufficient for the prediction). It is simple to understand why such set Z must exist. By definition of irrelevant feature, and because i ∈ X , then X is not an AXp. However, there must exist an AXp W ⊊ X which, by definition of irrelevant feature, must not include i . Furthermore, and invoking Occam's razor 4 , there is no reason to select X over Z , and this remark applies to any set of features containing some irrelevant feature.

Related work. Shapley values for explainability is one of the hallmarks of feature attribution methods in XAI [64, 65, 20, 48, 15, 47, 52, 17, 26, 16, 25, 62, 40, 58, 69, 5, 12, 30, 4, 67]. Motivated by the success of Shapley values for explainability, there exists a burgeoning body of work on using Shapley values for explainability (e.g. [39, 74, 71, 38, 54, 10, 6, 76, 44, 3, 63, 75, 49, 68, 45, 46, 77, 28, 29, 31, 1]). Recent work studied the complexity of exactly computing Shapley values in the context of explainability [8, 21, 22]. Finally, there have been proposals for the exact computation of Shapley values in the case of circuit-based classifiers [8]. Although there exist some differences in the proposals for the use of Shapley values for explainability, the basic formulation is the same and can be expressed as in Section 2.

A number of authors have reported pitfalls with the use of SHAP and Shapley values as a measure of feature importance [73, 42, 66, 52, 27, 72, 55, 2, 70, 41, 14]. However, these earlier works do not identify fundamental flaws with the use of Shapley values in explainability. Attempts at addressing those pitfalls include proposals to integrate Shapley values with abductive explanations, as reported in recent work [43].

Formal explainability is a fairly recent topic of research. Recent accounts include [51, 9, 50, 19].

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

Recent work [35] argued for the inadequacy of Shapley values for explainability, by demonstrating experimentally that the information provided by Shapley values can be misleading for a human decision-maker. The approach proposed in [35] is based on exhaustive function enumeration, and so does not scale beyond a few features. However, this paper uses the truth-table algorithms outlined in [35], in all the examples, both for computing Shapley values, for computing explanations, and for deciding feature relevancy.

3 Relating Shapley Values with Feature Relevancy

Recent work [35] showed the existence of boolean functions (with up to four variables) that revealed a number of issues with Shapley values for explainability. All those issues are related with taking feature relevancy into consideration. (In [35], these functions were searched by exhaustive enumeration of all the boolean functions up to a threshold on the number of variables.)

Issues with Shapley values for explainability. In this paper, we consider the following main issues of Shapley values for explainability:

I1. For a boolean classifier, with an instance ( v , c ) , and feature i such that,

̸

Thus, an I1 issue is such that the feature is irrelevant, but its Shapley value is non-zero.

I2. For a boolean classifier, with an instance ( v , c ) and features i 1 and i 2 such that,

Thus, an I2 issue is such that there is at least one irrelevant feature exhibiting a Shapley value larger (in absolute value) than the Shapley of a relevant feature.

I3. For a boolean classifier, with instance ( v , c ) , and feature i such that,

Thus, an I3 issue is such that the feature is relevant, but its Shapley value is zero.

I4. For a boolean classifier, with instance ( v , c ) , and features i 1 and i 2 such that,

̸

Thus, an I4 issue is such that there is at least one irrelevant feature with a non-zero Shapley value and a relevant feature with a Shapley value of 0.

I5. For a boolean classifier, with instance ( v , c ) and feature i such that,

̸

Thus, an I5 issue is such that there is one irrelevant feature exhibiting the highest Shapley value (in absolute value). (I5 can be viewed as a special case of the other issues, and so it is not analyzed separately in earlier work [35].)

The issues above are all related with Shapley values for explainability giving misleading information to a human decision maker, by assigning some importance to irrelevant features, by not assigning enough importance to relevant features, by assigning more importance to irrelevant features than to relevant features and, finally, by assigning the most importance to irrelevant features.

In the rest of the paper we consider mostly I1, I3, I4 and I5, given that I5 implies I2.

Proposition 2. If a classifier and instance exhibits issue I5, then they also exhibit issue I2.

Examples. This section studies the example functions of Figure 1, which were derived from the main results of this paper (see Section 4). These example functions will then be used to motivate the rationale for how those results are proved. In all cases, the reported Shapley values are computed using the truth-table algorithm outlined in earlier work [35]. Similarly, the relevancy/irrelevancy claims of features use the truth-table algorithms outlined in earlier work [35].

Example 5. Figure 1a illustrates a boolean function that exhibits issue I1. By inspection, we can conclude that the function shown corresponds to κ I 1 ( x 1 , x 2 , x 3 ) = ( x 1 ∧ x 2 ∧ ¬ x 3 ) ∨ ( x 1 ∧ x 3 ) . Moreover, for the instance ((0 , 0 , 1) , 0) , Table 1 confirms that an issue I1 is identified.

Example 6. Figure 1b illustrates a boolean function that exhibits issue I3. By inspection, we can conclude that the function shown corresponds to κ I 3 ( x 1 , x 2 , x 3 ) = ( x 1 ∧¬ x 3 ) ∨ ( x 2 ∧ x 3 ) . Moreover, for the instance ((1 , 1 , 1) , 1) , Table 1 confirms that an issue I3 is identified.

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

̸

̸

Example 7. Figure 1c illustrates a boolean function that exhibits issue I4. By inspection, we can conclude that the function shown corresponds to κ I 4 ( x 1 , x 2 , x 3 , x 4 ) = ( x 1 ∧ x 2 ∧ ¬ x 3 ) ∨ ( x 1 ∧ x 3 ∧ ¬ x 4 ) ∨ ( x 2 ∧ x 3 ∧ x 4 ) . Moreover, for the instance ((0 , 0 , 1 , 1) , 0) , Table 1 confirms that an issue I4 is identified.

Example 8. Figure 1d illustrates a boolean function that exhibits issue I5. By inspection, we can conclude that the function shown corresponds to κ I 5 ( x 1 , x 2 , x 3 , x 4 ) = (( x 1 ∧ x 2 ∧ ¬ x 3 ) ∨ ( x 1 ∧ x 3 ∧¬ x 2 ) ∨ ( x 2 ∧ x 3 ∧¬ x 1 )) ∧ x 4 . Moreover, for the instance ((1 , 1 , 1 , 1) , 0) , Table 1 confirms that an issue I5 is identified.

It should be underscored that Shapley values for explainability are not expected to give misleading information. Indeed, it is widely accepted that Shapley values measure the actual influence of a feature [64, 65, 48, 8, 21]. Concretely, [64] reads: ' ...if a feature has no influence on the prediction it is assigned a contribution of 0. ' But [64] also reads ' According to the 2nd axiom, if two features values have an identical influence on the prediction they are assigned contributions of equal size. The 3rd axiom says that if a feature has no influence on the prediction it is assigned a contribution of 0. ' (In this last quote, the axioms refer to the axiomatic characterization of Shapley values.) Furthermore, one might be tempted to look at the value of the prediction and relate that with the computed Shapley value. For example, in the last row of Table 1, the prediction is 0, and the irrelevant feature 4 has a positive Shapley value. As a result, one might be tempted to believe that the irrelevant feature 4 would contribute to changing the value of the prediction. This is of course incorrect, since an irrelevant feature does not occur in any CXp's (besides not occurring in any AXp's) and so it is never necessary to changing the prediction. The key point here is that irrelevant features are never necessary, neither to keep nor to change the prediction.

4 Refuting Shapley Values for Explainability

The purpose of this section is to prove that for arbitrary large numbers of variables, there exist boolean functions and instances for which the Shapley values exhibit the issues reported in recent work [35], and detailed in Section 3. (Instead of detailed proofs, this section describes the key ideas of each proof. The detailed proofs are included in Appendix A.)

Throughout this section, let m be the number of variables of the boolean functions we start from, and let n denote the number of variables of the functions we will be constructing. In this case, we set F = { 1 , . . . , n } . Furthermore, for the sake of simplicity, we opt to introduce the new features as the last features (e.g., feature n ). This choice does not affect the proof's argument in any way.

̸

Proposition 3. For any n ≥ 3 , there exist boolean functions defined on n variables, and at least one instance, which exhibit an issue I1, i.e. there exists an irrelevant feature i ∈ F , such that Sv ( i ) = 0 .

Proof idea. The proof proposes to construct boolean functions, with an arbitrary number of variables (no smaller than 3), and the picking of an instance, such that a specific feature is irrelevant for the prediction, but its Shapley value is non-zero. To illustrate the construction, the example function from Figure 1a is used (see also Example 5).

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

̸

The construction works as follows. We pick two non-constant functions κ 1 ( x 1 , . . . , x m ) and 289 κ 2 ( x 1 , . . . , x m ) , defined on m features, and such that: i) κ 1 ⊨ κ 2 (which signifies that ∀ ( x ∈ 290 F ) .κ 1 ( x ) → κ 2 ( x ) ), and ii) κ 1 = κ 2 . Observe that κ 1 can be any boolean function defined on m 291 variables, as long as κ 2 can also be defined. We then construct a new function by adding a new 292 feature n = m +1 , as follows: 293

For the resulting function κ , we pick an instance ( v , 0) such that: i) v n = 1 and ii) κ 1 ( v 1 ..m ) = κ 2 ( v 1 ..m ) = 0 . The proof hinges on the fact that feature n is irrelevant, but Sv ( n ) = 0 .

̸

̸

For the function Figure 1a, we set κ 1 ( x 1 , x 2 ) = x 1 ∧ x 2 and κ 1 ( x 1 , x 2 ) = x 1 . Thus, as shown in Example 5, κ I 1 ( x 1 , x 2 , x 3 ) = ( x 1 ∧ x 2 ∧ ¬ x 3 ) ∨ ( x 1 ∧ x 3 ) , which represents the function κ ( x 1 , x 2 , x 3 ) . It is also clear that κ 1 ⊨ κ 2 . Moreover, and as Example 5 and Table 1 show, it is the case that feature 3 is irrelevant and Sv (3) = 0 .

Proposition 4. For any odd n ≥ 3 , there exist boolean functions defined on n variables, and at least one instance, which exhibits an I3 issue, i.e. for which there exists a relevant feature i ∈ F , such that Sv ( i ) = 0 .

Proof idea. The proof proposes to construct boolean functions, with an arbitrary number of variables (no smaller than 3), and the picking of an instance, such that a specific feature is relevant for the prediction, but its Shapley value is zero. To illustrate the construction, the example function from Figure 1b is used (see also Example 6).

The construction works as follows. We pick two non-constant functions κ 1 ( x 1 , . . . , x m ) and κ 2 ( x m +1 , . . . , x 2 m ) , each defined on m features, where κ 2 corresponds to κ 1 , but with a change of variables. Observe that κ 1 can be any boolean function. We then construct a new function, defined in terms of κ 1 and κ 2 , by adding a new feature n = 2 m +1 , as follows:

For the resulting function κ , we pick an instance ( v , 1) such that: i) v n = 1 , ii) v i = v m + i for any 1 ≤ i ≤ m , and iii) κ 1 ( v 1 ..m ) = κ 2 ( v m +1 .. 2 m ) = 1 . The proof hinges on the fact that feature n is relevant, but Sv ( n ) = 0 .

For the function Figure 1b, we set κ 1 ( x 1 ) = x 1 and κ 1 ( x 2 ) = x 2 . Thus, as shown in Example 6, κ I 3 ( x 1 , x 2 , x 3 ) = ( x 1 ∧¬ x 3 ) ∨ ( ¬ x 2 ∧ x 3 ) , which represents the function κ ( x 1 , x 2 , x 3 ) . Moreover, and as Example 6 and Table 1 show, it is the case that feature 3 is relevant and Sv (3) = 0 .

̸

Proposition 5. For any even n ≥ 4 , there exist boolean functions defined on n variables, and at least one instance, for which there exists an irrelevant feature i 1 ∈ F , such that Sv ( i 1 ) = 0 , and a relevant feature i 2 ∈ F \ { i 1 } , such that Sv ( i 2 ) = 0 .

Proof idea. The proof proposes to construct boolean functions, with an arbitrary number of variables (no smaller than 4), and the picking of an instance, such that two specific features are such that one is relevant but has a Shapley value of 0, and the other one is irrelevant but has a non-zero Shapley values. To illustrate the construction, the example function from Figure 1c is used (see also Example 7).

The construction works as follows. We pick two non-constant functions κ 1 ( x 1 , . . . , x m ) and κ 2 ( x m +1 , . . . , x 2 m ) , each defined on m features, where κ 2 corresponds to κ 1 , but with a change of variables. Also, observe that κ 1 can be any boolean function. We then construct a new function, defined in terms of κ 1 and κ 2 , by adding two new features. We let the new features be n -1 and n , and so n = 2 m +2 . The function is organized as follows:

̸

For this function, we pick an instance ( v , 0) such that: i) v n -1 = v n = 1 , ii) v i = v m + i for any 1 ≤ i ≤ m , and iii) κ 1 ( v 1 ..m ) = κ 2 ( v m +1 .. 2 m ) = 0 . The proof hinges on the fact that feature n -1 is irrelevant, feature n is relevant, and Sv ( n -1) = 0 and Sv ( n ) = 0 .

̸

For the function Figure 1c, we set κ 1 ( x 1 ) = x 1 and κ 1 ( x 2 ) = x 2 , Thus, as shown in Example 7, κ I 4 ( x 1 , x 2 , x 3 , x 4 ) = ( x 1 ∧ x 2 ∧ ¬ x 3 ) ∨ ( x 1 ∧ x 3 ∧ ¬ x 4 ) ∨ ( x 2 ∧ x 3 ∧ x 4 ) , which represents the function κ ( x 1 , x 2 , x 3 , x 4 ) . Moreover, and as Example 7 and Table 1 show, it is the case that feature 3 is irrelevant, feature 4 is relevant, and also Sv (3) = 0 and Sv (4) = 0 .

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

Proposition 6. For any n ≥ 4 , there exists boolean functions defined on n variables, and at least one instance, for which there exists an irrelevant feature i ∈ F = { 1 , . . . , n } , such that | Sv ( i ) | = max {| Sv ( j ) | | j ∈ F} .

Proof idea. The proof proposes to construct boolean functions, with an arbitrary number of variables (no smaller than 4), and the picking of an instance, such that one specific feature is irrelevant but it has the Shapley value with the largest absolute values. To illustrate the construction, the example function from Figure 1d is used (see also Example 8).

The construction works as follows. We pick one non-constant function κ 1 ( x 1 , . . . , x m ) , defined on m features, such that: i) κ 1 predicts a specific point v 1 ..m as 0, moreover, for any point x 1 ..m such that d H ( x 1 ..m , v 1 ..m ) = 1 , κ 1 ( x 1 ..m ) = 1 , where d H ( · ) denotes the Hamming distance. ii) and κ 1 predicts all the other points as 0. For example, let κ 1 ( x 1 , . . . , x m ) = 1 iff ∑ m i =1 ¬ x 1 = 1 . We then construct a new function, defined in terms of κ 1 , by adding one new feature. We let the new feature be n , and so n = m +1 . The new function is organized as follows:

For this function, we pick the instance ( v , 0) such that: i) v n = 1 , ii) v 1 ..m is the only point within the Hamming ball and iii) κ 1 ( v 1 ..m ) = 0 . The proof hinges on the fact that feature n is irrelevant, but ∀ (1 ≤ j ≤ m ) . | Sv ( j ) | < | Sv ( n ) | .

For the function Figure 1d, we set κ 1 ( x 1 , x 2 , x 3 ) = ( x 1 ∧ x 2 ∧¬ x 3 ) ∨ ( x 1 ∧ x 3 ∧¬ x 2 ) ∨ ( x 2 ∧ x 3 ∧¬ x 1 ) (i.e. the function takes value 1 when exactly one feature is 0). Thus, as shown in Example 7, κ I 5 ( x 1 , x 2 , x 3 , x 4 ) = (( x 1 ∧ x 2 ∧¬ x 3 ) ∨ ( x 1 ∧ x 3 ∧¬ x 2 ) ∨ ( x 2 ∧ x 3 ∧¬ x 1 )) ∧ x 4 , which represents the function κ ( x 1 , x 2 , x 3 , x 4 ) . Moreover, and as Example 8 and Table 1 show, it is the case that feature 4 is irrelevant and ∀ (1 ≤ j ≤ 3) . | Sv ( j ) | < | Sv (4) | .

For I2, we can restate the previous result, but such the functions constructed in the proof capture a more general family of functions.

Proposition 7. For any n ≥ 4 , there exist boolean functions defined on n variables, and at least one instance, for which there exists an irrelevant feature i 1 ∈ F , and a relevant feature i 2 ∈ F \ { i 1 } , such that | Sv ( i 1 ) | > | Sv ( i 2 ) | .

As noted above, for Propositions 3 to 5, the choice of the starting function is fairly flexible. In contrast, for Proposition 6, we pick one concrete function, which represents a trivial lower bound. As a result, and with the exception of I5, we can prove the following (fairly loose) lower bounds on the number of functions exhibiting the different issues.

Proposition 8. For Propositions 3 to 5,and Proposition 7 the following are lower bounds on the numbers issues exhibiting the respective issues:

( n -1)

5 Conclusions

This paper gives theoretical arguments to the fact that Shapley values for explainability can produce misleading information about the relative importance of features. The paper distinguishes between the features that occur in one or more of the irreducible rule-based explanations, i.e. the relevant features, from those that do not occur in any irreducible rule-based explanation, i.e. the irrelevant features. The paper proves that, for boolean functions with arbitrary number of variables, irrelevant features can be deemed more important, given their Shapley value, than relevant features. Our results are also significant in practical deployment of explainability solutions. Indeed, misleading information about relative feature importance can induce human decision makers in error, by persuading them to look at the wrong causes of predictions.

One direction of research is to develop a better understanding of the distributions of functions exhibiting one or more of the issues of Shapley values.

.

384

References

J. Adeoye, L.-W. Zheng, P. Thomson, S.-W. Choi, and Y.-X. Su. Explainable ensemble 385 learning model improves identification of candidates for oral cancer screening. Oral Oncology , 386 136:106278, 2023. 387

D. Afchar, V. Guigue, and R. Hennequin. Towards rigorous interpretations: a formalisation of 388 feature attribution. In ICML , pages 76-86, 2021. 389

R. O. Alabi, A. Almangush, M. Elmusrati, I. Leivo, and A. A. Mäkitie. An interpretable machine 390 learning prognostic system for risk stratification in oropharyngeal cancer. International Journal 391 of Medical Informatics , 168:104896, 2022. 392

M. S. Alam and Y. Xie. Appley: Approximate Shapley value for model explainability in linear 393 time. In Big Data , pages 95-100, 2022. 394

E. Albini, J. Long, D. Dervovic, and D. Magazzeni. Counterfactual Shapley additive explana395 tions. In FACCT , pages 1054-1070, 2022. 396

B. Alsinglawi, O. Alshari, M. Alorjani, O. Mubin, F. Alnajjar, M. Novoa, and O. Darwish. 397 An explainable machine learning framework for lung cancer hospital length of stay prediction. 398 Scientific reports , 12(1):1-10, 2022. 399

M. Arenas, P. Barceló, L. E. Bertossi, and M. Monet. On the complexity of SHAP-score-based 400 explanations: Tractability via knowledge compilation and non-approximability results. CoRR , 401 abs/2104.08015, 2021. 402

M. Arenas, P. Barceló, L. E. Bertossi, and M. Monet. The tractability of SHAP-score-based 403 explanations for classification over deterministic and decomposable boolean circuits. In AAAI , 404 pages 6670-6678, 2021. 405

G. Audemard, S. Bellart, L. Bounia, F. Koriche, J. Lagniez, and P. Marquis. On the explanatory 406 power of boolean decision trees. Data Knowl. Eng. , 142:102088, 2022. 407

M. L. Baptista, K. Goebel, and E. M. Henriques. Relation between prognostics predictor 408 evaluation metrics and local interpretability SHAP values. Artificial Intelligence , 306:103667, 409 2022. 410

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Occam's razor. Inf. Process. 411 Lett. , 24(6):377-380, 1987. 412

413

414

F. Bodria, S. Rinzivillo, D. Fadda, R. Guidotti, F. Giannotti, and D. Pedreschi. Explaining Black Box with Visual Exploration of Latent Space. In EuroVis - Short Papers , 2022.

415

416

417

418

419

420

421

422

423

424

425

T. Bylander, D. Allemang, M. C. Tanner, and J. R. Josephson. The computational complexity of abduction. Artif. Intell. , 49(1-3):25-60, 1991.

T. W. Campbell, H. Roder, R. W. Georgantas III, and J. Roder. Exact Shapley values for local and model-true explanations of decision tree ensembles. Machine Learning with Applications , 9:100345, 2022.

J. Chen, L. Song, M. J. Wainwright, and M. I. Jordan. L-Shapley and C-Shapley: Efficient model interpretation for structured data. In ICLR , 2019.

I. Covert and S. Lee. Improving KernelSHAP: Practical Shapley value estimation using linear regression. In AISTATS , pages 3457-3465, 2021.

I. Covert, S. M. Lundberg, and S. Lee. Understanding global feature contributions with additive importance measures. In NeurIPS , 2020.

Y. Crama and P. L. Hammer. Boolean functions: Theory, algorithms, and applications . Cam426 bridge University Press, 2011. 427

A. Darwiche and A. Hirth. On the (complete) reasons behind decisions. J. Log. Lang. Inf. , 428 32(1):63-88, 2023. 429

A. Datta, S. Sen, and Y. Zick. Algorithmic transparency via quantitative input influence: Theory 430 and experiments with learning systems. In IEEE S&P , pages 598-617, 2016. 431

G. V. den Broeck, A. Lykov, M. Schleich, and D. Suciu. On the tractability of SHAP explanations. 432 In AAAI , pages 6505-6513, 2021. 433

G. V. den Broeck, A. Lykov, M. Schleich, and D. Suciu. On the tractability of SHAP explanations. 434 J. Artif. Intell. Res. , 74:851-886, 2022. 435

T. Eiter and G. Gottlob. The complexity of logic-based abduction. J. ACM , 42(1):3-42, 1995. 436

G. Friedrich, G. Gottlob, and W. Nejdl. Hypothesis classification, abductive diagnosis and 437 therapy. In ESE , pages 69-78, 1990. 438

439

440

C. Frye, D. de Mijolla, T. Begley, L. Cowton, M. Stanley, and I. Feige. Shapley explainability on the data manifold. In ICLR , 2021.

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

C. Frye, C. Rowat, and I. Feige. Asymmetric Shapley values: incorporating causal knowledge into model-agnostic explainability. In NeurIPS , 2020.

D. V. Fryer, I. Strümke, and H. D. Nguyen. Shapley values for feature selection: The good, the bad, and the axioms. IEEE Access , 9:144352-144360, 2021.

I. B. Galazzo, F. Cruciani, L. Brusini, A. M. A. Salih, P. Radeva, S. F. Storti, and G. Menegaz. Explainable artificial intelligence for magnetic resonance imaging aging brainprints: Grounds and challenges. IEEE Signal Process. Mag. , 39(2):99-116, 2022.

M. Gandolfi, I. B. Galazzo, R. G. Pavan, F. Cruciani, N. Valè, A. Picelli, S. F. Storti, N. Smania, and G. Menegaz. eXplainable AI allows predicting upper limb rehabilitation outcomes in sub-acute stroke patients. IEEE J. Biomed. Health Informatics , 27(1):263-273, 2023.

R. Guidotti, A. Monreale, S. Ruggieri, F. Naretto, F. Turini, D. Pedreschi, and F. Giannotti. Stable and actionable explanations of black-box models through factual and counterfactual rules. Data Mining and Knowledge Discovery , pages 1-38, 2022.

T. Huang, D. Le, L. Yuan, S. Xu, and X. Peng. Machine learning for prediction of in-hospital mortality in lung cancer patients admitted to intensive care unit. Plos one , 18(1):e0280606, 2023.

X. Huang, M. C. Cooper, A. Morgado, J. Planes, and J. Marques-Silva. Feature necessity & relevancy in ML classifier explanations. In TACAS , pages 167-186. Springer, 2023.

X. Huang, Y. Izza, A. Ignatiev, and J. Marques-Silva. On efficiently explaining graph-based classifiers. In KR , pages 356-367, 2021.

X. Huang, Y. Izza, and J. Marques-Silva. Solving explainability queries with quantification: The case of feature relevancy. In AAAI , 2 2023. In Press.

X. Huang and J. Marques-Silva. The inadequacy of Shapley values for explainability. CoRR , abs/2302.08160, 2023.

A. Ignatiev, N. Narodytska, N. Asher, and J. Marques-Silva. From contrastive to abductive explanations and back again. In AIxIA , pages 335-355, 2020.

A. Ignatiev, N. Narodytska, and J. Marques-Silva. Abduction-based explanations for machine learning models. In AAAI , pages 1511-1519, 2019.

T. Inoguchi, Y. Nohara, C. Nojiri, and N. Nakashima. Association of serum bilirubin levels with risk of cancer development and total death. Scientific reports , 11(1):1-12, 2021.

T. Jansen, G. Geleijnse, M. Van Maaren, M. P. Hendriks, A. Ten Teije, and A. Moncada-Torres. Machine learning explainability in breast cancer survival. In Digital Personalized Health and Medicine , pages 307-311. IOS Press, 2020.

N. Jethani, M. Sudarshan, I. C. Covert, S. Lee, and R. Ranganath. FastSHAP: Real-time Shapley 474 value estimation. In ICLR , 2022. 475

I. Kumar, C. Scheidegger, S. Venkatasubramanian, and S. A. Friedler. Shapley residuals: 476 Quantifying the limits of the Shapley value for explanations. In NeurIPS , pages 26598-26608, 477 2021. 478

I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. A. Friedler. Problems with 479 Shapley-value-based explanations as feature importance measures. In ICML , pages 5491-5500, 480 2020. 481

482

483

C. Labreuche. Explanation of pseudo-boolean functions using cooperative game theory and prime implicants. In SUM , pages 295-308, 2022.

484

485

486

487

488

489

C. Ladbury, R. Li, J. Shiao, J. Liu, M. Cristea, E. Han, T. Dellinger, S. Lee, E. Wang, C. Fisher, et al. Characterizing impact of positive lymph node number in endometrial cancer using machine-learning: A better prognostic indicator than figo staging? Gynecologic Oncology , 164(1):39-45, 2022.

Y. Liu, Z. Liu, X. Luo, and H. Zhao. Diagnosis of parkinson's disease based on SHAP value feature selection. Biocybernetics and Biomedical Engineering , 42(3):856-869, 2022.

H. W. Loh, C. P. Ooi, S. Seoni, P. D. Barua, F. Molinari, and U. R. Acharya. Application 490 of explainable artificial intelligence for healthcare: A systematic review of the last decade 491 (2011-2022). Comput. Methods Programs Biomed. , 226:107161, 2022. 492

S. M. Lundberg, G. G. Erion, H. Chen, A. J. DeGrave, J. M. Prutkin, B. Nair, R. Katz, 493 J. Himmelfarb, N. Bansal, and S. Lee. From local explanations to global understanding with 494 explainable AI for trees. Nat. Mach. Intell. , 2(1):56-67, 2020. 495

S. M. Lundberg and S. Lee. A unified approach to interpreting model predictions. In NeurIPS , 496 pages 4765-4774, 2017. 497

M. Ma, R. Liu, C. Wen, W. Xu, Z. Xu, S. Wang, J. Wu, D. Pan, B. Zheng, G. Qin, et al. 498 Predicting the molecular subtype of breast cancer and identifying interpretable imaging features 499 using machine learning algorithms. European Radiology , pages 1-11, 2022. 500

J. Marques-Silva. Logic-based explainability in machine learning. CoRR , abs/2211.00541, 501 2022. 502

J. Marques-Silva and A. Ignatiev. Delivering trustworthy AI through formal XAI. In AAAI , 503 pages 12342-12350, 2022. 504

L. Merrick and A. Taly. The explanation game: Explaining machine learning models using 505 Shapley values. In CDMAKE , pages 17-38, 2020. 506

T. Miller. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell. , 507 267:1-38, 2019. 508

509

510

511

A. Moncada-Torres, M. C. van Maaren, M. P. Hendriks, S. Siesling, and G. Geleijnse. Explainable machine learning can outperform cox regression predictions and provide insights in breast cancer survival. Scientific reports , 11(1):6968, 2021.

512

513

514

515

516

517

518

519

520

R. K. Mothilal, D. Mahajan, C. Tan, and A. Sharma. Towards unifying feature attribution and counterfactual explanations: Different means to the same end. In AIES , pages 652-663, 2021.

M. T. Ribeiro, S. Singh, and C. Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In KDD , pages 1135-1144, 2016.

W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K. Müller, editors. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning . Springer, 2019.

M. Sarvmaili, R. Guidotti, A. Monreale, A. Soares, Z. Sadeghi, F. Giannotti, D. Pedreschi, and S. Matwin. A modularized framework for explaining black box classifiers for text data. In CCAI , 2022.

B. Selman and H. J. Levesque. Abductive and default reasoning: A computational core. In 521 AAAI , pages 343-348, 1990. 522

L. S. Shapley. A value for n -person games. Contributions to the Theory of Games , 2(28):307523 317, 1953. 524

A. Shih, A. Choi, and A. Darwiche. A symbolic approach to explaining bayesian network 525 classifiers. In IJCAI , pages 5103-5111, 2018. 526

D. Slack, A. Hilgard, S. Singh, and H. Lakkaraju. Reliable post hoc explanations: Modeling 527 uncertainty in explainability. In NeurIPS , pages 9391-9404, 2021. 528

A. Sorayaie Azar, S. Babaei Rikan, A. Naemi, J. Bagherzadeh Mohasefi, H. Pirnejad, 529 M. Bagherzadeh Mohasefi, and U. K. Wiil. Application of machine learning techniques 530 for predicting survival in ovarian cancer. BMC Medical Informatics and Decision Making , 531 22(1):345, 2022. 532

533

534

E. Strumbelj and I. Kononenko. An efficient explanation of individual classifications using game theory. J. Mach. Learn. Res. , 11:1-18, 2010.

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

E. Strumbelj and I. Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. , 41(3):647-665, 2014.

M. Sundararajan and A. Najmi. The many Shapley values for model explanation. In ICML , pages 9269-9278, 2020.

V. Voukelatou, I. Miliou, F. Giannotti, and L. Pappalardo. Understanding peace through the world news. EPJ Data Sci. , 11(1):2, 2022.

Y. Wang, J. Lang, J. Z. Zuo, Y. Dong, Z. Hu, X. Xu, Y. Zhang, Q. Wang, L. Yang, S. T. Wong, et al. The radiomic-clinical model using the SHAP method for assessing the treatment response of whole-brain radiotherapy: a multicentric study. European Radiology , pages 1-11, 2022.

D. S. Watson. Rational Shapley values. In FAccT , pages 1083-1094, 2022.

D. S. Watson, L. Gultchin, A. Taly, and L. Floridi. Local explanations via necessity and sufficiency: unifying theory and practice. In UAI , volume 161, pages 1382-1392, 2021.

E. Withnell, X. Zhang, K. Sun, and Y. Guo. Xomivae: an interpretable deep learning model for cancer classification using high-dimensional omics data. Briefings in Bioinformatics , 22(6):bbab315, 2021.

T. Yan and A. D. Procaccia. If you like Shapley then you'll love the core. In AAAI , pages 5751-5759, 2021.

K. Young, G. Booth, B. Simpson, R. Dutton, and S. Shrapnel. Deep neural network or dermatologist? CoRR , abs/1908.06612, 2019.

D. Yu, Z. Liu, C. Su, Y. Han, X. Duan, R. Zhang, X. Liu, Y. Yang, and S. Xu. Copy number variation in plasma as a tool for lung cancer prediction using extreme gradient boosting (xgboost) classifier. Thoracic cancer , 11(1):95-102, 2020.

R. Zarinshenas, C. Ladbury, H. McGee, D. Raz, L. Erhunmwunsee, R. Pathak, S. Glaser, R. Salgia, T. Williams, and A. Amini. Machine learning to refine prognostic and predictive nodal burden thresholds for post-operative radiotherapy in completely resected stage iii-n2 non-small cell lung cancer. Radiotherapy and Oncology , 173:10-18, 2022.

G. Zhang, Y. Shi, P. Yin, F. Liu, Y. Fang, X. Li, Q. Zhang, and Z. Zhang. A machine learning model based on ultrasound image features to assess the risk of sentinel lymph node metastasis in breast cancer patients: Applications of scikit-learn and SHAP. Frontiers in Oncology , 12, 2022.

Y. Zhang, Y. Weng, and J. Lund. Applications of explainable artificial intelligence in diagnosis 565 and surgery. Diagnostics , 12(2):237, 2022. 566

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors formally define five anomalies for an\nexplainability score and prove that for every n >= 4, there exist\nBoolean classifiers defined over n features that exhibit one or more\nof these anomalies for the SHAP score. In this way, the authors\nprovide evidence of the inadequacy of Shapley values for\nexplainability.\n\nThe aforementioned anomalies are defined by considering the concept of\nabductive explanation. More precisely, given a binary classification\nmodel M : {0,1}^n -> {0,1} and a tuple v in {0,1}^n, a subset X of {1,\n..., n} is said to be a weak abductive explanation of (M,v) if for\nevery y in {0,1}^n such that y[i] = v[i] for every i in X, it holds\nthat M(y) = M(v). In other words, the values of v for the features in\nX are enough to obtain the same result as M(v), so they are enough to\nexplain the output of M for v. Moreover, a subset X of {1, ..., n} is\nsaid to be an abductive explanation of (M,v) if X is a weak abductive\nexplanation of (M,v), and there is no weak abductive explanation X' of\n(M,v) such that X' is a proper subset of X. In other words, X is an\nabductive explanation for (M,v) if X is a minimal weak abductive\nexplanation for (M,v). Then a feature i is said to be relevant for\n(M,v) if there exists an abductive explanation X of (M,v) such that i\nbelongs to X, and otherwise i is said to be irrelevant for (M,v). With\nthis notion of irrelevance, the anomaly I5 for the SHAP score is\ndefined as the existence of a feature i such that i is irrelevant for\n(M,v), but the absolute value of the SHAP score of i is greater than\nthe absolute value of the SHAP score of every other feature. Thus,\nthis can be considered as an anomaly of the SHAP score, as i is an\nirrelevant feature that is considered more relevant according to the\nSHAP score that all the other features (some of which are\nrelevant). The other four anomalies considered in the paper (I1, I2,\nI3, I4) are defined in a similar fashion.\n\nSTRENGTHS:\n1. The five notions of anomaly studied in the paper clearly represent\nanomalies for explainability scores. These notions are properly\nformalized in the paper.\n\n2. The paper provides valuable insights into the SHAP score,\nspecifically providing a formal framework to assess its adequacy as an\nexplainability score.\n\n3. The paper provides one of the first formal results of the\ninadequacy of Shapley values for explainability.\n\n4. The paper is well written.\n\nWEAKNESSES:\n1. The results of the paper show that a tiny proportion of the Boolean\nclassifiers defined over n features exhibit some of the anomalies I1,\nI2, I3, I4 or I5. For example, the paper proves that at least\n2^{2^{n-1} - n - 3} Boolean classifiers exhibits anomaly I1, which is\na tiny proportion of the 2^{2^n} possible Boolean classifiers defined\nover n features. Hence, it could be the case that the vast majority of\nBoolean classifiers do not exhibit the anomalies studied in the paper.\n\n2. In practice Boolean classifiers are given in some specific\nformalism, such as decision trees or binary decision diagrams. The\nauthors do not provide any results about the formalisms that are\nsuitable to express the Boolean functions exhibiting anomalies. For\nexample, is it possible to express the Boolean functions in the proofs\nof Propositions 3, 4, 5 and 6 as decision trees of polynomial size in\nthe number n of features? If this is not possible, can these functions\nbe expressed as FBDDs (or d-DNNFs) of polynomial size in the number n\nof features?\n\nQUESTIONS:\nCould you please comment on the point 1. and 2. mentioned in\nWeaknesses.\n\nLIMITATIONS:\nThe following are the main limitations of this work (see Weaknesses), \nwhich are not addressed in the paper. \n\n- The results of the paper show that a tiny proportion of the Boolean\nclassifiers defined over n features exhibits some of the anomalies I1,\nI2, I3, I4 or I5. \n\n- The authors do not provide any results about the practical\nformalisms (such as decision trees) that are suitable to express the\nBoolean functions exhibiting anomalies.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reviews previous work on ideas of feature importance and hi-lighted inconsistencies with Shaley values. It defines ideas of importance and irrelevance of features in a Boolean ML model. These definitions are based on the idea of a minimal set of inputs needed to freeze an model output. necessary inputs are in every minimal coalition that can freeze the output, relevant inputs are in at least one minimal coalition, and irrelevant inputs are in no coalitions. They then go on to show that, among other issues, there exist Boolean models and certain inputs where irrelevant inputs are given large Shapley values, while relevant inputs are given a Shapley value of zero. Thus, the logic goes, Shapley values do not track importance.\n\nThe paper's original contributions are to prove that model/input pairs with issues exist/can be found for models of any input size. Previously, only small models were exhibited to have these issues, but it was unknown if larger models also had these issues. They also give lower bounds on the number of models that have these issues.\n\nSTRENGTHS:\n- Generally clear and straightforward exposition.\n- Good background and presentation of previous results.\n- Results are easy to understand.\n- idea of necessary, relevant, and irrelevant is intuitive.\n\nWEAKNESSES:\n- Paper is based on a comparison of apples to oranges, without an in-depth analysis of the issue. It is possible that the whole paper is based on a misunderstanding. Further analysis is needed.\n- Some grammatical issues.\n- Contributions are not very significant.\n\nQUESTIONS:\n- I believe, based on my calculations, that the paper's definition of Shapley Values in equation (3) is incorrect. For example, take model $F(x_1, x_2) = x_1$. I get that the Shapely values of the input $(1,1)$, assuming the baseline is $(0,0)$, is 1/2 for $x_1$ and 0 for $x_2$ for your definition. The Shapley value is actually 1 for $x_1$ and 0 for $x_2$, based on widely known definitions. The Shapley value satisfies the axiom of completeness, or efficiency, so the definition you gave is definitely not correct if it indeed gives the values 1/2 and 0.\n- The heart of the paper is a critique of Shapley values. It is asserted that there is a metric of feature importance and irrelevance and that the Shapley value assigns importance to what this other metric indicates is not important, and assigns no importance to what this other metric identifies as important. Admittedly this critique is from another paper, and this paper builds on this critique.\n\nFundamentally, the Shaley Value indicates a feature's contribution to function value change in comparison to the function evaluated at the comparative baseline input. The metrics of AXp and CXp are metrics of the ability of inputs to determine or alter a model at an input. One is about function change from a baseline, the other is about fixing or altering an output. There does not seem to be any analysis as to whether these notions are tracking the same underlying, and undefined, concept of \"importance.\" This paper defined \"important\" one way, while Shaley values define it another way. Is the problem that Shapley values do not indicate importance, or is the problem that Shapley values indicate importance according to one definition, while people are confused and think it indicates importance by another definition? The second case would account for this papers results, while problematizing the conclusion that the Shapley values do not indicate importance at all. This question needs further investigation and exposition in the paper, I believe.\n\nAs an illustration, let $F$ be a model with one input defined as $F(x) = x$. Also, let the input of consideration be $x=0$. By the definition of the paper, $x$ very important because $x$ is a \"necessary\" input. However, The Shaley value of $x$ is $0$ at the input $0$. An inadequate analysis of this results it that the Shapley value indicated zero importance to a \"necessary\" input, so the Shaley value does not track importance. An equally inadequate analysis, opposite of the first, is that a \"necessary\" input did not cause any function change, so the idea of a \"necessary\" input is flawed. However, a more sophisticated analysis is that necessary inputs can have no contribution to a model changing from the baseline. Shapley values measure one thing, and \"necessary\" inputs measure another.\n\nIn summary, it appears possible that this paper deals with two different metrics of \"important,\" that these metrics disagree, but also, the two metrics measure different things and work in different ways. Shapley values do not indicate importance according to this paper's definition, but why is that necessarily an issue with Shapley values?\n\n\nMinor issues:\n-  (39) \"contributions of features to explainability\". Do you mean function output?\n- (97)  $2^\\mathcal{F}$ should be $2^{|\\mathcal{F}|}$ ? Sometimes you use one, sometimes another.\n- (108) \"did\" contribute\", not \"can\" contribute\n- (117-118) \"which corresponds to a PI-explanations\" odd grammar.\n- please define A PI-explanations, prime implicants. Seems that unknowns are defined in terms of unknowns.\n- Unsure if WAXp is a function?\n- I1-I5 are stated in what appears to be sentence fragments.\n- I5 -> I2, but also, I5-> I1, and I4 -> I3, I1, and I2.\n\nLIMITATIONS:\nThe author has not discussed the limitations of the claim that Shapley values are refuted. This statement seems not entirely supported. See questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper demonstrates / constructs functions with features whose Shapley values (i.e., attributive importance in a prediction) is misaligned with their true relevance.

STRENGTHS:
- Addresses a theoretical gap in our understanding of Shapley values.

WEAKNESSES:
- I find the problem being investigated to be mostly a mathematical curiosity that so happened to be open and has now been addressed.

QUESTIONS:
n/a

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on definitions of feature necessity, relevancy, and irrelevancy from previous work,as well as systematic issues with Shapley values for explainability on boolean classifiers (e.g. non-zero Shapley values assigned to irrelevant features, zero Shapley values assigned to relevant features, among others) identified in previous work, the authors offer proof for their existence in functions with an arbitrary number of variables. They conclude that the existence of such systematic issues is cause for concern in using Shapley values for explainability, as misleading information about feature importance can induce errors in human decision making.\n\nSTRENGTHS:\n- Originality: The work offers proof for the existence of issues with Shapley value explanations on boolean functions with an arbitrary number of variables that were previously only studied empirically.\n- Quality and Clarity: The theoretical framework, preliminaries, and proofs are described in a very concise manner. Despite the theoretical nature of the paper, the authors are able to concisely state to the reader what is described in each formula (e.g. lines 125-127: Thus, given an instance (v, c), a (weak) AXp is a subset of features which, if fixed to the values dictated by v, then the prediction is guaranteed to be c, independently of the values assigned to the other features). Similarly, the main idea for each proof is described in a very intuitive manner, increasing readability of the paper significantly.\n- Significance: The present work proves systematic issues exhibited by Shapley value explanations on boolean functions. Shapley values are one of the most popular solutions, as they are based on clearly defined axioms, i.e. properties deemed desirable for explanations. For boolean functions, the present work shows that these axioms (which Shapley values do fulfill) may be lacking for treating irrelevant and relevant features as would be expected.\n\nWEAKNESSES:\nI am a bit concerned with the novelty, as the present work only provides proof for observations about unexpected behavior of Shapley value explanations for boolean functions that were already observed empirically in previous work (however, the authors also state themselves that these issues have been identified empirically in previous work). To raise concern about e.g. I1, it would be sufficient to simply identify a case where irrelevant values are assigned nonzero Shapley values. \n\nI also believe the title promises a bit more than is provided by the paper. The proofs and resulting claims are restricted to boolean functions, however, it would be interesting to see how and if the described issues occur in continuous settings, e.g., when explaining DNNs.\n\nQUESTIONS:\nI would be interested in how the findings translate to non-boolean functions, i.e., continuous setting. E.g. the definitions of relevant, irrelevant, and necessary features over classification change makes sense in the boolean setting, but not in a continuous setting where logit values easily change with features being added.\n\nAlso, it would be interesting to see some concrete recommendations as to how the identified issues might be avoided.\n\nLIMITATIONS:\nrestriction to boolean functions, as described in \"Weaknesses\" section. I think a paragraph of how the described proofs and observations may impact Shapley value explanations in more real-world settings would go a long way here, as well as suggestions on how to mitigate the proven issues.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}