Title: On Formal Feature Attribution and Its Approximation
Venue: NeurIPS 2023
Accepted: no
Keywords: Feature Attribution

Explainable AI

Formal Explanation
Abstract: Recent years have witnessed the widespread use of artificial intelligence (AI) algorithms and machine learning (ML) models. Despite their tremendous success, a number of vital problems like ML model brittleness, their fairness, and the lack of interpretability warrant the need for the active developments in explainable artificial intelligence (XAI) and formal ML model verification.  The two major lines of work in XAI include feature selection methods, e.g. Anchors, and feature attribution techniques, e.g. LIME and SHAP. Despite their promise, most of the existing feature selection and attribution approaches are susceptible to a range of critical issues, including explanation unsoundness and  out-of-distribution sampling. A recent formal approach to XAI (FXAI) although serving as an alternative to the above and free of these issues suffers from a few other limitations. For instance and besides the scalability limitation, the formal approach is unable to tackle the feature attribution problem. Additionally, a formal explanation despite being formally sound is typically quite large, which hampers its applicability in practical settings. Motivated by the above, this paper proposes a way to apply the apparatus of formal XAI to the case of feature attribution based on formal explanation enumeration. Formal feature attribution (FFA) is argued to be advantageous over the existing methods, both formal and non-formal. Given the practical complexity of the problem, the paper then proposes an efficient technique for approximating exact FFA. Finally, it offers experimental evidence of the effectiveness of the proposed approximate FFA in comparison to the existing feature attribution algorithms not only in terms of feature importance and but also in terms of their relative order.

Abstract

Recent years have witnessed the widespread use of artificial intelligence (AI) algorithms and machine learning (ML) models. Despite their tremendous success, a number of vital problems like ML model brittleness, their fairness, and the lack of interpretability warrant the need for the active developments in explainable artificial intelligence (XAI) and formal ML model verification. The two major lines of work in XAI include feature selection methods, e.g. Anchors, and feature attribution techniques, e.g. LIME and SHAP. Despite their promise, most of the existing feature selection and attribution approaches are susceptible to a range of critical issues, including explanation unsoundness and out-of-distribution sampling. A recent formal approach to XAI (FXAI) although serving as an alternative to the above and free of these issues suffers from a few other limitations. For instance and besides the scalability limitation, the formal approach is unable to tackle the feature attribution problem. Additionally, a formal explanation despite being formally sound is typically quite large, which hampers its applicability in practical settings. Motivated by the above, this paper proposes a way to apply the apparatus of formal XAI to the case of feature attribution based on formal explanation enumeration. Formal feature attribution (FFA) is argued to be advantageous over the existing methods, both formal and non-formal. Given the practical complexity of the problem, the paper then proposes an efficient technique for approximating exact FFA. Finally, it offers experimental evidence of the effectiveness of the proposed approximate FFA in comparison to the existing feature attribution algorithms not only in terms of feature importance and but also in terms of their relative order.

1 Introduction

Thanks to the unprecedented fast growth and the tremendous success, Artificial Intelligence (AI) 24 and Machine Learning (ML) have become a universally acclaimed standard in automated decision 25 making causing a major disruption in computing and the use of technology in general [1, 29, 35, 47]. 26 An ever growing range of practical applications of AI and ML, on the one hand, and a number of 27 critical issues observed in modern AI systems (e.g. decision bias [3] and brittleness [64]), on the 28

other hand, gave rise to the quickly advancing area of theory and practice of Explainable AI (XAI). 29

Numerous methods exist to explain decisions made by what is called black-box ML models [46, 48]. 30

Here, model-agnostic approaches based on random sampling prevail [46], with the most popular 31

being feature selection [56] and feature attribution [40, 56] approaches. Despite their promise, model32

agnostic approaches are susceptible to a range of critical issues, like unsoundness of explanations [21, 33

24] and out-of-distribution sampling [34, 62], which exacerbates the problem of trust in AI. 34

An alternative to model-agnostic explainers is represented by the methods building on the success of 35

formal reasoning applied to the logical representations of ML models [42, 61]. Aiming to address 36

the limitations of model-agnostic approaches, formal XAI (FXAI) methods themselves suffer from 37

a few downsides, including the lack of scalability and the requirement to build a complete logical 38

T1 (

50

k)

Status

=

Married?

Education

=

Dropout?

Rel.

=

Not-in-family?

-0.1569

0.0770

-0.1089

-0.3167

yes

no

yes

no

yes

no

T2 (

50

k)

Hours/w

40?

Status

=

Married?

Status

=

Never-Married?

-0.0200

-0.2404

-0.1245

0.0486

yes

no

yes

no

yes

no

T3 (

50

k)

Education

=

Doctorate?

40

<

Hours/w

45?

Rel.

=

Own-child?

0.0605

0.3890

-0.2892

-0.0580

yes

no

yes

no

yes

no

|

|

|

|

|

|

|

|

|

|

0.03

Hours/w <= 40

|

|

|

|

|

|

|

|

|

|

-0.03

Relationship: Not-in-family

|

|

|

|

|

|

|

|

|

|

0.07

Status: Separated

(a) LIME

|

|

|

|

|

|

|

|

|

0.01

Education: Bachelors

|

|

|

|

|

|

|

|

|

0.09

Status: Separated

|

|

|

|

|

|

|

|

|

0.11

Hours/w <= 40

|

|

|

|

|

|

|

|

|

-0.12

Relationship: Not-in-family

(b) SHAP

X

1

=

{ Education, Hours/w }

IF

Education

=

Bachelors

AND

Hours/w

40

THEN

Target

<

50

k

X

2

=

{ Education, Status }

IF

Education

=

Bachelors

AND

Status

=

Separated

THEN

Target

<

50

k

(c) AXp's

X

1

and

X

2

|

|

|

|

|

|

|

|

|

|

0.50

Status: Separated

|

|

|

|

|

|

|

|

|

|

0.50

Hours/w <= 40

|

|

|

|

|

|

|

|

|

|

1.00

Education: Bachelors

(d) FFA

representation of the ML model. Formal explanations also tend to be larger than their model-agnostic 39 counterparts because they do not reason about (unknown) data distribution [65]. Finally and most 40 importantly, FXAI methods have not been applied so far to answer feature attribution questions. 41

Motivated by the above, we define a novel formal approach to feature attribution, which builds on the 42 success of existing FXAI methods [42]. By exhaustively enumerating all formal explanations, we can 43 give a crisp definition of formal feature attribution (FFA) as the proportion of explanations in which 44 a given feature occurs. We argue that formal feature attribution is hard for the second level of the 45 polynomial hierarchy. Although it can be challenging to compute exact FFA in practice, we show that 46 existing anytime formal explanation enumeration methods can be applied to efficiently approximate 47 FFA. Our experimental results demonstrate the effectiveness of the proposed approach in practice 48 and its advantage over SHAP and LIME given publicly available tabular and image datasets, as well 49 as on a real application of XAI in the domain of Software Engineering [45, 52]. 50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

2 Background

This section briefly overviews the status quo in XAI and background knowledge the paper builds on.

2.1 Classification Problems

Classification problems consider a set of classes K = { 1 , 2 , . . . , k } 1 , and a set of features F = { 1 , . . . , m } . The value of each feature i ∈ F is taken from a domain D i , which can be categorical or ordinal, i.e. integer, real-valued or Boolean. Therefore, the complete feature space is defined as F ≜ ∏ m i =1 D i . A concrete point in feature space is represented by v = ( v 1 , . . . , v m ) ∈ F , where each component v i ∈ D i is a constant taken by feature i ∈ F . An instance or example is denoted by a specific point v ∈ F in feature space and its corresponding class c ∈ K , i.e. a pair ( v , c ) represents an instance. Additionally, the notation x = ( x 1 , . . . , x m ) denotes an arbitrary point in feature space, where each component x i is a variable taking values from its corresponding domain D i and representing feature i ∈ F . A classifier defines a non-constant classification function κ : F →K .

Many ways exist to learn classifiers

κ

given training data, i.e. a collection of labeled instances

(

v

, c

)

,

including decision trees [23] and their ensembles [11, 12], decision lists [57], neural networks [35], etc. Hereinafter, this paper considers boosted tree (BT) models trained with the use of XGBoost [12].

Example 1.

Figure 1 shows a BT model trained for a simplified version of the a data instance

v

=

{

Education

=

Bachelors

,

Status

=

adult

Separated, Occupation dataset [33]. For

=

Sales, Relation-

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

ship = Not-in-family , Sex = Male , Hours/w ≤ 40 } , the model predicts < 50 k because the sum of the weights in the 3 trees for this instance equals -0 . 4073 = ( -0 . 1089 -0 . 2404 -0 . 0580) < 0 .

2.2 MLModel Interpretability and Post-Hoc Explanations

Interpretability is generally accepted to be a subjective concept, without a formal definition [39]. One way to measure interpretability is in terms of the succinctness of information provided by an ML model to justify a given prediction. Recent years have witnessed an upsurge in the interest in devising and applying interpretable models in safety-critical applications [48, 58]. An alternative to interpretable models is post-hoc explanation of black-box models, which this paper focuses on.

Numerous methods to compute explanations have been proposed recently [46, 48]. The lion's share of these comprise what is called model-agnostic approaches to explainability [40, 55, 56] of heuristic nature that resort to extensive sampling in the vicinity of an instance being explained in order to 'estimate' the behavior of the classifier in this local vicinity of the instance. In this regard, they rely on estimating input data distribution by building on the information about the training data [34]. Depending on the form of explanations model-agnostic approaches offer, they are conventionally classified as feature selection or feature attribution approaches briefly discussed below.

Feature Selection. A feature selection approach identifies subsets of features that are deemed sufficient for a given prediction c = κ ( v ) . As mentioned above, the majority of feature selection approaches are model-agnostic with one prominent example being Anchors [56]. As such, the sufficiency of the selected set of features for a given prediction is determined statistically based on extensive sampling around the instance of interest, by assessing a few measures like fidelity , precision , among others. As a result, feature selection explanations given as a set of features ω ⊆ F should be interpreted as the conjunction ∧ i ∈ ω ( x i = v i ) deemed responsible for prediction c = κ ( v ) , v ∈ F , c ∈ K . Due to the statistical nature of these explainers, they are known to suffer from various explanation quality issues [24, 34, 63]. An additional line of work on formal explainability [25, 61] also tackles feature selection while offering guarantees of soundness; these are discussed below.

Feature Attribution. A different view on post-hoc explanations is provided by feature attribution approaches, e.g. LIME [55] and SHAP [40]. Based on random sampling in the neighborhood of the target instance, these approaches attribute responsibility to all model's features by assigning a numeric value w i ∈ R of importance to each feature i ∈ F . Given these importance values, the features can then be ranked from most important to least important. As a result, a feature attribution explanation is conventionally provided as a linear form ∑ i ∈F w i · x i , which can be also seen as approximating the original black-box explainer κ in the local neighborhood of instance v ∈ F . Among other feature attribution approaches, SHAP [5, 6, 40] is often claimed to stand out as it aims at approximating Shapley values, a powerful concept originating from cooperative games in game theory [60].

Formal Explainability. In this work, we build on formal explainability proposed in earlier work [8, 13, 25, 42, 61]. where explanations are equated with abductive explanations (AXp's). Abductive explanations are subset-minimal sets of features formally proved to suffice to explain an ML prediction given a formal representation of the classifier of interest. Concretely, given an instance v ∈ F and a prediction c = κ ( v ) , an AXp is a subset-minimal set of features X ⊆ F , such that

Abductive explanations are guaranteed to be subset-minimal sets of features proved to satisfy (1). As other feature selection explanations, they answer why a certain prediction was made. An alternate way to explain a model's behavior is to seek an answer why not another prediction was made, or, in other words, how to change the prediction. Explanations answering why not questions are referred to as contrastive explanations (CXp's) [26, 42, 46]. As in prior work, we define a CXp as a subset-minimal set of features that, if allowed to change their values, are necessary to change the prediction of the model. Formally, a CXp for prediction c = κ ( v ) is a subset-minimal set of features Y ⊆ F , such that

̸

Finally, recent work has shown that AXp's and CXp's for a given instance v ∈ F are related through the minimal hitting set duality [26, 54]. The duality implies that each AXp for a prediction c = κ ( v )

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

is a minimal hitting set 2 (MHS) of the set of all CXp's for that prediction, and the other way around: each CXp is an MHS of the set of all AXp's. The explanation enumeration algorithm [26] applied in this paper heavily relies on this duality relation and is inspired by the MARCO algorithm originating from the area of over-constrained systems [36, 37, 53]. A growing body of recent work on formal explanations is represented (but not limited) by [2, 4, 7, 9, 10, 14, 18, 20, 27, 41-44, 65].

Example 2. In the context of Example 1, feature attribution computed by LIME and SHAP as well as all 2 AXp's are shown in Figure 2. AXp X 1 indicates that specifying Education = Bachelors and Hours/w ≤ 40 guarantees that any compatible instance is classified as < 50k independent of the values of other features, e.g. Status and Relationship, since the maximal sum of weights is 0 . 0770 -0 . 0200 -0 . 0580 = -0 . 0010 < 0 as long as the feature values above are used. Observe that another AXp X 2 for v is {Education, Status}. Since both of the two AXp's for v consist of two features, it is difficult to judge which one is better without a formal feature importance assessment.

3 Why Formal Feature Attribution?

On the one hand, abductive explanations serve as a viable alternative to non-formal feature selection approaches because they (i) guarantee subset-minimality of the selected sets of features and (ii) are computed via formal reasoning over the behavior of the corresponding ML model. Having said that, they suffer from a few issues. First, observe that deciding the validity of (1) requires a formal reasoner to take into account the complete feature space F , assuming that the features are independent and uniformly distributed [65]. In other words, the reasoner has to check all the combinations of feature values, including those that never appear in practice . This makes AXp's being unnecessarily conservative (long), i.e. they may be hard for a human decision maker to interpret. Second, AXp's are not aimed at providing feature attribution. The abundance of various AXp's for a single data instance [25], e.g. see Example 2, exacerbates this issue as it becomes unclear for a user which of the AXp's to use to make an informed decision in a particular situation.

On the other hand, non-formal feature attribution in general is known to be susceptible to out-ofdistribution sampling [34, 62] while SHAP is shown to fail to effectively approximate Shapley values [21]. Moreover and quite surprisingly, [21] argued that even the use of exact Shapley values is inadequate as a measure of feature importance. Our results below confirm that both LIME and SHAP often fail to grasp the real feature attribution in a number of practical scenarios.

To address the above limitations, we propose the concept of formal feature attribution (FFA) as defined next. Let us denote the set of all formal abductive explanations for a prediction c = κ ( v ) by A κ ( v , c ) . Then formal feature attribution of a feature i ∈ F can be defined as the proportion of abductive explanations where it occurs. More formally,

Definition 1: (FFA). The formal feature attribution ffa κ ( i, ( v , c )) of a feature i ∈ F to an instance ( v , c ) for machine learning model κ is

Formal feature attribution has some nice properties. First, it has a strict and formal definition, i.e. we can, assuming we are able to compute the complete set of AXp's for an instance, exactly define it for all features i ∈ F . Second, it is fairly easy to explain to a user of the classification system, even if they are non-expert. Namely, it is the percentage of (formal abductive) explanations that make use of a particular feature i . Third, as we shall see later, even though we may not be able to compute all AXp's exhaustively, we can still get good approximations fast.

Example 3. Recall Example 2. As there are 2 AXp's for instance v , the prediction can be attributed to the 3 features with non-zero FFA shown in Figure 2d. Also, observe how both LIME and SHAP (see Figure 2a and Figure 2b) assign non-zero attribution to the feature Relationship, which is in fact irrelevant for the prediction, but overlook the highest importance of feature Education.

One criticism of the above definition is that it does not take into account the length of explanations where the feature arises. Arguably if a feature arises in many AXp's of size 2, it should be considered

̸

163

164

165

166

167

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

more important than a feature which arises in the same number of AXp's but where each is of size 10. An alternate definition, which tries to take this into account, is the weighted formal feature attribution (WFFA), i.e. the average proportion of AXp's that include feature i ∈ F . Formally,

Definition 2: (WFFA). The weighted formal feature attribution wffa κ ( i, ( v , c )) of a feature i ∈ F to an instance ( v , c ) for machine learning model κ is

Note that these attribution values are not on the same scale although they are convertible: 168

FFA can be related to the problem of feature relevancy [22], where a feature is said to be relevant if it belongs to at least one AXp. Indeed, feature i ∈ F is relevant for prediction c = κ ( v ) if and only if ffa κ ( i, ( v , c )) > 0 . As a result, the following claim can be made.

Proposition 1. Given a feature i ∈ F and a prediction c = κ ( v ) , deciding whether ffa κ ( i, ( v , c )) > ω , ω ∈ (0 , 1] , is at least as hard as deciding whether feature i is relevant for the prediction.

The above result indicates that computing exact FFA values may be expensive in practice. For example and in light of [22], one can conclude that the decision version of the problem is Σ P 2 -hard in the case of DNF classifiers.

Similarly and using the relation between FFA and feature relevancy above, we can note that the decision version of the problem is in Σ P 2 as long as deciding the validity of (1) is in NP, which in general is the case (unless the problem is simpler, e.g. for decision trees [28]). Namely, the following result is a simple consequence of the membership result for the feature relevance problem [22].

Proposition 2. Deciding whether ffa κ ( i, ( v , c )) > ω , ω ∈ (0 , 1] , is in Σ P 2 if deciding (1) is in NP .

4 Approximating Formal Feature Attribution

As the previous section argues and as our experimental results confirm, it may be challenging in practice to compute exact FFA values due to the general complexity of the problem. Although some ML models admit efficient formal encodings and reasoning procedures, effective principal methods for FFA approximation seem necessary. This section proposes one such method.

Normally, formal explanation enumeration is done by exploiting the MHS duality between AXp's and CXp's and the use of MARCO-like [37] algorithms aiming at efficient exploration of minimal hitting sets of either AXp's or CXp's [26, 36, 37, 53]. Depending on the target type of formal explanation, MARCO exhaustively enumerates all such explanations one by one, each time extracting a candidate minimal hitting set and checking if it is a desired explanation. If it is then it is recorded and blocked such that this candidate is never repeated again. Otherwise, a dual explanation is extracted from the subset of features complementary to the candidate [25], gets recorded and blocked so that it is hit by each future candidate. The procedure proceeds until no more hitting sets of the set of dual explanations can be extracted, which signifies that all target explanations are enumerated. Observe that while doing so, MARCO also enumerates all the dual explanations as a kind of 'side effect'.

One of the properties of MARCO used in our approximation approach is that it is an anytime algorithm, i.e. we can run it for as long as we need to get a sufficient number of explanations. This means we can stop it by using a timeout or upon collecting a certain number of explanations.

The main insight of FFA approximation is as follows. Recall that to compute FFA, we are interested in AXp enumeration. Although intuitively this suggests the use of MARCO targeting AXp's, for the sake of fast and high-quality FFA approximation, we propose to target CXp enumeration with AXp's as dual explanations computed 'unintentionally'. The reason for this is twofold: (i) we need to get a good FFA approximation as fast as we can and (ii) according to our practical observations, MARCO needs to amass a large number of dual explanations before it can start producing target explanations. This is because the hitting set enumerator is initially 'blind' and knows nothing about the features

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

Algorithm 1 MARCO-like Anytime Explanation Enumeration

̸

it should pay attention to - it uncovers this information gradually by collecting dual explanations to hit. This way a large number of dual explanations can quickly be enumerated during this initial phase of grasping the search space, essentially 'for free'. Our experimental results demonstrate the effectiveness of this strategy in terms of monotone convergence of approximate FFA to the exact FFA with the increase of the time limit. A high-level view of the version of MARCO used in our approach targeting CXp enumeration and amassing AXp's as dual explanations is shown in Algorithm 1.

5 Experimental Evidence

This section assesses the formal feature attribution for gradient boosted trees (BT) [12] on multiple widely used images and tabular datasets, and compares FFA with LIME and SHAP. In addition, it also demonstrates the use of FFA in a real-world scenario of Just-in-Time (JIT) defect prediction, which assists teams in prioritizing their limited resources on high-risk commits or pull requests [52].

Setup and Prototype Implementation. All experiments were performed on an Intel Xeon 8260 CPU running Ubuntu 20.04.2 LTS, with the memory limit of 8 GByte. A prototype of the approach implementing Algorithm 1 and thus producing FFA was developed as a set of Python scripts and builds on [27]. As the FFA and WFFA values turn out to be almost identical (subject to normalization) in our experiments, here we report only FFA. WFFA results can be found in supplementary material.

Datasets and Machine Learning Models. The well-known MNIST dataset [15, 50] of handwritten digits 0-9 is considered, with two concrete binary classification tasks created: 1 vs. 3 and 1 vs. 7. We also consider PneumoniaMNIST [67], a binary classification dataset to distinguish X-ray images of pneumonia from normal cases. To demonstrate extraction of exact FFA values for the above datasets, we also examine their downscaled versions, i.e. reduced from 28 × 28 × 1 to 10 × 10 × 1. We also consider 11 tabular datasets often applied in the area of ML explainability and fairness [3, 16, 17, 19, 49, 59]. All the considered datasets are randomly split into 80% training and and 20% test data. For images, 15 test instances are randomly selected in each test set for explanation while all tabular test instances are explained. For all datasets, gradient boosted trees (BTs) are trained by XGBoost [12], where each BT consists of 25 trees of depth 3 per class. 3 Finally, we show the use of FFA on 2 JIT defect prediction datasets [52], with 500 instances per dataset chosen for analysis.

5.1 Formal Feature Attribution

In this section, we restrict ourselves to examples where we can compute the exact FFA values for explanations by computing all AXp's. To compare with LIME and SHAP, we take their solutions, replace negative attributions by the positive counterpart (in a sense taking the absolute value) and then normalize the values into [ 0 , 1 ] . We then compare these approaches with the computed FFA values, which are also in [ 0 , 1 ] . The error is measured as Manhattan distance, i.e. the sum of absolute differences across all features. We also compare feature rankings according to the competitors (again using absolute values for LIME and SHAP) using Kendall's Tau [31] and rank-biased overlap (RBO) [66]

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

Hispanic: 0

Native_American: 0

African_American: 1

Female: 0

Misdemeanor: 1

score_factor: 1

Age_Above_FourtyFive: 0

Number_of_Priors: 3

Age_Below_TwentyFive: 1

Other: 0

Asian: 0

0.00

0.00

0.17

0.17

0.17

0.67

0.67

0.83

0.83

0.83

1.00

(a) FFA

Hispanic: 0

Native_American: 0

African_American: 1

Female: 0

Misdemeanor: 1

score_factor: 1

Age_Above_FourtyFive: 0

Number_of_Priors: 3

Age_Below_TwentyFive: 1

Other: 0

Asian: 0

0.02

0.02

0.21

0.13

0.05

0.20

0.82

0.83

0.82

0.13

1.00

(b) LIME

Hispanic: 0

Native_American: 0

African_American: 1

Female: 0

Misdemeanor: 1

score_factor: 1

Age_Above_FourtyFive: 0

Number_of_Priors: 3

Age_Below_TwentyFive: 1

Other: 0

Asian: 0

0.00

0.00

0.12

0.10

0.05

0.14

0.37

1.00

0.89

0.06

0.03

(c) SHAP

Figure 3: Explanations for an instance of Compas v = { #Priors = 3 , Score_factor = 1 , Age_Above_FourtyFive = 0 , Age_Below_TwentyFive = 1 , African_American = 1 , Asian = 0 , Hispanic = 0 , Native_American = 0 , Other = 0 , Female = 0 , Misdemeanor = 1 } predicted as Two_yr_Recidivism = true.

metrics. 4 Kendall's Tau and RBO are measured on a scale [ -1 , 1 ] and [ 0 , 1 ] , respectively. A higher value in both metrics indicates better agreement or closeness between a ranking and FFA.

Tabular Data. Figure 3 exemplifies a comparison of FFA, LIME and SHAP on an instance of the Compas dataset [3]. While FFA and LIME agree on the most important feature, 'Asian', SHAP gives it very little weight. Neither LIME nor SHAP agree with FFA, though there is clearly some similarity.

Table 1 details the comparison conducted on 11 tabular datasets, including adult , compas , and recidivism datasets commonly used in XAI. For each dataset, we calculate the metric for each individual instance and then average the outcomes to obtain the final result for that dataset. As can be observed, the errors of LIME's feature attribution across these datasets span from 1.39 to 5.13. SHAP demonstrates similar errors within a range [ 1.40 , 4.76 ] . LIME and SHAP also exhibit comparable performance in relation to the two ranking comparison metrics. The values of Kendall's Tau for LIME (resp. SHAP) are between -0.36 and 0.22 (resp. -0.39 and 0.27). Regarding the RBO values, LIME exhibits values between 0.39 and 0.68, whereas SHAP demonstrates values ranging from 0.44 to 0.67. Overall, as Table 1 indicates, both LIME and SHAP fail to get close enough to FFA.

10 × 10 Digits. We now compare the results on 10 × 10 downscaled MNIST digits and PneumoniaMNIST images, where it is feasible to compute all AXp's. Table 2 compares LIME's, SHAP's feature attribution and approximate FFA. Here, we run AXp enumeration for a number of seconds, which is denoted as FFA ∗ , ∗ ∈ R + . The runtime required for each image by LIME and SHAP is less than one second. The results show that the errors of our approximation are small, even after 10 seconds it beats both LIME and SHAP, and decreases as we generate more AXp's. The results for the orderings show again that after 10 seconds, FFA ∗ ordering gets closer to the exact FFA than both LIME and SHAP. Observe how LIME is particularly far away from the exact FFA ordering.

Summary. These results make us confident that we can get useful approximations to the exact FFA without exhaustively computing all AXp's while feature attribution determined by LIME and SHAP is quite erroneous and fails to provide a human-decision maker with useful insights, despite being fast.

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

2

(a) LIME

(b) SHAP

(c) FFA10

(d) FFA30

(e) FFA120

(f) FFA600

(g) FFA1.2k

(h) FFA3.6k

(i) FFA7.2k

5.2 Approximating Formal Feature Attribution

Since the problem of formal feature attribution 'lives' in Σ P 2 , it is not surprising that computing FFA may be challenging in practice. Table 2 suggests that our approach gets good FFA approximations even if we only collect AXp's for a short time. Here we compare the fidelity of our approach versus the approximate FFA computed after 2 hours (7200s). Figure 4, 5, and 6 depict feature attributions generated by LIME, SHAP and FFA ∗ for the three selected 28 × 28 images. The comparison between LIME, SHAP, and the approximate FFA computation is detailed in Table 3. The LIME and SHAP processing time for each image is less than one second. The average findings detailed in Table 3 are consistent with those shown in Table 2. Namely, FFA approximation yields better errors, Kendall's Tau and RBO values, outperforming both LIME, and SHAP after 10 seconds. Furthermore, the results demonstrate that after 10 seconds our approach places feature attributions closer to FFA7200 compared to both LIME and SHAP hinting on the features that are truly relevant for the prediction.

5.3 Application in Just-in-Time Defect Prediction

Just-in-Time (JIT) defect prediction [30, 32, 38, 51] has been recently proposed to predict if a commit will introduce software defects in the future, enabling development teams to prioritize their limited Software Quality Assurance resources on the most risky commits/pull requests. The approach of JIT

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

(a) LIME

(b) SHAP

(c) FFA10

(d) FFA30

(e) FFA120

(f) FFA600

(g) FFA1.2k

(h) FFA3.6k

(i) FFA7.2k

中

美

(a) LIME

(b) SHAP

(c) FFA10

(d) FFA30

(e) FFA120

(f) FFA600

(g) FFA1.2k

(h) FFA3.6k

(i) FFA7.2k

defect prediction has often been considered a black-box, lacking explainability for practitioners. To tackle this challenge, our proposed approach to generating FFA can be employed, as model-agnostic approaches cannot guarantee to provide accurate feature attribution (see above). We use logistic regression models of [52] based on large-scale open-source Openstack and Qt datasets provided by [45] commonly used for JIT defect prediction [52]. Monotonicity of logistic regression enables us to enumerate explanations using the approach of [44] and so to extract exact FFA for each instance within a second . Table 4 details the comparison of FFA, LIME and SHAP in terms of the three considered metrics. As with the outcomes presented in Table 1, Table 2, and Table 3, neither LIME nor SHAP align with formal feature attribution, though there are some similarities between them.

6 Limitations

Despite the rigorous guarantees provided by formal feature attribution and high-quality of the result explanations, the following limitations can be identified. First, our approach relies on formal reasoning and thus requires an ML model of interest to admit a representation in some fragments of first-order logic, and the corresponding reasoner to deal with it [42]. Second, the problem complexity impedes immediate and widespread use of FFA and signifies the need to develop effective methods of FFA approximation. Finally, though our experimental evidence suggests that FFA approximations quickly converge to the exact values of FFA, whether or not this holds in general remains an open question.

7 Conclusions

Most approaches to XAI are heuristic methods that are susceptible to unsoundness and out-ofdistribution sampling. Formal approaches to XAI have so far concentrated on the problem of feature selection, detecting which features are important for justifying a classification decision, and not on feature attribution, where we can understand the weight of a feature in making such a decision. In this paper we define the first formal approach to feature attribution (FFA) we are aware of, using the proportion of abductive explanations in which a feature occurs to weight its importance. We show that we can compute FFA exactly for many classification problems, and when we cannot we can compute effective approximations. Existing heuristic approaches to feature attribution do not agree with FFA. Sometimes they markedly differ, for example, assigning no weight to a feature that appears in (a large number of) explanations, or assigning (large) non-zero weight to a feature that is irrelevant for the prediction. Overall, the paper argues that if we agree that FFA is a correct measure of feature attribution then we need to investigate methods that compute good FFA approximations quickly.

References 313

ACM. Fathers of the deep learning revolution receive ACM A.M. Turing award. http: 314

//tiny.cc/9plzpz , 2018. 315

L. Amgoud and J. Ben-Naim. Axiomatic foundations of explainability. In L. D. Raedt, editor, 316 IJCAI , pages 636-642, 2022. 317

J. Angwin, J. Larson, S. Mattu, and L. Kirchner. Machine bias. http://tiny.cc/dd7mjz , 318 2016. 319

M. Arenas, D. Baez, P. Barceló, J. Pérez, and B. Subercaseaux. Foundations of symbolic 320 languages for model interpretability. In NeurIPS , 2021. 321

M. Arenas, P. Barceló, L. E. Bertossi, and M. Monet. The tractability of SHAP-score-based 322 explanations for classification over deterministic and decomposable Boolean circuits. In AAAI , 323 pages 6670-6678. AAAI Press, 2021. 324

M. Arenas, P. Barceló, L. E. Bertossi, and M. Monet. On the complexity of SHAP-score-based 325 explanations: Tractability via knowledge compilation and non-approximability results. CoRR , 326 abs/2104.08015, 2021. 327

M. Arenas, P. Barceló, M. A. R. Orth, and B. Subercaseaux. On computing probabilistic 328 explanations for decision trees. In NeurIPS , 2022. 329

G. Audemard, F. Koriche, and P. Marquis. On tractable XAI queries based on compiled 330 representations. In KR , pages 838-849, 2020. 331

G. Blanc, J. Lange, and L. Tan. Provably efficient, succinct, and precise explanations. In 332 NeurIPS , 2021. 333

R. Boumazouza, F. C. Alili, B. Mazure, and K. Tabia. ASTERYX: A model-Agnostic SaT-basEd 334 appRoach for sYmbolic and score-based eXplanations. In CIKM , pages 120-129, 2021. 335

L. Breiman. Random forests. Mach. Learn. , 45(1):5-32, 2001. 336

T. Chen and C. Guestrin. XGBoost: A scalable tree boosting system. In KDD , pages 785-794, 337 2016. 338

A. Darwiche and A. Hirth. On the reasons behind decisions. In ECAI , pages 712-720, 2020. 339

A. Darwiche and P. Marquis. On quantifying literals in Boolean logic and its applications to 340 explainable AI. J. Artif. Intell. Res. , 72:285-328, 2021. 341

L. Deng. The MNIST database of handwritten digit images for machine learning research. 342 IEEE Signal Processing Magazine , 29(6):141-142, 2012. 343

D. Dua and C. Graff. UCI machine learning repository, 2017. http://archive.ics.uci. 344 edu/ml . 345

FairML. Auditing black-box predictive models. http://tiny.cc/6e7mjz , 2016. 346

J. Ferreira, M. de Sousa Ribeiro, R. Gonçalves, and J. Leite. Looking inside the black-box: 347 Logic-based explanations for neural networks. In KR , page 432-442, 2022. 348

S. Friedler, C. Scheidegger, and S. Venkatasubramanian. On algorithmic fairness, discrimination 349 and disparate impact. http://fairness.haverford.edu/ , 2015. 350

N. Gorji and S. Rubin. Sufficient reasons for classifier decisions in the presence of domain 351 constraints. In AAAI , pages 5660-5667, 2022. 352

X. Huang and J. Marques-Silva. The inadequacy of Shapley values for explainability. CoRR , 353 abs/2302.08160, 2023. 354

X. Huang, M. C. Cooper, A. Morgado, J. Planes, and J. Marques-Silva. Feature necessity & 355 relevancy in ML classifier explanations. In TACAS (1) , pages 167-186, 2023. 356

L. Hyafil and R. L. Rivest. Constructing optimal binary decision trees is NP-complete. Inf. Pro357 cess. Lett. , 5(1):15-17, 1976. URL https://doi.org/10.1016/0020-0190(76)90095-8 . 358

A. Ignatiev. Towards trustable explainable AI. In IJCAI , pages 5154-5158, 2020. 359

A. Ignatiev, N. Narodytska, and J. Marques-Silva. Abduction-based explanations for machine 360 learning models. In AAAI , pages 1511-1519, 2019. 361

A. Ignatiev, N. Narodytska, N. Asher, and J. Marques-Silva. From contrastive to abductive 362 explanations and back again. In AI*IA , pages 335-355, 2020. 363

A. Ignatiev, Y . Izza, P. J. Stuckey, and J. Marques-Silva. Using MaxSAT for efficient explanations 364 of tree ensembles. In AAAI , pages 3776-3785, 2022. 365

Y. Izza, A. Ignatiev, and J. Marques-Silva. On tackling explanation redundancy in decision trees. 366 J. Artif. Intell. Res. , 75:261-321, 2022. URL https://doi.org/10.1613/jair.1.13575 . 367

M. I. Jordan and T. M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science , 368 349(6245):255-260, 2015. 369

Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi. A Large370 Scale Empirical Study of Just-In-Time Quality Assurance. IEEE Transactions on Software 371 Engineering (TSE) , 39(6):757-773, 2013. 372

M. G. Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938. 373

S. Kim, T. Zimmermann, E. J. Whitehead Jr, and A. Zeller. Predicting Faults from Cached 374 History. In ICSE , pages 489-498, 2007. 375

R. Kohavi. Scaling up the accuracy of naive-Bayes classifiers: A decision-tree hybrid. In KDD , 376 pages 202-207, 1996. 377

378

379

H. Lakkaraju and O. Bastani. "How do I fool you?": Manipulating user trust via misleading black box explanations. In AIES , pages 79-85, 2020.

380

381

382

383

384

385

386

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature , 521(7553):436, 2015.

M. H. Liffiton and A. Malik. Enumerating infeasibility: Finding multiple MUSes quickly. In CPAIOR , pages 160-175, 2013.

M. H. Liffiton, A. Previti, A. Malik, and J. Marques-Silva. Fast, flexible MUS enumeration. Constraints An Int. J. , 21(2):223-250, 2016.

D. Lin, C. Tantithamthavorn, and A. E. Hassan. The impact of data merging on the interpretation of cross-project just-in-time defect models. IEEE Transactions on Software Engineering , 2021.

Z. C. Lipton. The mythos of model interpretability. Commun. ACM , 61(10):36-43, 2018. 387

S. M. Lundberg and S. Lee. A unified approach to interpreting model predictions. In NeurIPS , 388 pages 4765-4774, 2017. 389

E. L. Malfa, R. Michelmore, A. M. Zbrzezny, N. Paoletti, and M. Kwiatkowska. On guaranteed 390 optimal robust explanations for NLP models. In IJCAI , pages 2658-2665, 2021. 391

J. Marques-Silva and A. Ignatiev. Delivering trustworthy AI through formal XAI. In AAAI , 392 pages 12342-12350. AAAI Press, 2022. 393

J. Marques-Silva, T. Gerspacher, M. C. Cooper, A. Ignatiev, and N. Narodytska. Explaining 394 naive Bayes and other linear classifiers with polynomial time and delay. In NeurIPS , 2020. 395

J. Marques-Silva, T. Gerspacher, M. C. Cooper, A. Ignatiev, and N. Narodytska. Explanations 396 for monotonic classifiers. In ICML , pages 7469-7479, 2021. 397

S. McIntosh and Y. Kamei. Are fix-inducing changes a moving target? A longitudinal case 398 study of Just-in-Time defect prediction. IEEE Transactions on Software Engineering (TSE) , 399 pages 412-428, 2017. 400

T. Miller. Explanation in artificial intelligence: Insights from the social sciences. Artif. Intell. , 401 267:1-38, 2019. 402

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, 403 M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein404 forcement learning. Nature , 518(7540):529, 2015. 405

C. Molnar. Interpretable Machine Learning . Leanpub, 2020. http://tiny.cc/6c76tz . 406

R. S. Olson, W. G. L. Cava, P. Orzechowski, R. J. Urbanowicz, and J. H. Moore. PMLB: a 407 large benchmark suite for machine learning evaluation and comparison. BioData Min. , 10(1): 408 36:1-36:13, 2017. 409

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, 410 N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, 411 S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTorch: An imperative style, 412 high-performance deep learning library. In NeurIPS , pages 8024-8035, 2019. 413

C. Pornprasit and C. Tantithamthavorn. JITLine: A Simpler, Better, Faster, Finer-grained 414 Just-In-Time Defect Prediction. In MSR , pages 369-379, 2021. 415

C. Pornprasit, C. Tantithamthavorn, J. Jiarpakdee, M. Fu, and P. Thongtanunam. PyExplainer: 416 Explaining the predictions of Just-In-Time defect models. In ASE , pages 407-418, 2021. 417

A. Previti and J. Marques-Silva. Partial MUS enumeration. In AAAI . AAAI Press, 2013. 418

R. Reiter. A theory of diagnosis from first principles. Artif. Intell. , 32(1):57-95, 1987. 419

M. T. Ribeiro, S. Singh, and C. Guestrin. "Why should I trust you?": Explaining the predictions 420 of any classifier. In KDD , pages 1135-1144, 2016. 421

M. T. Ribeiro, S. Singh, and C. Guestrin. Anchors: High-precision model-agnostic explanations. 422 In AAAI , pages 1527-1535, 2018. 423

R. L. Rivest. Learning decision lists. Mach. Learn. , 2(3):229-246, 1987. 424

425

426

C. Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. , 1(5):206-215, 2019.

427

428

429

430

431

432

433

434

P. Schmidt and A. D. Witte. Predicting recidivism in North Carolina, 1978 and 1980. InterUniversity Consortium for Political and Social Research , 1988.

L. S. Shapley. A value of n -person games. Contributions to the Theory of Games , 2(28): 307-317, 1953.

A. Shih, A. Choi, and A. Darwiche. A symbolic approach to explaining Bayesian network classifiers. In IJCAI , pages 5103-5111, 2018.

D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju. Fooling LIME and SHAP: adversarial attacks on post hoc explanation methods. In AIES , pages 180-186, 2020.

D. Slack, A. Hilgard, S. Singh, and H. Lakkaraju. Reliable post hoc explanations: Modeling 435 uncertainty in explainability. In NeurIPS , pages 9391-9404, 2021. 436

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. 437 Intriguing properties of neural networks. In ICLR (Poster) , 2014. 438

S. Wäldchen, J. MacDonald, S. Hauch, and G. Kutyniok. The computational complexity of 439 understanding binary classifier decisions. J. Artif. Intell. Res. , 70:351-387, 2021. 440

W. Webber, A. Moffat, and J. Zobel. A similarity measure for indefinite rankings. ACM 441 Transactions on Information Systems (TOIS) , 28(4):1-38, 2010. 442

J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni. MedMNIST v2-a 443 large-scale lightweight benchmark for 2D and 3D biomedical image classification. Scientific 444 Data , 10(1):41, 2023. 445

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes and studies a notion of feature attribution in which features are scored for a given instance according to the proportion of minimal explanations for that instance in which they participate. Although an exact computation of this scores can be computationally unfeasible, the paper exploits the minimal-hitting-set duality between abductive and contrastive explanations to approximate them efficiently. Finally, they study their approach empirically over several datasets.\n\nSTRENGTHS:\n- The paper is really well written and easy to follow; presentation is excellent\n- The paper does a good job at showcasing the importance of the problem and providing references to issues that are present in SHAP \n- The proposed metric is simple and seems promising\n- The experiments have good results and are presented nicely\n- The sections on limitations and conclusions provide a nice and helpful starting point for further discussion\n- References seem appropriate and detailed\n\nWEAKNESSES:\n- The paper does not provide proofs in the supplementary material for its propositions. While proposition 1 is self-explanatory, proposition 2 is not and should be accompanied with a proof.\n\n- Despite mentioning the weighted variant (Definition 2),  the paper doesn't seem to do anything with it; there are no theoretical nor practical results about it as far as I can see, and it is only discussed in the appendix. I understand that due to the page limit not everything can fit in the paper, but including the definition of WFFA without doing anything with it in the paper seems like a poor choice to me.\n\n- Even though the paper is about formal explainability, and the scores themselves are formally defined, the approximation algorithm doesn't seem to have any formal guarantee, and thus it is unclear to me what the interpretation of the results should be. It seems to me that further theoretical studies are required; how do we know that their approach can't fall into cases where it gives answers that are arbitrarily far from the ground truth? The starting point of the paper is about how methods such as SHAP have pitfalls when certain conditions are met, and yet it is not clear at all that this approach is exempt of similar (or even worse) potential problems.\n\n- The experimental data is, unless I am missing something, a bit strange; LIME, SHAP and their FFA approximation compute different things, both in theory and practice, and thus I don't understand at all what the meaning of their comparison is; LIME and SHAP are not approximations to the FFA score, and thus it doesn't feel sound or fair to use them as such. Their idea of taking absolute values and normalizing might make sense, but this is far from obvious and limits how convincing their results are.\n\nQUESTIONS:\nAlthough I don't have particular questions to the authors,  my perspective (and therefore scores) could change by them convincingly addressing the points raised in the weaknesses section above.\n\nLIMITATIONS:\nYes; there doesn't seem to be much to address and the authors discuss appropriate points in section 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to XAI called Formal Feature Attribution. The authors address the limitations of existing model-agnostic methods and formal XAI approaches by proposing FFA as a solution for feature attribution.\nThe FFA method leverages formal explanation enumeration to define feature attribution as the proportion of explanations in which a specific feature occurs. The paper highlights the challenges in computing exact FFA but presents an efficient approximation technique using a dual trait of such explanations.\n\nSTRENGTHS:\nThe paper exhibits a high level of clarity and coherence in its writing style, making it easily understandable for readers. The claims and contributions are clearly stated, enabling the reader to grasp the main objectives of the research.\n\nThe paper makes a noteworthy contribution to the field of Explainable AI by introducing a novel approach. This fills a gap in the existing XAI landscape, where formal foundations are relatively loose, and offers a promising solution for feature attribution.\n\nThe authors demonstrate extensive work in various aspects of the research. They have invested effort in both the theoretical aspects, establishing formal foundations for FFA, as well as in the practical aspects, conducting experiments to validate the proposed approach. This comprehensive approach enhances the credibility and reliability of the findings.\n\nWEAKNESSES:\nAxiomatic Analysis of FFA: While Lime and Shap have established a set of axioms for explanations, it remains unclear which set of axioms the FFA explanations adhere to. It would be beneficial to explore and define the axioms underlying FFA explanations or engage in theoretical debates surrounding them. For example, I think that duplicate features increase the importance of other features that exists in a common AX'p. Using a specific example, think of a scenario where two correlated features, f1 and f2, are considered. In this case, a decision tree is constructed where f1 appears at a certain node while f2 is absent. Now, create a separate decision tree that includes the condition for either f1 or f2 to appear at that node, resulting in identical predictions due to their correlation. However, despite the consistent predictions, I'm pretty sure that the two models provide different explanations for the features, questioning whether the ratio of feature occurrence in formal explanations increases equally in the numerator and denominator. \n\nApproximation Guarantees for FFA: Since FFA is an NP problem, it necessitates further consideration regarding the guarantees provided by approximation techniques. Investigating the quality and limitations of these approximations would strengthen the practical utility of FFA.\n\nThe absence of code implementation limits the reproducibility and practical adoption of the proposed FFA approach.\n\nAddressing these aspects would further enhance the theoretical and practical implications of FFA within the field of Explainable AI.\n\nQUESTIONS:\nIsn't the approach of finding a minimal subset of features for which the classification remains the same too strict? \nCould a probabilistic approach potentially be more suitable, enabling the smoothing of results and accommodating variations in feature importance?\n\nLIMITATIONS:\nDistinguishing Out-of-Distribution and Manifold Sampling: The paper does not explicitly discuss the distinctions between out-of-distribution sampling and manifold sampling, despite their potential significance in this context. Exploring the differences and implications of these sampling techniques would contribute to a more comprehensive understanding of FFA.\n\nAddressing Local Explanations: The paper briefly touches on the issue of local explanations, but does not delve into their meanings, differences, or implications. Future research could focus on formulating FFA specifically for global explanations, shedding light on the disparities between local and global explanations within the context of FFA.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose formal feature attributions, a novel type of local feature attribution method for\nexplaining the predictions of black box models.  Their approach builds on the notion of abductive explanations (AXp's), a type of minimal sufficient subsets.  One issue with AXps is that there are a potentially exponential number of them, most of which fall outside of the data distribution.  The authors propose - essentially - to summarize the set of AXps via averaging into a per-feature relevance score akin to those provided by LIME and SHAP.  An inverse proposeity weighted variant is also introduced.  The authors then show that typically computing FFAs is computationally intractable\nand propose an approximation algorithm for quickly estimating approximate FFAs.  This approximation is shown to work well on two MNIST-like data sets.\n\n**Post-rebuttal update**: increased score, see the discussion.\n\nSTRENGTHS:\n+ Very clearly written, a pleasure to read.\n+ Ideas are clearly presented, with a couple of exceptions (see questions below).\n+ Related work is well done.\n+ FFAs are rooted on a simple and clear concept.\n+ Algorithm is sensible.\n+ Good empirical performance on a relatively varied set of data sets (for boosted DTs only).\n+ Some essential limitations are clearly discussed, but see below.\n\nWEAKNESSES:\n- The authors seem to assume FFAs are the \"real feature attribution\", but provide no real motivation for this (see Q1, this is the big one).\n- Unclear reasons why OOD sampling is an issue and how FFAs deal with it (Q2).\n- Missing discussion of information loss due to averaging over AXp's (Q3).\n- Experiments consider boosted DTs only (relatively minor).\n- Missing evaluation of approximate FFA algorithm on non-MNIST data (minor).\n\nQUESTIONS:\nQ1. I am puzzled by the statement in line 144 that \"LIME and SHAP often fail to grasp the real feature attribution in a number of practical scenarios\", and by the construction of the first experiment.  Here the authors compare LIME and SHAP against exact FFA relevance values, using the term \"error\" to indicate the difference between relevances output by the former and the latter.  It seems to me the authors automatically assume that Eq. 3 is the \"real feature attribution\", but it is not clear why this would be the case.  LIME, SHAP, and FFA measure *different things* with *different semantics* and *different properties*.  I'd like to understand why FFAs (which measure the chance that a certain feature occurs in the set of AXp's) are taken to be - essentially - *the* gold standard.  I could not find solid motivation for this in the text.  The authors write that FFAs are \"[...] the percentage of (formal abductive) explanations that make use of a particular feature i\", which I do agree with.  What I miss is a link between this and believing that FFAs are \"better\" in all respects than LIME or SHAP (two approaches that, honestly, I am not a fan of).\n\nQ2. The authors stress the issue of \"out-of-distribution sampling\".  They mention this is an issue for AXp's because it yields complex explanations, hindering human understanding.  They also mention it is an issue for other, less \"formal\" attribution techniques like LIME and SHAP, but they don't explain why it is a problem.  My take is whathever the impact is on LIME and SHAP, it is not related to explanation complexity or length.  Then the authors explain that FFAs deal with explanation complexity, which I do agree with as FFAs compress the full set of AXp's into a fixed-size relevance vector.  However, I do not see how FFAs deal with the *other*, unnamed problem affecting LIME and SHAP.  TL;DR: what's the issue with OOD sampling and LIME and SHAP, and why wouldn't the same issue affect AXp's and/or FFAs?\n\nQ3. My understanding is that each AXp is strongly determined by feature interactions.  Taking the average over AXp's entirely throws away this information.  I think this is an important limitation - not only of FFAs, but rather of all feature attribution techniques, chiefly due to their simple vector format - that should be openly discussed in the paper.  I am confident the authors are keenly aware of this issue, and I'd appreciate if they made the readers aware of it too.  To turn this into a question: would you agree that FFAs are by their own nature less expressive than (the set of) AXp's?\n\nI am more than willing to increase my scores once the authors address these questions.\n\nLIMITATIONS:\nThe paper has an explicit limitations section which is quite well done.  I have outlined a couple of other possible limitations in my questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new approach called formal feature attribution (FFA), inspired by successful FXAI methods, to compute feature attribution scores. FFA is defined as the proportion of explanations where a feature occurs. Experiments try to demonstrate the effectiveness of FFA compared to SHAP and LIME under several public datasets.\n\nSTRENGTHS:\n1. The authors proposed a new perspective on providing feature attributions. They did point out some limitations of existing popular methods.\n2. The proposed FFA is straightforward and with clear motivations.\n\nWEAKNESSES:\n1. Important questions regarding the reasonableness of the proposed method are unanswered. For instance, the three advantage properties of FFA claimed in this paper are not well reasoned. \n2. The experiments provided in this work are hard to support FFA is better than other existing XAI scoring methods. \n3. Several definitions and annotations throughout the text lack proper illustration or clarification, which can hinder the reader's understanding. Without clear explanations, it becomes challenging to grasp the intended meanings and implications of these terms and annotations.\n\nQUESTIONS:\n1. The authors claim that FFA gains three nice properties. However, the descriptions of those properties are too vague to understand. The authors are highly encouraged to explain more details about those points. Some example concerns are illustrated as follows. The first advantage is that FFA has a strict and formal definition. However, the author fails to explain why the Shapley Value lacks a strict and formal definition, despite it being commonly regarded as the ground truth value in feature attributions. I am unable to discern the distinction between the Shapley Value and FFA in terms of a strict and formal definition, as well as why the Shapley Value cannot provide formal feature attributions. Additionally, claiming that percentage-wise feature attributions are more user-friendly without conducting user experiment studies or related experiments is unsubstantiated. Insufficient case studies on image data are inadequate to provide support.\n2. There are a lot of advancements discussing the issues of out-of-distribution sampling. It would be better to see the comparison between these series of methods, making the proposed approach more convincing, as FFA is claimed to address the limitations of out-of-distribution sampling (see line 140 to 148).\n3. LIME and SHAP are two classical baselines to provide a model explanation but obtain inferior performance on image datasets. It is highly encouraged the authors compare with some SOTA Shapley-based methods on image data, such as DeepSHAP[1], FastSHAP[2], or CoRTX[3]. \n4. As for the experimental results in Figure 3, it is very limited for authors to use only one example to reveal that FFA is better than LIME and SHAP. Furthermore, Table 1 only demonstrates LIME and SHAP fail to get close enough to FFA, which is expected as neither of them is designed to approximate FFA. However, the conclusion in Table 1 does not mean that FFA is a better indicator than LIME, SHAP, or Shapley Values in providing a formal explanation. I did not see any direct evidence to support this point. \n\n[1] Delivering Trustworthy AI through Formal XAI. Marques-Silva et al. AAAI 2022.\n[2] A unified approach to interpreting model predictions. Lundberg et al. NeurIPS 2017.\n[3] FastSHAP: Real-Time Shapley Value Estimation. Neil et al. ICLR 2022.\n[4] CoRTX: Contrastive Framework for Real-time Explanation. Chuang et al. ICLR 2023.\n\nLIMITATIONS:\n1. The proposed FFA only focuses on the classification task, which makes it limited to be applied in several other common tasks, such as the regression task. \n2. The paper lacks clear illustrations of experiment settings and does not compare with relevant advancements, leading to unconvincing results for verifying the proposed hypothesis. More detailed descriptions of the experiment settings and comparative analyses with related advancements are necessary to strengthen the study's validity.\n3. Several annotations and definitions are not clearly illustrated. For example, the “right arrow” in Equation 1 and the definition of “formal explanation” is not well-illustrated in this work. This makes the work hard to follow.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}