Title: SHAP-IQ: Unified Approximation of any-order Shapley Interactions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Explainable Artificial Intelligence

Feature Interaction

Shapley Interaction

Shapley Value
Abstract: Predominately in explainable artificial intelligence (XAI) research, the Shapley value (SV) is applied to determine feature attributions for any black box model. Shapley interaction indices extend the SV to define any-order feature interactions. Defining a unique Shapley interaction index is an open research question and, so far, three definitions have been proposed, which differ by their choice of axioms. Moreover, each definition requires a specific approximation technique. Here, we propose SHAPley Interaction Quantification (SHAP-IQ), an efficient sampling-based approximator to compute Shapley interactions for arbitrary cardinal interaction indices (CII), i.e. interaction indices that satisfy the linearity, symmetry and dummy axiom. SHAP-IQ is based on a novel representation and, in contrast to existing methods, we provide theoretical guarantees for its approximation quality, as well as estimates for the variance of the point estimates. For the special case of SV, our approach reveals a novel representation of the SV and corresponds to Unbiased KernelSHAP with a greatly simplified calculation. We illustrate the computational efficiency and effectiveness by explaining language, image classification and high-dimensional synthetic models.

SHAP-IQ: Unified Approximation of any-order Shapley Interactions

Abstract

Predominately in explainable artificial intelligence (XAI) research, the Shapley value (SV) is applied to determine feature attributions for any black box model. Shapley interaction indices extend the SV to define any-order feature interactions. Defining a unique Shapley interaction index is an open research question and, so far, three definitions have been proposed, which differ by their choice of axioms. Moreover, each definition requires a specific approximation technique. Here, we propose SHAPley Interaction Quantification (SHAP-IQ), an efficient sampling-based approximator to compute Shapley interactions for arbitrary cardinal interaction indices (CII), i.e. interaction indices that satisfy the linearity, symmetry and dummy axiom. SHAP-IQ is based on a novel representation and, in contrast to existing methods, we provide theoretical guarantees for its approximation quality, as well as estimates for the variance of the point estimates. For the special case of SV, our approach reveals a novel representation of the SV and corresponds to Unbiased KernelSHAP with a greatly simplified calculation. We illustrate the computational efficiency and effectiveness by explaining language, image classification and high-dimensional synthetic models.

1 Introduction

Feature attributions are a prevalent approach to interpret black box machine learning (ML) models [2, 7, 26]. However, in many real-world applications, such as understanding drug-drug interactions, mutational events or complex language models, quantifying interactions between features is essential, too [48, 23, 43]. Feature interactions provide a more comprehensive explanation, which can be seen as an enrichment of feature attributions [3, 39, 40]. While feature attributions quantify the contribution of single features to the model's prediction or performance, feature interactions quantify the contribution of a group of features to the model's prediction or performance.

I have never forgot this movie. All these years and it has remained in my life.

movie review excerpt (input)

interactions

never forgot

never

this

forgot this

never forgot this

(order 2)

(order 2)

(order 2)

(order 3)

(order 1)

never forgot this

sentiment (output)

+ 0.990

(positive)

interaction scores

- 0.080

+ 1.993

- 0.924

+ 0.412

+ 0.85 - 1.30 + 0.17

In this work, we are interested in feature interactions that make use of the Shapley value (SV) and its extension to Shapley interactions. The SV is a concept from cooperative game theory that has been used, apart from feature attributions [7], as a basis for many Shapley-based explanations [21, 15, 49]. It distinguishes itself through uniqueness given a set of intuitive axioms. A number of approaches extend Shapley-based explanations to feature interactions [17, 3, 39, 40]. Yet, in contrast to the SV, a 'natural' extension of the intuitive set of axioms for a unique Shapley interaction index is less clear. Moreover, its efficient computation is challenging and, so far, approximation approaches are specifically tailored to the particular definition.

In this paper, we consider a more general class of interaction indices, known as cardinal interaction indices (CII) [17], which covers all currently proposed definitions and all other that satisfy the (generalized) linearity, symmetry and dummy axiom. We present SHAPley Interaction Quantification (SHAP-IQ), a sampling-based unified approximation method. It is substantiated by mathematical guarantees and can be applied to any CII to approximate any-order interaction scores efficiently.

Contribution. Our main contributions include:

Weconsider a general form of interaction indices, known as CII (Definition 3.3) and establish a novel representation (Theorem 4.1), which we utilize to construct SHAP-IQ (Definition 4.2), an efficient sampling-based estimator. 2

We show that SHAP-IQ is unbiased, consistent and provide a general approximation bound (Theorem 4.3). We further prove that SHAP-IQ maintains the efficiency condition for n-Shapley Values [3] and the Shapley Taylor Interaction Index [39] (Theorem 4.7).

For the SV, we find a novel representation (Theorem 4.4). We further prove that SHAP-IQ is linked to Unbiased KernelSHAP [8] (Theorem 4.5) and greatly simplifies its representation.

We use SHAP-IQ to compute any-order n-Shapley Values on different ML models and demonstrate that it outperforms existing baseline methods. We further contrast different existing CIIs and compare SHAP-IQ to the corresponding baseline approximation method.

2 Related Work

The Shapley Interaction Index (SII) [17], its efficiency preserving aggregation as n-Shapley Values (n-SII) [3], the Shapley Taylor Interaction (STI) [39] and the Faithful Shapley Interaction Index (FSI) [40] offer different ways of extending the SV to interactions, which extend on the linearity, symmetry and dummy axiom to provide a uniquely defined interaction index. SII and STI extend on axiomatic properties of the weighted sum for SV [35], whereas FSI extends on the axiomatic properties of the Shapley interaction as the solution to a weighted least square solution [32, 33].

In the field of cooperative game theory, interactions have also been studied from a theoretical perspective as the solution of a weighted least square problem and the sum of marginal contributions with constant weights, which both yield a generalized Banzhaf value [18, 16].

In the ML community, interactions of features have been studied from a practical perspective for text [29] and image [41] data, for specific models, such as neural networks [42, 10, 36, 20] or tree based models [25]. Other concepts of interactions have been discussed in [43] using marginal contributions, from a statistical perspective with functional decomposition [28] and improved white box models with interaction terms [24].

To approximate SII and STI, a permutation-based method [39, 40], as an extension of ApproShapley [5], was suggested, whereas FSI relies on a kernel-based approximation, similar to KernelSHAP [26], which utilizes a representation of the SV as the solution of a weighted least square problem [6]. Unlike these specific approaches, we consider general CIIs, which subsume all of the above mentioned measures, and we propose a generic approximation technique, which can be accompanied by mathematical guarantees. In case of the SV, our approximation of the CII is related to Unbiased KernelSHAP [8], which is a variant of KernelSHAP [26]. It is further related to stratified sampling approximations for the SV and our sampling approach can be seen as flexible framework to find the optimum allocation for each stratum [4].

3 The Cardinal Interaction Index (CII) and Shapley-based Explanations

In this section, we review Shapley-based explanations and introduce the CII, which we aim to approximate in Section 4. We further introduce existing baseline methods for specific CIIs and Unbiased KernelSHAP for the SV, which is linked to our proposed method.

̸

Notations. We refer to the model behavior on a set of features D = { 1 , . . . , d } as a function ν : P ( D ) → R , where P ( D ) refers to the power set of D . We denote ν 0 ( T ) := ν ( T ) -ν ( ∅ ) , which is the default setting in game theory and also known as set function [17, 14]. The subsets S ⊆ D refer to the set of features (or players in game theory) of which the interaction is computed, where we use lower case letters for the cardinality, i.e. s := | S | . The maximum order interaction of interest is denoted with s 0 and we use the set T k := { T ⊆ D : k ≤ t ≤ d -k } and the set of interactions S s 0 := { S ⊆ D | s ≤ s 0 , S = ∅} . For a subset T ⊆ D , we refer to the binary representation as Z T = ( z 1 , . . . , z d ) ∈ { 0 , 1 } d with z i = 1 ( i ∈ T ) for i = 1 , . . . , d , where 1 refers to the indicator function. We further denote the Shapley kernel [6, 26] as µ ( t ) := 1 d -1 ( d -2 t -1 ) -1 .

Removal-based explanations [9] consider a model that is trained on d features, where the goal is to examine a model behavior that is defined on a subset of features. The model behavior ν for a subset of features could, for instance, be a particular model prediction for one input (local explanation) or an overall measure of model performance (global explanation), if only this subset of features is known [9]. To quantify the contribution for individual features, the change in model behavior is evaluated, if the feature is removed from the model. To restrict a ML model on a subset of features, different feature removal techniques have been proposed, such as marginalization of features or retraining the model [9]. To quantify the impact of a single feature i ∈ D on the model behavior ν it is then intuitive to compute the difference δ ν { i } ( T ) = ν ( T ∪ { i } ) -ν ( T ) for subsets T ⊆ D \ { i } . For a distinct pair of features ( i, j ) with i, j ∈ D , a natural extension is δ ν { i,j } ( T ) = ν ( T ∪ { i, j } ) -ν ( T ) -δ ν { i } ( T ) -δ ν { j } ( T ) for T ∈ D \ { i, j } , i.e. subtracting the contribution of single features from the joint impact of both features. The following definition generalizes this recursion and is known as discrete derivative or S -derivative [14].

Definition 3.1 (Discrete Derivative [14]) . For S ⊆ D the S-derivative of ν at T ⊆ D \ S is

To obtain an attribution score, the marginal contributions on different subsets T ⊆ D \ S are aggregated using a specific summary technique . In this work, we are interested in the approximation and extension of one particular summary technique for single features i ∈ D , called the Shapley value [35], independent of model behavior and feature removal.

Definition 3.2 (Shapley Value (SV) [35]) . The SV is I SV ( i ) = ∑ T ⊆ D \{ i } ( d -t -1)! t ! d ! δ ν { i } ( T ) , i ∈ D .

The SV is the unique attribution method that fulfills the axioms: symmetry (attributions are independent of feature ordering), linearity (in terms of the model behavior ν ), dummy (if a feature does not change ν then its attribution is zero) and efficiency (the sum of attributions are equal to ν 0 ( D ) ) [35]. However, the SV does not give any information about the interactions between two or more features. A suitable extension of the SV for interactions of features S ⊆ D remains an open question, as different axiomatic extensions have been proposed [17, 39, 40]. In this work, we thus consider a broad class of interaction indices, known as CIIs [17, 40], which subsumes popular choices.

Definition 3.3 (Cardinal Interaction Index (CII) [17]) . A CII is an interaction index of the form

Remark 3.4. It was shown that every interaction index satisfying the generalized linearity, symmetry and dummy axioms can be represented as a CII [17]. If ∑ d -s t =0 ( d -s t ) m s ( t ) = 1 , then the CII is also referred to as a cardinal-probabilistic interaction index (CPII) [14].

In this paper, we present a unified approximation technique for arbitrary CIIs.

3.1 Shapley Interaction Index (SII) and other CIIs

In the following, we introduce prominent examples of CIIs. For further details on the axioms and exact definitions, we refer to the appendix. The SII [17] is a direct extension of the SV , that relies on an additional recursive axiom to obtain a unique CII.

Definition 3.5 (Shapley Interaction Index (SII)[17]) . The SII is a CII defined as

It has been shown that the SII is a CPII [14]. In contrast to the SV , the SII does not fulfill the efficiency axiom, which is a desirable property in the context of ML. Therefore an extension of SII, as well as other interaction indices have been proposed.

n-Shapley Values (n-SII) and other interaction indices. The efficiency axiom for interaction indices of maximum interaction order 1 ≤ s 0 ≤ d requires that the sum of I m ( S ) up to order s 0 equals ν 0 ( D ) .

Definition 3.6 (Efficiency [39, 40]) . A CII is efficient of order s 0 , if ∑ S ∈S s 0 I m ( S ) = ν 0 ( D ) , where S s 0 is the set of interactions up to order s 0 .

In [3], an aggregation of SII was proposed to obtain n-SII I n-SII s 0 ( S ) of order s 0 that satisfies efficiency. Other axiomatic approaches directly require efficiency together with the linearity, symmetry and dummy axioms, and omit the recursive axiom of SII. However, in contrast to the SV, this axiom alone does not yield a unique interaction index [39, 40]. The STI [39] requires the efficiency axiom and an additional interaction distribution axiom. On the other hand, the FSI [40] requires the efficiency axiom and the faithfulness property, that relates the interaction index to a solution of a constrained weighted least square problem. The choice of axioms of SII (n-SII), STI and FSI yield a unique interaction index that reduces to the SVs for s 0 = 1 . For FSI, it was shown that the top-order interactions define a CPII [40, Proposition 21], which is also easily verified for STI. All orders of interactions of FSI and STI can in general be represented as a CII, as they fulfill the linearity, symmetry and dummy axioms [17, Proposition 5]. However, it was noted that for FSI a simple closed-form solution for lower-order interactions in terms of discrete derivatives remains unclear [40, Lemma 70].

3.2 Baseline Approximations of SII, STI and FSI.

By definition, the number of evaluations of ν in I , which constitutes the limiting factor in ML, grows exponentially with d and thus, in practice, approximation methods are required. Currently, there does not exist an approximation for the general CII definition, as each index (SII, STI, FSI) requires a specifically tailored technique. Approximations of CII can be distinguished into permutation-based approximation (SII and STI) and kernel-based approximation (FSI). Both extend on existing methods

for the SV, namely permutation sampling [5] for SII and STI, and KernelSHAP [26] for FSI. For a comprehensive overview of the original SV methods, we refer to the appendix. We now briefly discuss existing approaches, which will be used as baselines in our experiments.

Permutation-based (PB) Approximation for STI and SII [39, 40]. The permutation-based (PB) approximation computes estimates of SII and STI based on a representation of uniformly sampled random permutations π ∼ unif ( S D ) , where S D is the set of all permutations, i.e. the set of all ordered sequences of the elements in D . Then,

Here, u -S ( π ) refers to the set of indices in π preceding the first occurrence of any element of S in π and S ∈ π is fulfilled, if all elements of S appear as a consecutive sequence in π . The estimators for SII and STI then compute an approximation by Monte Carlo integration by sampling π ∼ unif ( S D ) .

Kernel-based (KB) Approximation for FSI [40, 8]. Kernel-based (KB) approximation estimates FSI based on the representation of I as a solution to a constrained weighted least square problem

where p ( T ) ∝ µ ( t ) , is a probability distribution over T 1 and d s 0 := |S s 0 | . KB approximation for FSI estimates the expectation using Monte Carlo integration by sampling from p ( T ) and solves the approximated least-squares problem explicitly, similar to KernelSHAP [26, 8, 40]. For more details and pseudo code, we refer to the appendix.

3.3 Unbiased KernelSHAP (U-KSH) for the SV

U-KSH constitutes a variant of KernelSHAP (KSH) [26], which relies on KB approximation for the SV. In contrast to KSH, U-KSH is theoretically well understood and it was shown that the estimator is unbiased and consistent [8]. U-KSH finds an exact solution to (1) with s 0 = 1 as

U-KSH then approximates this solution using Monte Carlo integration.

Definition 3.7 (Unbiased KernelSHAP (U-KSH) [8]) . Given T 1 , . . . , T K ∼ p ( T ) ∝ µ ( t ) with binary representation Z 1 , . . . , Z K ∈ { 0 , 1 } d , U-KSH is defined as

The main idea of U-KSH is that A can be computed explicitly independent of ν and only b has to be estimated [8]. By linking U-KSH to our method (Theorem 4.5), we will show that ˆ I SV U can be greatly simplified to a weighted sum.

4 SHAP-IQ: Unified Approximation of any-order CII

So far, there exists no unified approximation technique for the general CII. In particular, it is unknown if existing approximation techniques, such PB and KB, generalize to other indices [39, 40, 13]. Furthermore, PB approximation for SII and STI is very inefficient as each update of all estimates requires a significant number of model evaluations. KB approximation for FSI efficiently computes estimates, where one model evaluation can be used to update all interaction scores. It is, however, impossible to compute only a selection of interaction estimates and theoretical results for the estimator are difficult to establish. In the following, we introduce SHAP-IQ (Section 4.1), a unified samplingbased approximation method that can be applied to any CII . SHAP-IQ is based on a Monte Carlo estimate of a novel representation of the CII and well-known statistical results are applicable. In the special case of SV, we find a novel representation of the SV and show that SHAP-IQ is linked to

U-KSH (Section 4.2). SHAP-IQ therefore greatly reduces the computational complexity of U-KSH. We further show (Section 4.3), that the sum of interaction estimates of SHAP-IQ remains constant and therefore maintains the efficiency property for STI and SII. Interestingly, for FSI this property does not hold, which should be investigated in future research. All proofs can be found in the appendix.

4.1 SHAPley Interaction Quantification (SHAP-IQ)

A key challenge in approximating the CII efficiently is that the sum changes for every interaction subset S . We thus first establish a novel representation of the CII. Based on this representation, we construct SHAP-IQ, an efficient estimator of the CII. We show that SHAP-IQ is unbiased, consistent and provide a general approximation bound.

Our novel representation of the CII is defined as a sum over all subsets T ⊆ D . In previous works, it was shown that such a representation does exist for games with ν ( ∅ ) = 0 , if the linearity axiom is fulfilled [17, Proposition 1]. We now explicitly specify this representation and show that the weights, for a CII, only depend on the sizes of T and the intersection T ∩ S . 3

Theorem 4.1 yields a novel representation of the CII, where the model evaluations ν 0 ( T ) are independent of S . This allows to utilize every model evaluation to compute all CII scores simultaneously by properly weighting with γ s m . Notably, our representation relies on ν 0 instead of ν , which constitutes an important choice for approximation, on which we elaborate in the appendix.

To approximate I , we introduce a sampling order k 0 ≥ s 0 , for which we split the sum in Theorem 4.1 to subsets with T ∈ T k 0 and T / ∈ T k 0 and rewrite

where p k 0 is over T k 0 . SHAP-IQ then estimates the CII by Monte Carlo integration.

Definition 4.2 (SHAP-IQ) . The Shapley Interaction Quantification (SHAP-IQ) of order k 0 with K samples is

SHAP-IQ is outlined in the appendix and we establish the following important theoretical guarantees.

SHAP-IQ provides efficient estimates of all CII scores with important theoretical guarantees. The sample variance ˆ σ 2 can further be used for statistical analysis of the estimates.

Finding the sampling order k 0 and distribution p k 0 . In line with KSH and U-KSH [26, 8], we find k 0 in an iterative procedure, outlined in the appendix. We consider sampling weights q ( t ) ≥ 0 for 0 ≤ t ≤ d that grow symmetrically towards the center and consider a distribution p k 0 ( T ) ∝ q ( t ) . Given a budget M and initial k 0 = 0 , we consider I m ( S ) = E T ∼ p k 0 ( T ) [ ν 0 ( T ) γ s m ( t, | T ∩ S | ) p k 0 ( T ) ] and iteratively increase k 0 , if for a subset T of size k 0 and d -k 0 , the condition M · p k 0 ( T ) ≥ 1 is fulfilled. The budget is then decreased by the number of subsets of that size, i.e. 2 ( d k 0 ) . This essentially verifies iteratively, if the expected number of subsets exceeds the total number of subsets. For more details and possible choices of sampling weights q , we refer to the appendix.

Computational Complexity. In contrast to PB approximations, SHAP-IQ allows to iteratively update all interaction estimates with one single model evaluation for any-order interactions. The

̸

weights γ s m ( t, k ) used for the updates can be efficiently precomputed. The updating process can be implemented efficiently using Welford's algorithm [45], where estimates have to be maintained for all interactions sets, i.e. d s 0 in total. In contrast to KB approximation, which requires to solve a weighted least square optimization problem with d s 0 variables, the computational effort per interaction increases linearly for SHAP-IQ. Furthermore, SHAP-IQ even allows to update selected interaction estimates, whereas, for instance, KB approximation for FSI requires to estimate all interactions. For more details on the implementation and computational complexity of the baseline methods, we refer to the appendix.

4.2 SHAP-IQ for the Shapley Value

In this section, we show that SHAP-IQ, in the special case of single feature subsets s 0 = 1 , yields novel insights into the SV. Furthermore, SHAP-IQ corresponds to U-KSH and greatly simplifies its calculation. Utilizing Theorem 4.1, we find a novel representation of the SV for every feature i ∈ D .

SHAP-IQ admits a similar form (see appendix) and corresponds to U-KSH ˆ I SV U .

Theorem 4.5 (SHAP-IQ simplifies U-KSH) . For p ( T ) ∝ µ ( t ) it holds that ˆ I SV U = ˆ I m 1 .

Theorem 4.5 implies that the U-KSH estimator can be computed using the SHAP-IQ estimator, which greatly simplifies the calculation to a weighted sum. The main idea of the proof relies on the observation that not only A can be explicitly computed, but also A -1 , cf. the appendix.

4.3 The Sum of Interaction Scores and SHAP-IQ Efficiency

In this section, we are interested in the sum of CII scores, which we link to a property of SHAPIQ estimates to maintain the efficiency axiom. By Theorem 4.1, we have ∑ S ∈S s 0 I m ( S ) = ∑ T ⊆ D ν 0 ( T ) ∑ S ∈S s 0 γ s m ( t, | T ∩ S | ) . For the SV, by Theorem 4.4, this sum is zero for every T ∈ T 1 . For higher order CIIs, we introduce the following definition.

Definition 4.6. A CII is s-efficient , if ∑ S ⊆ D, | S | = s 0 γ s m ( t, | T ∩ S | ) = 0 for every T ∈ T s 0 .

Theorem 4.7. SII and STI are s-efficient. In particular, SHAP-IQ estimates maintain efficiency for n-SII and STI.

Further, if a CII is s-efficient, then the sum of SHAP-IQ estimates remains constant. Although we did not provide a rigorous statement, it is easy to validate numerically that FSI is not s-efficient. This finding suggests that there are conceptional differences between these indices, that should be further investigated in future work. Using s-efficiency it is also possible to find an explicit formula for the sum of interaction scores for SII, which we give in the appendix.

5 Experiments

We conduct multiple experiments to illustrate the approximation quality of SHAP-IQ compared to current baseline approaches. 4 We showcase SHAP-IQ estimates on any-order SII (n-SII), on top-order STI and FSI. For each interaction index, we use its specific approximation method as a baseline. For SII and STI, we use the PB approximation and for FSI the KB approximation, further described in the appendix. We then compute n-SII based on the estimated SII values. We compare the baseline methods with SHAP-IQ using p ( T ) ∝ µ ( t ) . For each iteration we evaluate the approximation quality with different budgets up to a maximum budget of 2 14 model evaluations. To account for variation, we randomly evaluate the approximation method on 50 randomly chosen instances, further described below. To quantify the approximation quality, we compute multiple evaluation metrics for each interaction order: mean-squared error (MSE), MSE for the top-K interactions (MSE@K) and the ratio (precision) of estimated top-K interactions (Prec@K). The top-K interactions are determined in regards to their absolute value.

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

MSE

SII for the LM (

d

=14)

Method

SHAP-IQ

Baseline

Order

s

= 1

s

= 2

s

= 3

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

Prec@10

SII for the LM (

d

=14)

Method

SHAP-IQ

Baseline

Order

s

= 2

s

= 3

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

Prec@10

SII for the ICM (

d

=14)

Method

SHAP-IQ

Baseline

Order

s

= 2

s

= 3

Models. For a language model (LM), we use a fine-tuned version of the DistilBERT transformer architecture [34] on movie review sentences from the original IMDB dataset [27, 22] for sentiment analysis, i.e. ν has values in [ -1 , 1] . In the LM, for a given sentence, different feature coalitions are computed by masking absent features in a tokenized sentence. The implementation is based on the transformers API [47]. We randomly sample 50 reviews of length d = 14 and explain each model prediction. For an image classification model (ICM), we use ResNet18 [19] pre-trained on ImageNet [11] as provided by torch [30]. We randomly sample 50 images and explain the prediction of the corresponding true class. To obtain the prediction of different coalitions, we pre-compute super-pixels with SLIC [1, 44] to obtain a function on d = 14 features and apply mean imputation on absent features. For a high-dimensional synthetic model with d = 30 , we use a sum of unanimity model (SOUM) ν ( T ) := ∑ N n =1 a n 1 ( Q n ⊆ T ) , where N = 50 interaction subsets Q 1 , . . . , Q N ⊆ D are chosen uniformly from all subset sizes and a 1 , . . . , a N ∈ R are generated uniformly a n ∼ unif ([0 , 1]) . Note that the SOUM could also be viewed as an extension of the induced subgraph game [12] for a hypergraph with edges of different order.We randomly generate 50 instances of such SOUMs.

Ground-Truth (GT) Values. For the LM and the ICM we compute the ground-truth (GT) values explicitly using the representation from Theorem 4.1. For the high-dimensional SOUM it is impossible to compute the GT values naively. However, due to the linearity of the CII and the simple structure of a SOUM, we can compute the exact GT values of for any CII efficiently, cf. the appendix.

5.1 Approximation of any-order SII and n-SII scores using SHAP-IQ

In this experiment, we apply SHAP-IQ on SII and compute estimates for the LM and the ICM up to order s = 4 . We then compare the estimates with the baseline using the GT values for each order. The results are shown in Figure 2. We display the MSE for the LM (left) and the Prec@10 for the LM (middle) and ICM (right). We further compute the n-SII estimates by aggregating the SII estimates with s 0 = 4 and visualize positive and negative interactions on single individuals as proposed in [3]. Thereby, interactions are distributed equally among each participating feature, which was justified in [3, Theorem 6]. This representation amplifies the variance of our sampling-based estimator. We thus also present SHAP-IQ without sampling, i.e. c k 0 . The results are shown in Figure 3 (left) and from left to right: GT values, SHAP-IQ, SHAP-IQ without sampling and baseline. Lastly, we illustrate the n-SII scores estimates for s 0 = 3 of a movie review excerpt classified by the LM (right), where the interactions ('is','not'), ('not','bad'), and (''ll','love','this') yield a highly positive score.

The results show that SHAP-IQ outperforms the baseline methods across different models and metrics. For the n-SII visualization, we conclude that the SHAP-IQ estimator without sampling is preferable, which yields more accurate results than SHAP-IQ and the baseline methods. In general, SHAP-IQ without sampling performs surprisingly strong, and we encourage further work in this direction.

5.2 Approximation of different CIIs using SHAP-IQ

In this experiment, we apply SHAP-IQ on different CIIs, namely SII, STI and FSI. We compute top-order interactions for s 0 = 3 and compare the results with the baselines. Our results are shown in Figure 4 (left) and further experiments and results can be found in the appendix. For the LM,

it

just

seems

bizarre

that

someone

read

this

script

and

thought

this

is

funny

40

20

0

20

40

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

id: 7567

,

d

= 23

sentiment:

+ 0.916 (positive)

+ 0.740

+ 0.654

+ 2.448

It is a gruesome cannibal movie. But it's not bad. If you like Hannibal, you'll love this.

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

MSE

SII, STI, and FSI for LM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000 12000 14000 16000

model evaluations

0.0

0.1

0.2

0.3

0.4

0.5

MSE

SII, STI, and FSI for SOUM (

s

0 =2,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000 12000 14000 16000

model evaluations

0.0

0.2

0.4

0.6

0.8

1.0

Prec@10

SII, STI, and FSI for SOUM (

s

0 =2,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

SHAP-IQ clearly outperforms the baseline for SII and STI. For FSI, SHAP-IQ is outperformed by the KB approximation of the baseline. As SII and STI rely on PB approximation, our results indicate that KB approximation is more effective than PB approximation for this setting, which is in line with the strong performance of KernelSHAP [26] for the SV. However, SHAP-IQ, in contrast to KB approximation, provides a solid mathematical foundation with theoretical guarantees and we now consider a high-dimensional synthetic game, where SHAP-IQ outperforms all baselines.

SHAP-IQ on high-dimensional synthetic models. For the SOUM we compute the average and standard deviation of each evaluation metric for SHAP-IQ and the baselines for pairwise interactions ( s 0 = 2 ) of each index. Our results are shown in Figure 4 (middle and right) and further experiments and results can be found in the appendix. SHAP-IQ outperforms all baseline methods in this setting, in particular, the KB approximation of FSI that performed strongly in ML context. The experiment highlights that there exists no approximation method that performs universally best.

Runtime Analysis. The runtime of SHAP-IQ is affected by different parameters. The computation of the sampling order k 0 is a constant time operation given a number of features (cf. Algorithm 2 in the appendix). While the pre-computation of the weights ( m s ) scales linearly with the number of features, the additional computational burden is negligible as it does not depend on ν 0 . The main computational cost stems from the model evaluations (access to the value function ν 0 ), which is bounded by a model's inference time. To illustrate the runtime performance, we compare SHAP-IQ with the baseline methods on the LM using different number of model evaluations K . Figure 5 displays the runtime of SHAP-IQ and the corresponding baseline approaches, including all pre-computations. With increasing K the runtime complexity scales linearly, but the overhead of SHAP-IQ remains low. Note that the difference in STI can be attributed to less than K model evaluations, which is required to maintain efficiency, cf. lines 15-16 in Algorithm 6 of the appendix.

6 Limitations

We presented SHAP-IQ, a unified approximation algorithm for any-order CIIs with important theoretical guarantees. SHAP-IQ relies on the specific structure in terms of discrete derivatives from Definition 3.3. This representation exists for every interaction indices that fulfills the linearity, symmetry and dummy axiom [17, Proposition 5]. However, for FSI, which has been defined as the solution to the weighted least square problem, a closed-form representation for lower-order interactions in terms of discrete derivatives remains difficult to establish [40, Lemma 70]. This limits the applicability of SHAP-IQ to top-order interactions of FSI, for which this representation is given in [40, Theorem 19]. The FSI baseline performs strongly in ML context, in line with empirical findings for KernelSHAP [26] for the SV. However, the estimator is theoretically not well understood [8] and we have shown that it is not universally best. Moreover, for other CIIs, such as SII and STI, it is unlikely [13] that such an explicit form in terms of a weighted least square problem can be found, which limits the applicability of KB approximation to FSI. SHAP-IQ outperforms the baselines of SII and STI by a large margin, is generally applicable and supported by a solid mathematical foundation.

7 Conclusion

How to extend the SV to interactions is an open research question. In this work, we considered CIIs, a broad class of interaction indices, which covers all currently proposed indices, as well as all indices that fulfill the linearity, symmetry and dummy axiom. We established a novel representation of the CII, which we used to introduce SHAP-IQ, an efficient sampling-based approximation algorithm that is unbiased and consistent. For the special case of SV, SHAP-IQ can be seen as a generalization of U-KSH [8] and greatly simplifies its calculation as well as providing a novel representation of the SV. Furthermore, for n-SII and STI, SHAP-IQ maintains the efficiency condition, which is a direct consequence of a specific property, which we coin s-efficiency for CIIs. We applied SHAP-IQ in multiple experimental settings to compute any-order interactions of SII and n-SII, where SHAP-IQ consistently outperforms the baseline method and showcased the applicability of feature interaction scores to understand black-box language and image classification models. SHAP-IQ further benefits from a solid statistical foundation, which can be leveraged to improve the approximation quality.

Future work. Applying SHAP-IQ to real-world applications, such as NLP tasks [43] and genomics [48, 23], could yield valuable insights. However, the exponentially increasing number of interactions requires human-centered post-processing to enhance interpretability for practitioners and ML engineers, e.g. through automated dialogue systems [37]. Further, it would be beneficial to discover the statistical capabilities of SHAP-IQ to provide confidence bounds or approximate interaction scores sequentially. Beyond model-agnostic approximation, model-specific variants could substantially reduce computational complexity. For instance, it is likely that ideas of TreeSHAP [25] for tree-based models can be extended to Shapley-based interactions.

References

Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., and Süsstrunk, S. (2012). Slic superpixels compared to state-of-the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 34(11):2274-2282.

Adadi, A. and Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (xai). IEEE Access , 6:52138-52160.

Bordt, S. and von Luxburg, U. (2023). From shapley values to generalized additive models and back. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, (AISTATS 2023) , volume 206 of Proceedings of Machine Learning Research , pages 709-745. PMLR.

Castro, J., Gómez, D., Molina, E., and Tejada, J. (2017). Improving polynomial estimation of the shapley value by stratified random sampling with optimum allocation. Computers & Operations Research , 82:180-188.

Castro, J., Gómez, D., and Tejada, J. (2009). Polynomial calculation of the shapley value based on sampling. Computers & Operations Research , 36(5):1726-1730.

Charnes, A., Golany, B., Keane, M., and Rousseau, J. (1988). Extremal Principle Solutions of Games in Characteristic Function Form: Core, Chebychev and Shapley Value Generalizations , volume 11 of Advanced Studies in Theoretical and Applied Econometrics , page 123-133. Springer Netherlands.

Chen, H., Covert, I. C., Lundberg, S. M., and Lee, S.-I. (2023). Algorithms to estimate shapley value feature attributions. Nature Machine Intelligence , 5(6):590-601.

Covert, I. and Lee, S.-I. (2021). Improving kernelshap: Practical shapley value estimation using linear regression. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, (AISTATS 2021) , volume 130 of Proceedings of Machine Learning Research , page 3457-3465. PMLR.

Covert, I., Lundberg, S. M., and Lee, S. (2021). Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research , 22:209:1-209:90.

Cui, T., Marttinen, P., and Kaski, S. (2020). Learning global pairwise interactions with bayesian neural networks. In 24th European Conference on Artificial Intelligence, (PAIS 2020) , volume 325 of Frontiers in Artificial Intelligence and Applications , pages 1087-1094. IOS Press.

Deng, J., Dong, W., Socher, R., Li, L., Li, K., and Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR 2009) , pages 248-255. IEEE Computer Society.

Deng, X. and Papadimitriou, C. H. (1994). On the complexity of cooperative solution concepts. Mathematics of Operations Research , 19(2):257-266.

Ding, G., Lax, R. F., Chen, J., and Chen, P. P. (2008). Formulas for approximating pseudoboolean random variables. Discrete Applied Mathematics , 156(10):1581-1597.

Fujimoto, K., Kojadinovic, I., and Marichal, J. (2006). Axiomatic characterizations of probabilistic and cardinal-probabilistic interaction indices. Games Econ. Behav. , 55(1):72-99.

Ghorbani, A. and Zou, J. Y. (2019). Data shapley: Equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning, (ICML 2019) , volume 97 of Proceedings of Machine Learning Research , pages 2242-2251. PMLR.

Grabisch, M., Marichal, J., and Roubens, M. (2000). Equivalent representations of set functions. Mathematics of Operations Research , 25(2):157-178.

Grabisch, M. and Roubens, M. (1999). An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of Game Theory , 28(4):547-565.

Hammer, P. L. and Holzman, R. (1992). Approximations of pseudo-boolean functions; applications to game theory. ZOR Mathematical Methods of Operations Research , 36(1):3-21.

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, (CVPR 2016) , pages 770-778. IEEE Computer Society.

Janizek, J. D., Sturmfels, P., and Lee, S. (2021). Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research , 22:104:1-104:54.

Jia, R., Dao, D., Wang, B., Hubis, F. A., Hynes, N., Gürel, N. M., Li, B., Zhang, C., Song, D., and Spanos, C. J. (2019). Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, (AISTATS 2019) , volume 89 of Proceedings of Machine Learning Research , pages 1167-1176. PMLR.

Lhoest, Q., Villanova del Moral, A., von Platen, P., Wolf, T., Šaško, M., Jernite, Y ., Thakur, A., Tunstall, L., Patil, S., Drame, M., Chaumond, J., Plu, J., Davison, J., Brandeis, S., Sanh, V ., Le Scao, T., Canwen Xu, K., Patry, N., Liu, S., McMillan-Major, A., Schmid, P., Gugger, S., Raw, N., Lesage, S., Lozhkov, A., Carrigan, M., Matussière, T., von Werra, L., Debut, L., Bekman, S., and Delangue, C. (2021). Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, (EMNLP 2021) , pages 175-184. Association for Computational Linguistics.

Liu, G., Zeng, H., and Gifford, D. K. (2019). Visualizing complex feature interactions and feature sharing in genomic deep neural networks. BMC Bioinformatics , 20(1):401:1-401:14.

Lou, Y., Caruana, R., Gehrke, J., and Hooker, G. (2013). Accurate intelligible models with pairwise interactions. In The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (KDD 2013) , pages 623-631. ACM.

Lundberg, S. M., Erion, G. G., Chen, H., DeGrave, A. J., Prutkin, J. M., Nair, B., Katz, R., Himmelfarb, J., Bansal, N., and Lee, S. (2020). From local explanations to global understanding with explainable ai for trees. Nature Machine Intelligence , 2(1):56-67.

Lundberg, S. M. and Lee, S. (2017). A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, (NeurIPS 2017) , pages 4765-4774. Curran Associates, Inc.

Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. (2011). Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, (HLT 2011) , pages 142-150. Association for Computational Linguistics.

Molnar, C., Casalicchio, G., and Bischl, B. (2019). Quantifying model complexity via functional decomposition for better post-hoc interpretability. In Machine Learning and Knowledge Discovery in Databases - International Workshops of ECML PKDD 2019 , pages 193-204.

Murdoch, W. J., Liu, P. J., and Yu, B. (2018). Beyond word importance: Contextual decomposition to extract interactions from lstms. In 6th International Conference on Learning Representations, (ICLR 2018) .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, (NeurIPS 2019) , pages 8024-8035. Curran Associates, Inc.

Rabitti, G. and Borgonovo, E. (2019). A shapley-owen index for interaction quantification. SIAM/ASA Journal on Uncertainty Quantification , 7(3):1060-1075.

Ruiz, L. M., Valenciano, F., and Zarzuelo, J. M. (1996). The least square prenucleolus and the least square nucleolus. two values for tu games based on the excess vector. International Journal of Game Theory , 25(1):113-134.

Ruiz, L. M., Valenciano, F., and Zarzuelo, J. M. (1998). The family of least square values for transferable utility games. Games and Economic Behavior , 24(1):109-130.

Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. CoRR , abs/1910.01108.

Shapley, L. S. (1953). A value for n-person games. In Contributions to the Theory of Games (AM-28), Volume II , pages 307-318. Princeton University Press.

Singh, C., Murdoch, W. J., and Yu, B. (2019). Hierarchical interpretations for neural network predictions. In 7th International Conference on Learning Representations, (ICLR 2019) .

Slack, D., Krishna, S., Lakkaraju, H., and Singh, S. (2023). Explaining machine learning models with interactive natural language conversations using talktomodel. Nature Machine Intelligence , 5(8):873-883.

Spivey, M. Z. (2007). Combinatorial sums and finite differences. Discrete Mathematics , 307(24):3130-3146.

Sundararajan, M., Dhamdhere, K., and Agarwal, A. (2020). The shapley taylor interaction index. In Proceedings of the 37th International Conference on Machine Learning, (ICML 2020) , volume 119 of Proceedings of Machine Learning Research , pages 9259-9268. PMLR.

Tsai, C., Yeh, C., and Ravikumar, P. (2023). Faith-shap: The faithful shapley interaction index. Journal of Machine Learning Research , 24(94):1-42.

Tsang, M., Cheng, D., Liu, H., Feng, X., Zhou, E., and Liu, Y. (2020a). Feature interaction interpretability: A case for explaining ad-recommendation systems via neural interaction detection. In 8th International Conference on Learning Representations, (ICLR 2020) .

Tsang, M., Cheng, D., and Liu, Y. (2018). Detecting statistical interactions from neural network weights. In 6th International Conference on Learning Representations, (ICLR 2018) .

Tsang, M., Rambhatla, S., and Liu, Y. (2020b). How does this interaction affect me? interpretable attribution for feature interactions. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, (NeurIPS 2020) , pages 6147-6159. Curran Associates, Inc.

van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J. D., Yager, N., Gouillart, E., Yu, T., and the scikit-image contributors (2014). scikit-image: image processing in python. PeerJ , 2(6):e453.

Welford, B. P. (1962). Note on a method for calculating corrected sums of squares and products. Technometrics , 4(3):419-420.

Williamson, B. D. and Feng, J. (2020). Efficient nonparametric statistical inference on population feature importance using shapley values. In Proceedings of the 37th International Conference on Machine Learning, (ICML 2020) , volume 119 of Proceedings of Machine Learning Research , pages 10282-10291. PMLR.

Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y ., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, (EMNLP 2020) , pages 38-45. Association for Computational Linguistics.

Wright, M. N., Ziegler, A., and König, I. R. (2016). Do little interactions get lost in dark random forests? BMC Bioinformatics , 17(145).

Yeh, C., Kim, J. S., Yen, I. E., and Ravikumar, P. (2018). Representer point selection for explaining deep neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, (NeurIPS 2018) , pages 9311-9321. Curran Associates, Inc.

Appendix of 'SHAP-IQ: Unified Approximation of any-order Shapley Interactions'

Organisation of the Appendix

We provide further theoretical and experimental results for SHAP-IQ. The appendix is organized as follows: In Appendix A, we formally introduce specific CIIs, such as SII, n-SII, STI and FSI and their axiomatic foundation and theoretical results and provide a novel theoretical result for the sum of SII scores. In Appendix B, we provide all proofs of theoretical results from the main paper. In Appendix C, we give further insights into the implementation of SHAP-IQ. In Appendix D, we provide further experimental results, information about the used models, explicit formulas for the SOUM interaction scores, and pseudo-code and information on the computational complexity of baseline implementations. In Appendix E, we provide further theoretical results for the special case of SV, in particular the explicit form of the covariance matrix from [8] and a simplified representation, similar to Theorem 4.4, of SHAP-IQ in this case. In Appendix F, we describe the approximation methods ApproShapley [5] and KernelSHAP [26] for the SV on which our baseline methods are built.

A Shapley-based Interaction Indices and further theoretical Results

In this section, we review the axiomatic structures of n-SII, SII, STI and FSI. We further provide an additional result for the sum of SII interaction scores.

A.1 Shapley-based Interaction Indices

We consider G as the set of all games ν : P ( D ) → R . In the following, we formalize the axioms for the Shapley interaction indices I ν : P ( D ) → R

Definition A.1 (Linearity Axiom [17]) . I ν is linear , if for any two games ν 1 , ν 2 ∈ G and any S ⊆ D , it holds I ν 1 + ν 2 ( S ) = I ν 1 ( S ) + I ν 2 ( S ) .

Definition A.2 (Dummy Axiom [17]) . For a dummy player i ∈ D for a game ν ∈ G , i.e. constant contribution c ( i ) added to any coalition ν ( T ∪ { i } = ν ( T ) + c ( i ) , then for every T ⊆ D \ { i } , the dummy axiom requires I ν ( S ∪ { i } ) = 0 for any S ⊆ D \ { i } . That is, a dummy player has no interaction with any coalition.

Definition A.3 (Symmetry Axiom [17]) . I ν is said to fulfill the symmetry axiom , if for any permutation π on D it holds I ν ( S ) = I πν ( πS ) , where πν ( πS ) := ν ( S ) and πS := { π ( i ) : i ∈ S } changes the ordering of the players.

Definition A.4 (Recursive Axiom [17]) . I ν fulfills the recursive axiom , if for any S ⊆ D with | S | > 1 and any game ν ∈ G

̸

where ν [ S ] is the game, where all players in S is considered as one player, and ν D \ K is a game defined on the subset of players D \ K .

The recursive axiom defines higher order interactions using lower order interaction. For pairwise interactions it can be stated as I ν ( ij ) = I ν [ ij ] ([ ij ]) -I ν D \{ j } ( i ) -I ν D \{ i } ( j ) , i.e. the pairwise interaction is the difference of the value for the reduced player [ ij ] and the individual player values for the reduced game.

Definition A.5 (Shapley Interaction Index [17]) . The Shapley interaction index (SII) is the unique interaction index that satisfies the linearity, dummy, symmetry and recursive axiom, where the values for | S | = 1 correspond to the Shapley value. It can be represented as a CII as

In contrast to the SV, the SII does not yield an efficiency property, which is desirable in ML context. The efficiency axiom was therefore introduced for interactions. The following axioms rely on a maximum interaction order s 0 , where the values of the interaction indices change for different maximum interaction orders.

Definition A.6 (Efficiency Axiom [39]) . For all ν ∈ G , it holds ∑ S ∈S s 0 I ν ( S ) = ν ( D ) -ν ( ∅ ) .

The efficiency axiom is an extension of the SV efficiency axiom and requires that all interaction scores up to the maximum order s 0 sum up to ν ( D ) -ν ( ∅ ) . For SII there exists a unique recursive aggregation, such that the efficiency axiom is fulfilled. This novel interaction index is referred to as n-Shapley Values (n-SII) [3].

Definition A.7 (n-Shapley Values (n-SII) [3]) . Given s 0 the n-Shapley Values (n-SII) are defined as

Besides n-SII, there have been two axiomatic extensions to CIIs that directly require the efficiency axiom together with the linearity, symmetry and dummy axiom. However, unlike for the SV, it is not sufficient to require efficiency for a unique interaction index. The Shapley Taylor Interaction Index (STI) further specifies the interaction distribution axiom, which then yields a unique index [39].

Definition A.8 (Interaction Distribution Axiom [39]) . For an interaction function ν T parametrized by T ⊆ D it holds ν T ( S ) = 0 , if T ⊊ S and a constant value ν T ( S ) = c for T ⊆ S and c ∈ R . The interaction distribution axiom requires for all S ⊆ D with S ⊊ T and s < s 0 that I ν T ( S ) = 0 .

Definition A.9 (The Shapley Taylor Interaction Index [39]) . The Shapley Taylor interaction index (STI) is the unique interaction index that satisfies the linearity, dummy, symmetry, efficiency and interaction distribution axiom.

The STI yields a unique interaction index by introducing the interaction distribution axiom, which favors the maximum order interactions as discussed in [40]. It was thus argued that instead the representation of the interaction index as a solution to a weighted least square problem is preferable [40], which yields the Faith-Interaction Index.

Definition A.10 (Faith-Interaction Index [40]) . I is called a Faith-interaction index if it can be expressed as

Definition A.11 (Faithful Shapley Interaction Index [40]) . The Faithful Shapley interaction index (FSI) is the unique faith-interaction index that satisfies the linearity, dummy, symmetry and efficiency axiom.

While n-SII, SII, STI and FSI offer differnt ways of characterizing an interaction index, it was shown in [17] that every interaction index satisfying the linearity, symmetry and dummy axiom admits a CII representation.

Proposition A.12. Every interaction index satisfying the linearity, symmetry and dummy axiom can be represented as a CII

Furthermore, the weights m s 0 for the top-order interaction indices, i.e. s = s 0 are defined as

The definitions for lower order interactions can be found in [17] for SII, in [39] for STI and in [40] for FSI. Note that for FSI a closed-form solution for lower-order interactions in terms of discrete derivatives remains unknown [40, Lemma 70].

Proof. For the proofs, we refer to the corresponding paper of each index. The general statement is proven in [17, Proposition 5].

A.2 Explicit Formula for Sum of SII Scores

Using s-efficiency, it is easy to calculate the sum of SII scores, which provide an explicit representation of a formula considered in [31, Theorem 4.2].

Proposition A.13 (Sum of SII Scores) . For SII it holds

Proof. For SII, we let m ( t ) := m SII s ( t ) = 1 d -s +1 ( d -s t ) -1 and have by Theorem 4.7 and the definition of s-efficiency

For t < s 0 we have

For t > d -s there are at least k min = t -( d -s ) elements in the intersection of | T ∩ S | and thus with ¯ t := d -t < s 0 and k min = s 0 -¯ t

We can explicitly compute ρ ( t ) as

where we have used that ∑ t k =0 ( -1) k ( s 0 k ) = ( -1) t ( s 0 -1 t ) for t < s 0 . Hence,

B Proofs

This section contains the proofs of the claims made in the main paper.

B.1 Proof of Theorem 4.1

Proof. By definition, the sum I m ( S ) := ∑ T ⊆ D \ S m ( t ) δ ν S ( T ) ranges over all subsets T ⊆ D , where every subset is exactly once evaluated. On the one hand, it is easy to see that every evaluated subset in I m ( S ) is different, as T ∪ L is unique. Furthermore, given any subset T ⊆ D , we decompose T = ˜ T ∪ L , where ˜ T ⊆ D \ S and L := T ∩ S ⊆ S . The corresponding weight is m ( ˜ t ) = m ( t -l ) = m ( t -| T ∩ S | ) and the sign from δ ν S ( ˜ T ) is ( -1) s -l = ( -1) s -| T ∩ S | . This yields with ν 0 ( T ) := ν ( T ) -ν ( ∅ )

as the sum over all γ m is zero, if the dummy axiom is fulfilled.

Remark B.1. It is important to note, that the sum ∑ T ⊆ D γ s m ( t, | T ∩ S | ) is not zero, if not all subsets are considered, which makes it crucial to use ν 0 instead of ν . In fact, the estimates of I m would be heavily skewed by ν ( ∅ ) . While the estimator would still be unbiased, its variance would scale with ν ( ∅ ) 2 .

B.2 Proof of Theorem 4.3

Proof. We aim to show that ˆ I m ( S ) is unbiased and consistent, i.e. E [ ˆ I m k 0 ( S ) ] = I m ( S ) and lim K →∞ ˆ I m ( S ) = I m ( S ) . Given

it is clear that due to the linearity of the expectation

Furthermore, let σ 2 ( S ) := V T ∼ p k 0 ( T ) [ ν 0 ( T ) γ s m ( | T | , | T ∩ S | ) p k 0 ( T ) ] be the variance of each estimate, then, by the law of large numbers

and thus lim K →∞ ˆ I m ( S ) = I m ( S ) . Lastly, as ˆ I m ( S ) is unbiased, we have for ϵ > 0 by Chebyshev's inequality

B.3 Proof of Theorem 4.4

Proof. We let m ( t ) := ( d -t -1)! t ! d ! and apply Theorem 4.1. With γ s m (0 , 0) = -m (0) , γ s m ( d, 1) = m ( d -1) and m (0) = m ( d -1) = 1 d , we have

B.4 Proof of Theorem 4.5

Proof. According to Proposition E.3, our goal is to show that

with T k iid ∼ p ( T ) := µ ( t ) / (2 h d -1 ) , where p is a probability distribution over T 1 and h n := ∑ n t =1 t -1 . The proof is structured in the following steps:

Exact computation of A -1 using the exact structure of A with diagonal entries µ 1 and off-diagonal entries µ 2 , cf. [8, Appendix A].

Exact computation of ˆ I SV U , which yields with Proposition E.3 ˆ I SV U ( i ) = ˆ I m 1 ( i ) , if ( µ 1 -µ 2 )2 h d -1 = 1 .

We show that ( µ 1 -µ 2 )2 h d -1 = 1 .

Calculation of A -1 . It has been shown in [8, Appendix A] that all off-diagonal entries are equal and all diagonal entries are equal, i.e. A may be written as A = µ 2 J +( µ 1 -µ 2 ) I with off-diagonal entries µ 2 := p ( Z i = Z j = 1) and diagonal entries µ 1 := p ( Z i = 1) , where Z i refers to the i-th component of the binary vector Z and J is a matrix of ones and I is the identity matrix. The simple structure of A allows to compute the inverse exactly by using the following Lemma.

̸

Lemma B.2. Let µ 1 , µ 2 > 0 with µ 1 = µ 2 , then

with

Proof. We compute and hence

which yields ( µ 1 -µ 2 )(˜ µ 1 -˜ µ 2 ) = 1 and µ 2 ˜ µ 2 d +( µ 1 -µ 2 )˜ µ 1 +(˜ µ 1 -˜ µ 2 ) µ 2 = 0 . From the first equation we have ˜ µ 1 -˜ µ 2 = 1 / ( µ 1 -µ 2 ) and thus by the second equation

Calculation of ˆ I SV U . By Lemma B.2, we proceed to compute the different components of

First,

Then the denominator yields 1 T A -1 1 = d µ 1 +( d -1) µ 2 . We then obtain

which, with A -1 1 = (˜ µ 1 +( d -1)˜ µ 2 ) 1 = 1 µ 1 +( d -1) µ 2 1 , yields

It remains to show that

With ˆ b L = 1 K ∑ K k =1 z k ν 0 ( z k ) it follows

Then A -1 z k = t k ˜ µ 2 1 +(˜ µ 1 -˜ µ 2 ) z k where t k is the subset size, i.e. t k is the sum of all entries in z k . It follows with A -1 J = 1 µ 1 +( d -1) µ 2 J

For the i-th component, we have with set notation T k and for the SV weights m ( t ) := ( d -t -1)! t ! d ! then

where we have used Proposition E.3 for ˆ I m 1 ( i ) . It remains to show that ( µ 1 -µ 2 )2 h d -1 = 1 .

Show that ( µ 1 -µ 2 )2 h d -1 = 1 . We let p ( T k ) := µ ( t ) /R be a probability distribution over T 1 . By definition, and as subsets of size t have equal probability, we have

and

Hence,

where we have used the recursion for the binomial coefficient and µ ( t ) = 1 d -1 ( d -2 t -1 ) -1 . Lastly, we have seen in the proof of Proposition E.3 that R = 2 h d -1 , which finishes the proof.

B.5 Proof of Theorem 4.7

Proof. We prove the statements in separate subsections. We consider interactions of maximum order s 0 ≥ 1 , summarized in I := { S ⊂ D | s = s 0 } . To show that SII and STI are s-efficient, by Theorem 4.1 it suffices to show that ∑ S ∈I γ s m ( t, | T ∩ S | ) = 0 for all T ∈ T s 0 . Given a subset T ∈ T s 0 with | T ∩ S | = k and k ∈ { 0 , . . . , s 0 } , we have

B.5.1 Proof of s-efficiency for SII.

We let m ( t ) := m SII s 0 ( t ) = ( d -t -s 0 )! t ! ( d -s 0 +1)! = 1 d -s 0 +1 ( d -s 0 t ) -1 and obtain

We now consider

where we have used the binomial expansion for (1 -1) s 0 . Hence,

which finishes the proof of SII s-efficiency.

B.5.2 Proof of s-efficiency of STI.

For STI, i.e. m STI s 0 := s 0 ( d -t -1)! t ! d ! , we have

As ( d -t + k -1)! ( d -t -s 0 + k )! is a polynomial with orders less than s 0 , we can use ∑ s 0 k =0 ( s 0 k ) k m = 0 for m<s 0 [38] to obtain

which finishes the proof of STI s-efficiency.

B.5.3 Proof of efficiency for STI.

It is clear that s-efficiency of a CII implies by Theorem 4.1 with sampling order k ≥ s 0 that the sum of top-order interaction estimates is

On the other hand, it also implies that the sum of STI scores for the top-order interactions are

While lower-order estimates are computed exactly for STI, it follows that the sum of STI scores and sum of SHAP-IQ estimates over all interaction sets S s 0 are equal. Furthermore, due to the definition of STI, they must fulfill the efficiency axiom, which finishes the proof.

B.5.4 Proof of efficiency for n-SII.

This result follows from the aggregation suggested by n-SII, which is independent of the index. The efficiency condition follows directly from the SV efficiency and the Bernoulli numbers, independent of higher-order interaction values, cf. proof in [3, Proposition 12]. The proof is based on induction, starting from the SV, where the efficiency condition holds. The specific aggregation based on the Bernoulli numbers then ensures that this efficiency condition is maintained. To apply this observation to SHAP-IQ estimates of n-SII, we observe that due to Proposition E.3, SHAP-IQ maintains the efficiency condition for order s = 1 , i.e. the SV estimates. The aggregation of n-SII for the SHAP-IQ estimates of higher orders then immediately implies that this efficiency condition is always preserved, as the arguments presented therein hold independent of the interaction index, cf. [3, Proof of Proposition 12].

C Algorithmic Details of SHAP-IQ

In this section, we decribe further algorithmic details that are used in the SHAP-IQ implementation.

C.1 Algorithm of SHAP-IQ

The pseudo-code for SHAP-IQ is outlined in Algorithm 1.

C.2 Sampling Weights q ( t )

In our implementation, we rely on q ( t ) ∝ µ ( t ) for 1 ≤ t ≤ d -1 , where the weights q (0) = q ( d ) = q 0 ≫ 0 are set to a high positive constant, which favors these subsets before weighting the remaining subsets in T 1 . These weights ensure that SHAP-IQ is equal to U-KSH for the SV. Another choice of weights is q ( t ) = ( d -t -s 0 )!( t -s 0 )! ( d -s 0 +1)! for s 0 ≤ t ≤ d -s 0 and q ( t ) = q 0 otherwise, which prefers all orders up to s 0 and from d to d -s 0 . This choice of subset weights may be beneficial for very low budgets, as it is important to ensure that k 0 ≥ s 0 for SHAP-IQ to maintain the efficiency condition. The algorithm to find k 0 given weight q and budget K is outlined in Algorithm 2. The sampling procedure to generate a subset according to p ( T ) is outlined in Algorithm 3.

C.3 Welford's Algorithm

Welford's algorithm [45] allows to iteratively update the mean and variance using a single pass. The algorithm is outlined in Algorithm 4.

Algorithm 2 Determine the the sampling order k 0 for the deterministic part

Require: weights q over 0 , . . . , d , budget K > 0

1: initialize k 0 = 0

2: for t = 0 , . . . , Floor ( d/ 2) do

4: p [ t ] ← q [ t ] ( d t ) /R

5: p [ d -t ] ← q [ d -t ] ( d t ) /R

9: end if

10: end for

11: return k 0

Algorithm 3 Sample a subset T ∼ p ( T )

Require: p with ∑ d -k 0 k = k 0 p [ k ] = 1 , sampling order k 0

3: end for

4: choose subset size t 0 with probability p ( | T | = t )

5: choose subset T of size t 0 with probability ( d t 0 )

6: return T

Algorithm 4 Welford Algorithm for Mean and Variance [45]

1: n ← n +1

6: return µ, s 2

-

1

▷ Normalization

D Experiments

For the interested reader, we provide a more detailed view on our empirical evaluation of Section 5. We give descriptions and pseudocode of the baseline algorithms approximating the three considered interaction indices SII, STI, and FSI, formal definitions of our synthetic games, and finally further obtained results that we omitted in the main part due to space constraints.

D.1 Baseline Algorithms for SII, STI, and FSI

In this section, we describe our baseline algorithms for SII, STI and FSI. We distinguish between permutation-based approximation (SII and STI) and kernel-based approximation (FSI).

D.1.1 Permutation-based (PB) Approximation

The algorithm for SII is outlined in Algorithm 5. Note that with each permutation only d -s +1 interaction estimates of order s are updated.

The sampling-based algorithm for top-order interactions of STI is outlined in Algorithm 6. Note that with each permutation all top-order interaction estimates can be updated. However, the update requires a significant amount (permutationCost) of model evaluations.

D.1.2 Kernel-based (KB) Approximation

Given a budget of K , we first find the sampling budget by identifying k 0 according to Algorithm 2 with weights q ( t ) := µ ( t ) and subtracting the number of subsets used for the deterministic part. We then sample the remaining subsets according to p ( T ) ∝ µ ( t ) according to Algorithm 3.

Given the collection of K subsets (deterministic and sampled), we solve an approximated weighted least square objective as

where k 0 is found similar to SHAP-IQ and p k 0 is a probability distribution over T k 0 with p k 0 ( T ) ∝ µ ( t ) , which is related to p with p k 0 ( T ) = p ( T ) /p ( T ∈ T k 0 ) . The expectation over p k 0 is then found by Monte Carlo integration. Approximating this objective yields a weighted sum of sampled subsets that approximates the weighted least square problem. This approximated least-square problem is then computed explicitly using

where Z ∈ { 0 , 1 } K × d s 0 is a matrix that represents a binary encoding for each sampled subset where an entry in column S ∈ S s 0 is equal to one, if the subset contains S and zero otherwise. The matrix W ∈ R K × K contains the weights for each subset on the diagonal, e.g. p ( T ) for subsets of the deterministic part or p ( T ∈ T k 0 ) /m for subsets of the sampled part, where m refers to the number of sampled subsets for Monte Carlo integration. The vector y consists of all model evaluations ν 0 ( T ) , where T is in the collection of subsets. To include the optimization constraint, we add D to the

collection with weight set to a high positive constant (mimicking infinite). The algorithm is outlined in Algorithm 7.

D.1.3 Computational Complexity of Baseline Methods

To evaluate one permutation for STI, the PB algorithm requires 2 s model evaluations per interaction, i.e. in total ( d s 0 ) · 2 s 0 for all top-order interactions. With each evaluated permutation all interaction estimates can be updated. For lower order interactions, STI requires to compute model evaluations for all subsets with | S | ≤ s 0 . For SII, the complexity is ( d -s +1) · 2 s per permutation and only interaction estimates with S ∈ π can be updated per permutation, i.e. d -s +1 interaction estimates with one permutation. This constitutes a significant drawback over the PB approximations for SV, which iterates only once through the permutation requiring d -1 evaluations to update all estimates of the SV. In contrast, as for SV, the KB approach of FSI allows to update all interaction estimates using one single model evaluation. However, the KB approach of FSI always requires to estimate all interactions with order s ≤ s 0 and its computational effort increases non-linear with the number of subsets used, as solving the weighted least square problem requires inverting a K × d s 0 matrix.

D.2 Further Information about the Models

This section contains a detailed information about the models (SOUM, LM, and ICM) used in our experiments.

D.2.1 Sum of Unanimity Model (SOUM)

Definition D.1 (Sum of Unanimity Model (SOUM)) . For N subsets Q 1 , . . . , Q N ⊆ D and coefficients a 1 , . . . , a N ∈ R the sum of unanimity model (SOUM) is defined as

For SOUMs, it is possible to efficiently compute the ground-truth values for CII.

Proposition D.2 (Ground-truth values for SOUM) . For a SOUM, it holds

with

and k max ( r ) := min( t -q, s -r )

Proof. Due to the linearity of the CII, it suffices to compute the CII for ν Q ( T ) := 1 ( Q ⊆ T ) . By Theorem 4.1, we have

where we used that Q ∩ S ⊆ T ∩ S due to 1 ( Q ⊆ T ) and | T ∩ S | = | S ∩ Q | +( | T ∩ S | ) \ ( | S ∩ Q | ) , where k := | ( T ∩ S ) \ ( S ∩ Q ) | ranges from 0 to k max := min( t -q, s -| S ∩ Q | ) . Since S ∩ Q is fixed, we need to count the number of subsets T of size t , given k , such that | T ∩ S | = | S ∩ Q | + k . We count ( s -| S ∩ Q | k ) ways to choose subsets of elements that are not in S ∩ Q but are in S . Then q -( s -| S ∩ Q | ) elements of T are fixed. We thus select from d -q -( s -| S ∩ Q | ) elements exactly t -q -k elements, as q and k elements are already contained in T .

Finally, the CII value is given as

where the weights ω can be precomputed with | S ∩ Q n | ∈ { 0 , . . . , s } .

D.2.2 Language Model (LM)

For a language model (LM), we use a fine-tuned version of the DistilBERT 5 transformer architecture [34] on movie review sentences from the original IMDB dataset [27] for sentiment analysis, i.e. ν has values in [ -1 , 1] . The IMDB data stems from the dataset library [22]. In the LM, for a given sentence, different feature coalitions are computed by masking absent features in a tokenized sentence. The implementation is based on the transformers API [47].

D.2.3 Image Classifier (ICM)

The image classification model (ICM) is a ResNet18 [19] pre-trained on ImageNet [11] as provided by torch [30]. We randomly sample 50 images from ImageNet [11] and explain the prediction of the highest probability class given the original image. To obtain the prediction of different coalitions, we pre-compute super-pixels with SLIC [1, 44] to obtain a function on d = 14 features and apply mean imputation on absent features.

D.3 Hardware Details and Environmental Impact of the Experiments

Running the experiments required approximately 2 000 CPU hours in total. The experiments concerning the approximation quality of SHAP-IQ compared to the baselines were run on an computation cluster on hyperthreaded Intel Xeon E5-2697 v3 CPUs clocking at with 2.6Ghz. To further increase the efficiency of the experiments, the outputs of the LM and ICM were pre-computed given the powerset of all features. Around 1 500 CPU hours were consumed for these experiments on the cluster. Before running the experiments on the cluster, the implementations were validated on a Dell XPS 15 9510 containing an Intel i7-11800H at 2.30GHz. For this and further small-scale experiments like the n-SII values approximately 500 CPU hours were consumed.

D.4 Further Experimental Results

This section describes the further results and experiments omitted in the main body of the work.

D.4.1 Approximation Quality of top Order Interactions.

We further compute interaction scores for s 0 = 1 , s 0 = 2 , s 0 = 3 , and s 0 = 4 of all three interaction indices SII, STI, and FSI on the LM. We plot the MSE and Prec@10 based on g = 50 independent iterations for these settings. All results are summarized in Figure 6. Moreover, we compute interaction scores for s 0 = 1 , s 0 = 2 , s 0 = 3 , and s 0 = 4 of all three interaction indices SII, STI, and FSI on the ICM. The MSE and Prec@10 based on g = 50 independent iterations for these settings are shown in Figure 7. Further results to the plots show in Section 5.2 for the SOUM are presented in Figure 8.

D.4.2 n-SII Estimation on Example Sentences.

We further probe the LM with randomly selected sentences from the IMDB dataset and estimate the SII scores up to order 4 . For the example sentence ' It is a gruesome cannibal movie. But it's not bad. If you like Hannibal, you'll love this. ' in Section 5.2 all orders of n-SII with s 0 = 1 , 2 , 3 , 4 are illustrated in Figure 9. Another example sentence ' I have never forgot this movie. All these years and it has remained in my life. ' with all orders of n-SII up to the maximum interaction order s 0 = 1 , 2 , 3 , 4 is shown in Figure 10. Lastly, for four sentences n-SII for all orders with s 0 = 4 are visualized in Figure 11.

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0

1

2

3

4

5

MSE

1e

1

SII, STI, and FSI for LM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

MSE

1e

1

SII, STI, and FSI for LM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for LM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

MSE

1e

1

SII, STI, and FSI for LM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

MSE

1e

1

SII, STI, and FSI for LM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for LM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

MSE

1e

2

SII, STI, and FSI for LM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0

1

2

3

4

5

MSE

1e

3

SII, STI, and FSI for LM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for LM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

MSE

1e

3

SII, STI, and FSI for LM (

s

0 =1,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

MSE

1e

2 SII, STI, and FSI for ICM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0

1

2

3

4

5

MSE

1e

3

SII, STI, and FSI for ICM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for ICM (

s

0 =4,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

MSE

1e

2

SII, STI, and FSI for ICM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0

1

2

3

4

5

MSE

1e

3

SII, STI, and FSI for ICM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for ICM (

s

0 =3,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

MSE

1e

3

SII, STI, and FSI for ICM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.0

0.2

0.4

0.6

0.8

1.0

MSE

1e

3

SII, STI, and FSI for ICM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for ICM (

s

0 =2,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

0.12

4000

0.24

6000

0.37

8000

0.49

10000

0.61

12000

0.73

14000

0.85

16000

0.98

model evaluations (absolute, relative)

0

1

2

3

4

MSE

1e

4

SII, STI, and FSI for ICM (

s

0 =1,

d

=14)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000 12000 14000 16000

model evaluations

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

MSE

SII, STI, and FSI for SOUM (

s

0 =2,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000

12000

14000

16000

model evaluations

0

1

2

3

4

5

MSE

1e

1

SII, STI, and FSI for SOUM (

s

0 =2,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000 12000 14000 16000

model evaluations

0.00

0.20

0.40

0.60

0.80

1.00

Pr@10

SII, STI, and FSI for SOUM (

s

0 =2,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

2000

4000

6000

8000

10000 12000 14000 16000

model evaluations

0.0

0.5

1.0

1.5

2.0

MSE

1e

1SII, STI, and FSI for SOUM (

s

0 =1,

d

=30)

Method

SHAP-IQ

Baseline

Order

FSI

SII

STI

it

is

a

gr

##ues

##ome

can

##ni

##bal

movie

but

it

is

not

bad

if

you

like

hannibal

you

will

love

this

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.25

0.30

n-SII values

n-SII values for a sentence provided to the LM

Order 1

it

is

a

gr

##ues

##ome

can

##ni

##bal

movie

but

it

is

not

bad

if

you

like

hannibal

you

will

love

this

1.0

0.5

0.0

0.5

1.0

1.5

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

it

is

a

gr

##ues

##ome

can

##ni

##bal

movie

but

it

is

not

bad

if

you

like

hannibal

you

will

love

this

4

2

0

2

4

6

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

it

is

a

gr

##ues

##ome

can

##ni

##bal

movie

but

it

is

not

bad

if

you

like

hannibal

you

will

love

this

30

20

10

0

10

20

30

40

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

i

have

never

forgot

this

movie

all

these

years

and

it

has

remained

in

my

life

0.3

0.2

0.1

0.0

0.1

n-SII values

n-SII values for a sentence provided to the LM

Order 1

i

have

never

forgot

this

movie

all

these

years

and

it

has

remained

in

my

life

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

i

have

never

forgot

this

movie

all

these

years

and

it

has

remained

in

my

life

4

2

0

2

4

6

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

i

have

never

forgot

this

movie

all

these

years

and

it

has

remained

in

my

life

20

10

0

10

20

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

absolutely

nothing

is

red

##eem

##ing

about

this

total

piece

of

trash

10

5

0

5

10

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

i

actually

liked

the

ending

even

though

it

did

not

make

a

lot

of

sense

15

10

5

0

5

10

15

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

however

i

really

liked

the

ending

so

much

i

actually

smiled

and

cried

tears

of

joy

i

felt

good

20

10

0

10

20

30

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

it

is

a

gr

##ues

##ome

can

##ni

##bal

movie

but

it

is

not

bad

if

you

like

hannibal

you

will

love

this

30

20

10

0

10

20

30

40

n-SII values

n-SII values for a sentence provided to the LM

Order 1

Order 2

Order 3

Order 4

E Further theoretical Results for the Shapley Value

In this section, we give two additional results for the special case of the SV . We explicitly state the inverse of the covariance matrix from [8] and further present a simplified representation of SHAP-IQ, if applied on the SV, which aligns with Theorem 4.4.

E.1 Explicit Inverse of Covariance Matrix of Unbiased KernelSHAP

The covariance matrix has been explicitly computed in [8, Appendix A]. In this section, we provide the explicit form of the inverse A -1 . This inverse admits the same structure and is a central element in the explicit calculation of U-KSH as a weighted sum, which is linked to SHAP-IQ.

Proposition E.1 (Explicit covariance matrix [8]) . For the covariance matrix it holds

with i, j ∈ D and constants

Proof. The proof is given in [8, Appendix A].

We have for ∑ d -1 k =1 1 k ( d -k ) = 2 h d -1 , where h n is the n-th harmonic number. Furthermore µ 1 -µ 2 = 1 2 h d -1 . We now give the explicit form of A -1 .

Proposition E.2 (Explicit inverse of covariance matrix) . Let A := E [ ZZ T ] . Then, we have an explicit form of the inverse A -1 as

with constants

Proof. The proof follows directly from Proposition E.1 and Lemma B.2.

E.2 Simplified Representation of SHAP-IQ for the Shapley Value

In this section, give an explicit form of SHAP-IQ for the SV that admits a similar form as the SV representation in Theorem 4.4. We consider the SV weights m ( t ) := ( d -t -1)! t ! d ! .

Proposition E.3 (SHAP-IQ for SV) . For SHAP-IQ with p ( T ) ∝ µ ( t ) and sampling order k 0 = 1 , it holds

where h n is the n -th harmonic number.

Proof. Recall the definition of SHAP-IQ of order 1

with p ( T k ) := µ ( t k ) /R ∝ µ ( t k ) . We proceed by rewriting γ s m ( t k , | T k ∩ { i }| ) = γ s m ( t k , 1 ( i ∈ T k )) for T k ∈ T 1 as

Hence,

For the normalizing constant, we have

which finishes the proof.

F Approximation Methods for the SV

There are two prominent representations of the SV, which are used for sampling-based approximation. Both allow to update all SVs simultaneously with one sample as well as maintaining the efficiency property.

F.1 Permutation-based (PB) Approximation

Permutation-based (PB) approximation was introduced for SV [5]. It is based on the observation that the marginal contributions δ ν i ( T ) = ν ( T ∪{ i } ) -ν ( T ) can be computed by using permutations π ∈ S D of D and ν ( u -i ( π )) -ν ( u + i ( π )) , where u -i ( π ) , u + i ( π ) are the sets that consist of all elements preceding i in π with and without i , respectively. For for each subset T ⊆ D \ { i } of size t there are exactly t !( d -t -1)! permutations with T = u -i ( π ) and thus

This expectation can be efficiently approximated by sampling π ∼ unif ( S D ) and using a Monte Carlo estimate for the expectation. As ∑ i ∈ D δ ν i ( u -i ( π )) = ν ( D ) -ν ( ∅ ) for arbitrary permutations π , the efficiency constraint is maintained throughout the sampling procedure. The Monte Carlo estimates allows to apply well-established statistical results to obtain bounds on the approximation error [4].

F.2 Kernel-based (KB) Approximation

Kernel Shapley Additive Explanation Values [26], short KernelSHAP (KSH), and Unbiased KernelSHAP (U-KSH) [8] make use of the representation of the SV as the solution to a constrained quadratic optimization problem [6]

with ν 0 ( T ) := ν ( T ) -ν ( ∅ ) , T k := { T ⊆ D : k ≤ t ≤ d -k } and µ ( t ) := 1 d -1 ( d -2 t -1 ) -1 . This quadratic optimization problem can be solved explicitly using the weighted least square solution

where Z ∈ { 0 , 1 } 2 d × d is a row-wise binary encoding of all subsets of T ⊆ D , W ∈ R 2 d × 2 d is a diagonal matrix with the subset weights µ and y consists of the evaluations of ν 0 ( T ) for each subset. To include the optimization constraint, the (otherwise undefined) weights µ ( d ) , µ (0) of D and ∅ are set to a high positive constant. Solving (5) still requires 2 d model evaluations and thus KSH [26] approximates I SV by considering (5) as an expectation

with p ( T ) ∝ µ ( t ) . Note that the optimization problem is invariant in terms of scaling. This expectation is the approximated similarly to SHAP-IQ, by computing high and low subset sizes explicitly and using Monte Carlo integration for the center sizes. The KSH estimator is difficult to analyze and it is only known that it is asymptotically unbiased [46].

KSH constructs a collection of subsets by determining a sampling order k 0 , such that subsets with k 0 ≤ t ≤ d -k 0 are sampled from p ( T ) ∝ µ ( t ) and for t < k 0 or t > d -k 0 all possible subsets are used. The value of k 0 is thereby found by successively comparing the expected number of subsets with the total number of subsets of that size. As the number of subsets ( d t ) for fixed d is a symmetric

log-concave sequence of positive terms, it has a maximum at the middle term(s) ⌊ d 2 ⌋ , ⌈ d 2 ⌉ and grows monotonically and symmetrically as ( d t ) = ( d d -t ) towards this maximum from both sides. Thus, the implementation starts the comparison at k 0 = 0 and iteratively increases the k 0 candidate.

Peer reviews:
comments: SUMMARY:
This paper provided an unified perspective to several existing Shapley-based interaction attribution methods as Cardinal Interaction Index (CII), in addition provide a more efficient sampling based approximation method. Connection between their formulation with unbiased KernelSHAP is established and they showed that their formulation can also improve the standard SHAP algorithm.

STRENGTHS:
The unified formulation of several interaction attribution methods as CII is novel to the best of my knowledge in the X-AI literature. The authors made their claims and novelty very clear as well.

WEAKNESSES:
I think the content of the paper is very well written so not much weakness to comment. One thing that tricked me a bit is in line 33-34 when first mentioned CII it would be nice to have a citation to show that this is not your original contribution. Also in line 129 the \mathcal{S}_{s_0} notation is not defined properly so that stopped my flow a bit too.

QUESTIONS:
Not really from my side.

LIMITATIONS:
Not really from my side. Potential improvements are stated in the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-25

comments: SUMMARY:
This paper presents a unified approximation technique for Shapley interaction indices. It reformulates the general form of various Shapley interaction methods (Shapley interaction index, Shapley-Taylor, Faith-Shap) from a sum involving the set in question to a sum not involving the set, but instead a computation involving the method weights. This allows one call of the function to be a Monte Carlo integration sample of every set of inputs the method is calculating a value for. Theoretical guarantees of unbiasedness and efficiency are provided, as well as probabilistic error bounds.

STRENGTHS:
Clear writing, very easy to follow and presented plainly.
Interesting and practical results which apply to multiple methods.
improvements over baseline is apparent in experiments.
The results are general enough that they are likely to apply to future methods.

WEAKNESSES:
Faith-Shap not fully addressed. But this seems to be in part to the inherent differences of that method.

QUESTIONS:
-Could you clarify, is this method an application of a technique used to approximate Shaley Values, but applied to interaction indices? If so, would you point out where this is is stated. If not, could you elaborate on the novel contributions?

Minor issues:
(153) Please define $\mathcal{G}_D$.
(130-132)_ Please clarify, this secion is dificult to follow.
(174) - the commas and grammar are off in this sentence.

LIMITATIONS:
Application to Faith-Shap as mentioned above.
The method applies to to a type of interaction form that is fairly general.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method, SHAPley Interaction Quantification (SHAP-IQ), for efficiently computing Shapley interaction indices in the context of explainable artificial intelligence (XAI). Shapley interaction indices extend the concept of Shapley values, a method from cooperative game theory, to measure the contribution of interactions between features in a model. The authors propose SHAP-IQ as a unified sampling-based approximation method that can be applied to any Cardinal Interaction Index (CII). They provide theoretical guarantees for SHAP-IQ, showing that it is unbiased and consistent, and provide a general approximation bound. They also discuss the computational complexity of SHAP-IQ and how it compares to other methods. The paper also presents a new representation for the Shapley value (SV) and shows that in the special case of single feature subsets, SHAP-IQ provides novel insights into the SV and greatly simplifies the calculation of Unbiased KernelSHAP (U-KSH). The authors conducted multiple experiments to illustrate the approximation quality of SHAP-IQ compared to current baseline approaches. They compare the baseline methods with SHAP-IQ using different evaluation metrics. The paper concludes with an appendix that provides further theoretical and experimental results for SHAP-IQ, including additional results for specific CIIs such as SII, n-SII, STI, and FSI. The appendix also includes all proofs of theoretical results from the main paper, further insights into the implementation of SHAP-IQ, and additional experimental results. In summary, this paper presents a novel method for computing Shapley interactions, provides theoretical guarantees for this method, and demonstrates its effectiveness through a series of experiments.\n\nSTRENGTHS:\n**Originality**: The authors propose a novel method, SHAPley Interaction Quantification (SHAP-IQ), for computing Shapley interactions for arbitrary cardinal interaction indices (CII). This is a unique contribution to the field of explainable artificial intelligence (XAI), particularly in the context of black box models. The paper not only presents the new method but also includes a detailed comparison with existing methods, an exploration of its theoretical properties, and a demonstration of its practical application. This comprehensive approach strengthens the authors' claims about the effectiveness of SHAP-IQ.\n\n**Quality**: The paper provides a rigorous theoretical foundation for SHAP-IQ, including proofs of its properties such as being unbiased and consistent. The authors also provide a general approximation bound for SHAP-IQ.\n\n**Clarity**: The paper is well-structured and clearly written. The authors provide a clear explanation of their method and its theoretical underpinnings, making it accessible to readers with a background in the field. The appendix provides additional theoretical and experimental results, further supporting the claims made in the main paper. It also includes all proofs of theoretical results, providing transparency and allowing readers to fully understand the theoretical foundation of SHAP-IQ.\n\n**Significance**: The authors demonstrate the practical application of SHAP-IQ through a series of experiments. They show that it can be applied to different machine learning models and datasets, and that it outperforms existing methods in terms of approximation quality and computational efficiency. This suggests that SHAP-IQ could have a significant impact on the practice of XAI.\n\nWEAKNESSES:\nThis paper has no obvious weakness. The main concern from me is the reproducibility of the experiments, which seems like requiring too many computational resources.\n\nQUESTIONS:\nCould the authors discuss some future works about the SHAP-IQ?\n\nLIMITATIONS:\nYes, the authors have discussed limitations well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new sampling-based approximation method called Shapley interaction quantification(SHAP-IQ) for the cardinal interaction index(CII), which is a generalization of the Shapley interaction index. To this end, the authors give a novel representation, and showed that SHAP-IQ is an unbiased and provide a general  approximation bound for the quality and variances of the point estimates. Finally, they demonstrated that SHAP-IQ provides the effectiveness and nice explanations for language model and image classification models.\n\nSTRENGTHS:\n- Provide a unified approach for CII, which is a generalization of unbiased KernelSHAP, and so on. \n- SHAP-IQ is an unbiased and its quality and variance are guaranteed to bound theoretically.\n- SHAP-IQ empirically provides lowest mean square error compared with other baseline algorithms.\n\nWEAKNESSES:\n- The results are solid, but they are not particularly surprising. The contributions are somewhat limited to achieve an unbiased method and theoretically bounding the accuracy.\n\n- Shapley interaction index implies that the number of features to be observed exponentially increases, which in turn may make interpretation more difficult. In some cases, multiple sets of features may have similar importance scores. In such situations, selecting specific sets of features may not be done in a naive way. It would be better to provide a measure to choose them or an approach to reduce the number of feature subsets to be observed.\n\nQUESTIONS:\n- In Figure 2, how are the top-k interactions determined? Are they based on the largest absolute value or some other criterion?\n- How should the illustration on the left of Figure 3 be understood?  Each word is assigned both positive and negative values. What does this mean?\n\nLIMITATIONS:\nYes.\n\nBelow are the comments for improvements.\n- SOUM is known as the induced subgraph game[1]. The interaction subsets correspond to hyperedges of a graph G and  a is its weight. The value of a subset T is given by the summation of edge weights of a subgraph induced by T.\n- In Definition 4.2, K -> k_0? or K seems not be defined.\n\n[1] Deng, Xiaotie, and Christos H. Papadimitriou. 1994. “On the Complexity of Cooperative Solution Concepts.” Mathematics of Operations Research 19 (2): 257–66.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Context.** This paper aims at quantifying the contribution of a set of features to the prediction of a machine learning model by leveraging the Shapley value (SV), a principled tool coming from cooperative game theory. Previous works either focused on (i) estimating the contribution of a single feature (via feature importance scores) or on (i) estimating the contribution of a group of features (via feature interaction scores) through different extensions of the SV to features interaction (e.g. SII, STI or FSI). As of today, no unified framework or methodology to compute feature interaction scores has been proposed. This paper tries to fill this gap.\n\n**Contributions.** The authors made the following set of contributions.\n* They propose a novel perspective and representation of the cardinal interaction index, agnostic to the size of the features' group to be valued. This representation leads to the SHAP-IQ function unifying previously proposed feature interaction scores.\n* They propose a theoretical analysis of SHAP-IQ with both statistic properties (bias, consistency) and non-asymptotic theoretical guarantees in the form of high-probability deviation bounds.\n* They illustrate the benefits of their approach on several experiments.\n\nSTRENGTHS:\n* The paper and contributions are timely and of interest for the ML community.\n* The paper is clear and well-written.\n* I did not fully check in details the Appendix but the first derivations (first 5 pages) in the Appendix are correct and well detailed.\n* The related work part spanning both the Introduction and Section 3 is clear and relevant. I appreciated the links made between SHAP-IQ and previous works.\n* SHAP-IQ (based on Monte Carlo integration) allows to re-use the same sample for all groups of features.\n* Experiments are convincing and well-chosen.\n\nWEAKNESSES:\n* Theorem 4.3 is quite classical since SHAP-IQ stands for a Monte Carlo approximation and proofs techniques to obtain high-probability deviation bounds via Chebyshev inequalities have been commonly used in works on Shapley value. Could the authors go beyond this simple theoretical analysis and provide more insights on the impact of key quantities of the problem (e.g. $\\nu_0$ and $S$)?\n* Full computational complexity and benchmark with competitors: The computational complexity of SHAP-IQ depends on several things: number of Monte Carlo draws, finding the right sampling order $k_0$, pre-computing of the weights $\\gamma_s^m$,... Could the authors clarify the full cost and compare it to previous approaches competing with SHAP-IQ?\n\nQUESTIONS:\nSee previous section.\n\nLIMITATIONS:\nYes, done at the end of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}