SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3Our representation generalizes a result for SII [16, Table 3] to functions with  $\nu (\emptyset)\neq 0$  and the class of CIIIs.
weights  $\gamma_s^m (t,k)$  used for the updates can be efficiently precomputed. The updating process can be implemented efficiently using Welford's algorithm [45], where estimates have to be maintained for all interactions sets, i.e.  $d_{s_0}$  in total. In contrast to KB approximation, which requires to solve a weighted least square optimization problem with  $d_{s_0}$  variables, the computational effort per interaction increases linearly for SHAP-IQ. Furthermore, SHAP-IQ even allows to update selected interaction estimates, whereas, for instance, KB approximation for FSI requires to estimate all interactions. For more details on the implementation and computational complexity of the baseline methods, we refer to the appendix.
4.2 SHAP-IQ for the Shapley Value
In this section, we show that SHAP-IQ, in the special case of single feature subsets  $s_0 = 1$ , yields novel insights into the SV. Furthermore, SHAP-IQ corresponds to U-KSH and greatly simplifies its calculation. Utilizing Theorem 4.1, we find a novel representation of the SV for every feature  $i \in D$ .
Theorem 4.4. With  $c_{1}(i) = \frac{\nu_{0}(D)}{d}$  the SV is  $I^{SV}(i) = c_{1}(i) + \sum_{T\in \mathcal{T}_{1}}\nu_{0}(T)\mu (t)\left[\mathbf{1}(i\in T) - \frac{t}{d}\right]$ .
SHAP-IQ admits a similar form (see appendix) and corresponds to U-KSH  $\tilde{I}_U^{\mathrm{SV}}$ .
Theorem 4.5 (SHAP-IQ simplifies U-KSH). For  $p(T) \propto \mu(t)$  it holds that  $\hat{I}_U^{SV} = \hat{I}_1^m$ .
Theorem 4.5 implies that the U-KSH estimator can be computed using the SHAP-IQ estimator, which greatly simplifies the calculation to a weighted sum. The main idea of the proof relies on the observation that not only  $A$  can be explicitly computed, but also  $A^{-1}$ , cf. the appendix.
4.3 The Sum of Interaction Scores and SHAP-IQ Efficiency
In this section, we are interested in the sum of CII scores, which we link to a property of SHAP-  $\mathbf{IQ}$  estimates to maintain the efficiency axiom. By Theorem 4.1, we have  $\sum_{S\in \mathcal{S}_{s_0}}I^m (S) =$ $\begin{array}{r}\sum_{T\subseteq D}\nu_0(T)\sum_{S\in \mathcal{S}_{s_0}}\gamma_s^m (t,|T\cap S|). \end{array}$  For the SV, by Theorem 4.4, this sum is zero for every  $T\in \mathcal{T}_1$  . For higher order CIs, we introduce the following definition.
Definition 4.6. A CII is s-efficient, if  $\sum_{S\subseteq D,|S| = s_0}\gamma_s^m (t,|T\cap S|) = 0$  for every  $T\in \mathcal{T}_{s_0}$ .
Theorem 4.7. SII and STI are  $s$ -efficient. In particular, SHAP-IQ estimates maintain efficiency for  $n$ -SII and STI.
Further, if a CII is s-efficient, then the sum of SHAP-IQ estimates remains constant. Although we did not provide a rigorous statement, it is easy to validate numerically that FSI is not s-efficient. This finding suggests that there are conceptual differences between these indices, that should be further investigated in future work. Using s-efficiency it is also possible to find an explicit formula for the sum of interaction scores for SII, which we give in the appendix.
We conduct multiple experiments to illustrate the approximation quality of SHAP-IQ compared to current baseline approaches. We showcase SHAP-IQ estimates on any-order SII (n-SII), on top-order STI and FSI. For each interaction index, we use its specific approximation method as a baseline. For SII and STI, we use the PB approximation and for FSI the KB approximation, further described in the appendix. We then compute n-SII based on the estimated SII values. We compare the baseline methods with SHAP-IQ using  $p(T) \propto \mu(t)$ . For each iteration we evaluate the approximation quality with different budgets up to a maximum budget of  $2^{14}$  model evaluations. To account for variation, we randomly evaluate the approximation method on 50 randomly chosen instances, further described below. To quantify the approximation quality, we compute multiple evaluation metrics for each interaction order: mean-squared error (MSE), MSE for the top-K interactions (MSE@K) and the ratio (precision) of estimated top-K interactions (Prec@K). The top-K interactions are determined in regards to their absolute value.
4All code and implementations for conducting the experiments can be found at https://github.com/FFmg11/shapiq. Running the experiments required a computational cost of approximately 2000 CPU hours. For more details we refer to the appendix.
Figure 2: Approximation quality of SHAP-IQ and the baseline for orders  $s = 1, 2, 3$  of SII measured by MSE for the LM (left) and Prec@10 for orders  $s = 2, 3$  for the LM (middle) and ICM (right).
Models. For a language model (LM), we use a fine-tuned version of the DistilBERT transformer architecture [34] on movie review sentences from the original IMDB dataset [27, 22] for sentiment analysis, i.e.  $\nu$  has values in  $[-1,1]$ . In the LM, for a given sentence, different feature coalitions are computed by masking absent features in a tokenized sentence. The implementation is based on the transformers API [47]. We randomly sample 50 reviews of length  $d = 14$  and explain each model prediction. For an image classification model (ICM), we use ResNet18 [19] pre-trained on ImageNet [11] as provided by torch [30]. We randomly sample 50 images and explain the prediction of the corresponding true class. To obtain the prediction of different coalitions, we pre-compute super-pixels with SLIC [1, 44] to obtain a function on  $d = 14$  features and apply mean imputation on absent features. For a high-dimensional synthetic model with  $d = 30$ , we use a sum of unanimity model (SOUM)  $\nu(T) := \sum_{n=1}^{N} a_n \mathbf{1}(Q_n \subseteq T)$ , where  $N = 50$  interaction subsets  $Q_1, \ldots, Q_N \subseteq D$  are chosen uniformly from all subset sizes and  $a_1, \ldots, a_N \in \mathbb{R}$  are generated uniformly  $a_n \sim \mathrm{unif}([0,1])$ . Note that the SOUM could also be viewed as an extension of the induced subgraph game [12] for a hypergraph with edges of different order. We randomly generate 50 instances of such SOUMs.
Ground-Truth (GT) Values. For the LM and the ICM we compute the ground-truth (GT) values explicitly using the representation from Theorem 4.1. For the high-dimensional SOUM it is impossible to compute the GT values naively. However, due to the linearity of the CII and the simple structure of a SOUM, we can compute the exact GT values of for any CII efficiently, cf. the appendix.
5.1 Approximation of any-order SII and n-SII scores using SHAP-IQ
In this experiment, we apply SHAP-IQ on SII and compute estimates for the LM and the ICM up to order  $s = 4$ . We then compare the estimates with the baseline using the GT values for each order. The results are shown in Figure 2. We display the MSE for the LM (left) and the Prec@10 for the LM (middle) and ICM (right). We further compute the n-SII estimates by aggregating the SII estimates with  $s_0 = 4$  and visualize positive and negative interactions on single individuals as proposed in [3]. Thereby, interactions are distributed equally among each participating feature, which was justified in [3, Theorem 6]. This representation amplifies the variance of our sampling-based estimator. We thus also present SHAP-IQ without sampling, i.e.  $c_{k_0}$ . The results are shown in Figure 3 (left) and from left to right: GT values, SHAP-IQ, SHAP-IQ without sampling and baseline. Lastly, we illustrate the n-SII scores estimates for  $s_0 = 3$  of a movie review excerpt classified by the LM (right), where the interactions ("is", "not"), ("not", "bad"), and ("ll", "love", "this") yield a highly positive score.
The results show that SHAP-IQ outperforms the baseline methods across different models and metrics. For the n-SII visualization, we conclude that the SHAP-IQ estimator without sampling is preferable, which yields more accurate results than SHAP-IQ and the baseline methods. In general, SHAP-IQ without sampling performs surprisingly strong, and we encourage further work in this direction.
5.2 Approximation of different CIIs using SHAP-IQ
In this experiment, we apply SHAP-IQ on different CIIs, namely SII, STI and FSI. We compute top-order interactions for  $s_0 = 3$  and compare the results with the baselines. Our results are shown in Figure 4 (left) and further experiments and results can be found in the appendix. For the LM,
Figure 3: Visualization of n-SII and  $s_0 = 4$  [3] (left) with (from left to right): GT, SHAP-IQ, SHAP-IQ without sampling, and baseline. Estimated n-SII scores ( $s_0 = 3$ ) for a movie review (right).
Figure 4: Approximation quality for top-order interactions of SII, STI, and FSI of the LM with  $s_0 = 3$  (left) and the SOUM with  $s_0 = 2$  (middle and right).
SHAP-IQ clearly outperforms the baseline for SII and STI. For FSI, SHAP-IQ is outperformed by the KB approximation of the baseline. As SII and STI rely on PB approximation, our results indicate that KB approximation is more effective than PB approximation for this setting, which is in line with the strong performance of KernelSHAP [26] for the SV. However, SHAP-IQ, in contrast to KB approximation, provides a solid mathematical foundation with theoretical guarantees and we now consider a high-dimensional synthetic game, where SHAP-IQ outperforms all baselines.
SHAP-IQ on high-dimensional synthetic models. For the SOUM we compute the average and standard deviation of each evaluation metric for SHAP-IQ and the baselines for pairwise interactions  $(s_0 = 2)$  of each index. Our results are shown in Figure 4 (middle and right) and further experiments and results can be found in the appendix. SHAP-IQ outperforms all baseline methods in this setting, in particular, the KB approximation of FSI that performed strongly in ML context. The experiment highlights that there exists no approximation method that performs universally best.
Runtime Analysis. The runtime of SHAP-IQ is affected by different parameters. The computation of the sampling order  $k_{0}$  is a constant time operation given a number of features (cf. Algorithm 2 in the appendix). While the pre-computation of the weights  $(m_{s})$  scales linearly with the number of features, the additional computational burden is negligible as it does not depend on  $\nu_{0}$ . The main computational cost stems from the model evaluations (access to the value function  $\nu_{0}$ ), which is bounded by a model's inference time. To illustrate the runtime performance, we compare SHAP-IQ with the baseline methods on the LM using different number of model evaluations  $K$ . Figure 5 displays the runtime of SHAP-IQ and the corresponding baseline approaches, including all pre-computations. With increasing  $K$  the runtime complexity scales linearly, but the overhead of SHAP-IQ remains low. Note that the difference in STI can be attributed to less than  $K$  model evaluations, which is required to maintain efficiency, cf. lines 15-16 in Algorithm 6 of the appendix.
We presented SHAP-IQ, a unified approximation algorithm for any-order CIIs with important theoretical guarantees. SHAP-IQ relies on the specific structure in terms of discrete derivatives from Definition 3.3. This representation exists for every interaction indices that fulfills the linearity, symmetry and dummy axiom [17, Proposition 5]. However, for FSI, which has been defined as the solution to the weighted least square problem, a closed-form representation for lower-order interactions in terms of discrete derivatives remains difficult to establish [40, Lemma 70]. This limits the applicability of SHAP-IQ to top-order interactions of FSI, for which this representation is given in [40, Theorem 19]. The FSI baseline performs strongly in ML context, in line with empirical findings for KernelSHAP [26] for the SV. However, the estimator is theoretically not well understood [8] and we have shown that it is not universally best. Moreover, for other CIIs, such as SII and STI, it is unlikely [13] that such an explicit form in terms of a weighted least square problem can be found, which limits the applicability of KB approximation to FSI. SHAP-IQ outperforms the baselines of SII and STI by a large margin, is generally applicable and supported by a solid mathematical foundation.
Figure 5: For each interaction index (SII: solid, STI: dashed, FSI: dashdotted), we compare the run-time (in seconds) of SHAP-  $\mathrm{IQ}$  (pink) compared to baseline estimators (violet) of one instantiation of the LM (16 words) over five independent runs at different levels of K.
How to extend the SV to interactions is an open research question. In this work, we considered CIIs, a broad class of interaction indices, which covers all currently proposed indices, as well as all indices that fulfill the linearity, symmetry and dummy axiom. We established a novel representation of the CII, which we used to introduce SHAP-IQ, an efficient sampling-based approximation algorithm that is unbiased and consistent. For the special case of SV, SHAP-IQ can be seen as a generalization of U-KSH [8] and greatly simplifies its calculation as well as providing a novel representation of the SV. Furthermore, for n-SII and STI, SHAP-IQ maintains the efficiency condition, which is a direct consequence of a specific property, which we coin s-efficiency for CIIs. We applied SHAP-IQ in multiple experimental settings to compute any-order interactions of SII and n-SII, where SHAP-IQ consistently outperforms the baseline method and showcased the applicability of feature interaction scores to understand black-box language and image classification models. SHAP-IQ further benefits from a solid statistical foundation, which can be leveraged to improve the approximation quality.
Future work. Applying SHAP-IQ to real-world applications, such as NLP tasks [43] and genomics [48, 23], could yield valuable insights. However, the exponentially increasing number of interactions requires human-centered post-processing to enhance interpretability for practitioners and ML engineers, e.g. through automated dialogue systems [37]. Further, it would be beneficial to discover the statistical capabilities of SHAP-IQ to provide confidence bounds or approximate interaction scores sequentially. Beyond model-agnostic approximation, model-specific variants could substantially reduce computational complexity. For instance, it is likely that ideas of TreeSHAP [25] for tree-based models can be extended to Shapley-based interactions.
We sincerely thank the anonymous reviewers for their work and helpful comments. We gratefully acknowledge funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation): TRR 318/1 2021 - 438445824. Patrick Kolpaczki is supported by the research training group Dataninja (Trustworthy AI for Seamless Problem Solving: Next Generation Intelligence Joins Robust Data Analysis) funded by the German federal state of North Rhine-Westphalia.
Appendix of "SHAP-1Q: Unified Approximation of any-order Shapley Interactions"
Organisation of the Appendix
We provide further theoretical and experimental results for SHAP-IQ. The appendix is organized as follows: In Appendix A, we formally introduce specific CIs, such as SII, n-SII, STI and FSI and their axiomatic foundation and theoretical results and provide a novel theoretical result for the sum of SII scores. In Appendix B, we provide all proofs of theoretical results from the main paper. In Appendix C, we give further insights into the implementation of SHAP-IQ. In Appendix D, we provide further experimental results, information about the used models, explicit formulas for the SOUM interaction scores, and pseudo-code and information on the computational complexity of baseline implementations. In Appendix E, we provide further theoretical results for the special case of SV, in particular the explicit form of the covariance matrix from [8] and a simplified representation, similar to Theorem 4.4, of SHAP-IQ in this case. In Appendix F, we describe the approximation methods ApproShapley [5] and KernelSHAP [26] for the SV on which our baseline methods are built.
A Shapley-based Interaction Indices and further theoretical Results 15
A.1 Shapley-based Interaction Indices 15
A.2 Explicit Formula for Sum of SII Scores 16
B Proofs 18
B.1 Proof of Theorem 4.1 18
B.2 Proof of Theorem 4.3 18
B.3 Proof of Theorem 4.4 19
B.4 Proof of Theorem 4.5 19
B.5 Proof of Theorem 4.7 21
C Algorithmic Details of SHAP-IQ 23
C.1 Algorithm of SHAP-IQ 23
C.2 Sampling Weights  $q(t)$  23
C.3 Welford's Algorithm 23
D Experiments 25
D.1 Baseline Algorithms for SII, STI, and FSI 25
D.2 Further Information about the Models 28
D.3 Hardware Details and Environmental Impact of the Experiments 29
D.4 Further Experimental Results 29
E Further theoretical Results for the Shapley Value 34
E.1 Explicit Inverse of Covariance Matrix of Unbiased KernelSHAP 34
E.2 Simplified Representation of SHAP-IQ for the Shapley Value 34
F Approximation Methods for the SV 36
F.1 Permutation-based (PB) Approximation 36
F.2 Kernel-based (KB) Approximation 36
A Shapley-based Interaction Indices and further theoretical Results
In this section, we review the axiomatic structures of n-SII, SII, STI and FSI. We further provide an additional result for the sum of SII interaction scores.
A.1 Shapley-based Interaction Indices
We consider  $\mathcal{G}$  as the set of all games  $\nu : \mathcal{P}(D) \to \mathbb{R}$ . In the following, we formalize the axioms for the Shapley interaction indices  $I_{\nu} : \mathcal{P}(D) \to \mathbb{R}$
Definition A.1 (Linearity Axiom [17]).  $I_{\nu}$  is linear, if for any two games  $\nu_{1}, \nu_{2} \in \mathcal{G}$  and any  $S \subseteq D$ , it holds  $I_{\nu_{1} + \nu_{2}}(S) = I_{\nu_{1}}(S) + I_{\nu_{2}}(S)$ .
Definition A.2 (Dummy Axiom [17]). For a dummy player  $i \in D$  for a game  $\nu \in \mathcal{G}$ , i.e. constant contribution  $c(i)$  added to any coalition  $\nu(T \cup \{i\}) = \nu(T) + c(i)$ , then for every  $T \subseteq D \setminus \{i\}$ , the dummy axiom requires  $I_{\nu}(S \cup \{i\}) = 0$  for any  $S \subseteq D \setminus \{i\}$ . That is, a dummy player has no interaction with any coalition.
Definition A.3 (Symmetry Axiom [17]).  $I_{\nu}$  is said to fulfill the symmetry axiom, if for any permutation  $\pi$  on  $D$  it holds  $I_{\nu}(S) = I_{\pi \nu}(\pi S)$ , where  $\pi \nu(\pi S) \coloneqq \nu(S)$  and  $\pi S \coloneqq \{\pi(i) : i \in S\}$  changes the ordering of the players.
Definition A.4 (Recursive Axiom [17]).  $I_{\nu}$  fulfills the recursive axiom, if for any  $S \subseteq D$  with  $|S| > 1$  and any game  $\nu \in \mathcal{G}$
$$
I (S) = I _ {\nu_ {[ S ]}} ([ S ]) - \sum_ {K \subsetneq S, K \neq \emptyset} I _ {\nu^ {D \backslash K}} (S \backslash K),
$$
where  $\nu_{[S]}$  is the game, where all players in  $S$  is considered as one player, and  $\nu^{D\setminus K}$  is a game defined on the subset of players  $D\setminus K$ .
The recursive axiom defines higher order interactions using lower order interaction. For pairwise interactions it can be stated as  $I_{\nu}(ij) = I_{\nu_{[ij]}}([ij]) - I_{\nu^{D \setminus \{j\}}}(i) - I_{\nu^{D \setminus \{i\}}}(j)$ , i.e. the pairwise interaction is the difference of the value for the reduced player  $[ij]$  and the individual player values for the reduced game.
Definition A.5 (Shapley Interaction Index [17]). The Shapley interaction index (SII) is the unique interaction index that satisfies the linearity, dummy, symmetry and recursive axiom, where the values for  $|S| = 1$  correspond to the Shapley value. It can be represented as a CII as
In contrast to the SV, the SII does not yield an efficiency property, which is desirable in ML context. The efficiency axiom was therefore introduced for interactions. The following axioms rely on a maximum interaction order  $s_0$ , where the values of the interaction indices change for different maximum interaction orders.
Definition A.6 (Efficiency Axiom [39]). For all  $\nu \in \mathcal{G}$ , it holds  $\sum_{S\in \mathcal{S}_{s_0}}I_\nu (S) = \nu (D) - \nu (\emptyset)$ .
The efficiency axiom is an extension of the SV efficiency axiom and requires that all interaction scores up to the maximum order  $s_0$  sum up to  $\nu(D) - \nu(\emptyset)$ . For SII there exists a unique recursive aggregation, such that the efficiency axiom is fulfilled. This novel interaction index is referred to as n-Shapley Values (n-SII) [3].
Definition A.7 (n-Shapley Values (n-SII) [3]). Given  $s_0$  the  $n$ -Shapley Values (n-SII) are defined as
$$
I_{s_{0}}^{n\text{-} SII}(S):= \left\{ \begin{array}{ll}I^{SII}(S), & for|S| = s_{0}\\ I_{s_{0} - 1}^{n\text{-} SII}(S) + B_{d - |S|}\sum_{\substack{K\subseteq D\setminus S\\ k + s = d}}I^{SII}(S\cup K), & for|S| <   s_{0}. \end{array} \right.
$$
Besides n-SII, there have been two axiomatic extensions to CIIs that directly require the efficiency axiom together with the linearity, symmetry and dummy axiom. However, unlike for the SV, it is not sufficient to require efficiency for a unique interaction index. The Shapley Taylor Interaction Index (STI) further specifies the interaction distribution axiom, which then yields a unique index [39].
Definition A.8 (Interaction Distribution Axiom [39]). For an interaction function  $\nu_{T}$  parametrized by  $T\subseteq D$  it holds  $\nu_{T}(S) = 0$ , if  $T\subsetneq S$  and a constant value  $\nu_{T}(S) = c$  for  $T\subseteq S$  and  $c\in \mathbb{R}$ . The interaction distribution axiom requires for all  $S\subseteq D$  with  $S\subsetneq T$  and  $s < s_0$  that  $I_{\nu_T}(S) = 0$ .
Definition A.9 (The Shapley Taylor Interaction Index [39]). The Shapley Taylor interaction index (STI) is the unique interaction index that satisfies the linearity, dummy, symmetry, efficiency and interaction distribution axiom.
The STI yields a unique interaction index by introducing the interaction distribution axiom, which favors the maximum order interactions as discussed in [40]. It was thus argued that instead the representation of the interaction index as a solution to a weighted least square problem is preferable [40], which yields the Faith-Interaction Index.
Definition A.10 (Faith-Interaction Index [40]). I is called a Faith-interaction index if it can be expressed as
$$
I = \arg \min  _ {\beta \in \mathbb {R} ^ {d s _ {0}}} \sum_ {T \subseteq D: \mu (T) <   \infty} \mu (T) \left(\nu (T) - \sum_ { \begin{array}{c} T \subseteq S \\ t \leq s _ {0} \end{array} } \beta (S)\right) ^ {2} \tag {2}
$$
$$
s. t. \nu (T) = \sum_ {S \subseteq T, t \leq s _ {0}} \beta (S), \forall T: \mu (T) = \infty .
$$
Definition A.11 (Faithful Shapley Interaction Index [40]). The Faithful Shapley interaction index (FSI) is the unique faith-interaction index that satisfies the linearity, dummy, symmetry and efficiency axiom.
While n-SII, SII, STI and FSI offer different ways of characterizing an interaction index, it was shown in [17] that every interaction index satisfying the linearity, symmetry and dummy axiom admits a CII representation.
Proposition A.12. Every interaction index satisfying the linearity, symmetry and dummy axiom can be represented as a CII
$$
I ^ {m} (S) := \sum_ {T \subseteq D \backslash S} m _ {s} (t) \delta_ {S} ^ {\nu} (T).
$$
Furthermore, the weights  $m_{s_0}$  for the top-order interaction indices, i.e.  $s = s_0$  are defined as
$$
m _ {s _ {0}} ^ {S I I} (t) := \frac {(d - t - s _ {0}) ! t !}{(d - s _ {0} + 1) !},
$$
$$
m _ {s _ {0}} ^ {S T I} (t) := s _ {0} \frac {(d - t - 1) ! t !}{d !}
$$
$$
m _ {s _ {0}} ^ {F I I} (t) := \frac {(2 s _ {0} - 1) !}{((s _ {0} - 1) !) ^ {2}} \frac {(d - t - 1) ! (t + s _ {0} - 1) !}{(d + s _ {0} - 1) !}.
$$
The definitions for lower order interactions can be found in [17] for SII, in [39] for STI and in [40] for FSI. Note that for FSI a closed-form solution for lower-order interactions in terms of discrete derivatives remains unknown [40, Lemma 70].
Proof. For the proofs, we refer to the corresponding paper of each index. The general statement is proven in [17, Proposition 5].  $\square$
A.2 Explicit Formula for Sum of SII Scores
Using s-efficiency, it is easy to calculate the sum of SII scores, which provide an explicit representation of a formula considered in [31, Theorem 4.2].
Proposition A.13 (Sum of SII Scores). For SII it holds