InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantly
Authors: James Enouen
Yan Liu
Venue: ICLR 2025 Poster (2025)
Keywords: additive models
GAM
SHAP
Shapley
Primary area: interpretability and explainable AI

Abstract: In recent years, the Shapley value and SHAP explanations have emerged as one
of the most dominant paradigms for providing post-hoc explanations of blackbox models. Despite their well-founded theoretical properties, many recent works
have focused on the limitations in both their computational efficiency and their
representation power. The underlying connection with additive models, however,
is left critically under-emphasized in the current literature. In this work, we find
that a variational perspective linking GAM models and SHAP explanations is able
to provide deep insights into nearly all recent developments. In light of this connection, we borrow in the other direction to develop a new method to train interpretable GAM models which are automatically purified to compute the Shapley
value in a single forward pass. Finally, we provide theoretical results showing the
limited representation power of GAM models is the same Achilles’ heel existing
in SHAP and discuss the implications for SHAP’s modern usage in CV and NLP.

Published as a conference paper at ICLR 2025
INSTASHAP: INTERPRETABLE ADDITIVE MODELS EXPLAIN SHAPLEY VALUES INSTANTLY
James Enouen
Yan Liu
Department of Computer Science University of Southern California Los Angeles, CA enouen@usc.edi
Department of Computer Science University of Southern California Los Angeles, CA yanliu. cs@usc. edu
ABSTRACT
In recent years, the Shapley value and SHAP explanations have emerged as one of the most dominant paradigms for providing post-hoc explanations of black- box models. Despite their well-founded theoretical properties, many recent works have focused on the limitations in both their computational efficiency and their representation power. The underlying connection with additive models, however, is left critically under-emphasized in the current literature. In this work, we find that variational perspective linking GAM models and SHAP explanations is able to provide deep insights into nearly all recent developments. In light of this con- nection, we borrow in the other direction to develop new method to train inter- pretable GAM models which are automatically purified to compute the Shapley value in single forward pass. Finally, provide theoretical results showing the limited representation power of GAM models is the same Achilles' heel existing in SHAP and discuss the implications for SHAP's modern usage in CV and NLP.
INTRODUCTION
Since their introduction into machine learning. Shapley values have had a meteoric rise within the space of model explanation. The principled axioms of Shapley (Shapley 1953) and the casy-to-use framework of SHAP (Lundberg & Lee 2017) have led to their widespread adoption when compared with alternatives in gradient-based and black-box explanation methods. The developments which then followed quickly pushed beyond tabular datasets into higher dimensional data like computer vision and natural language, with abundant research investigating how to improve the speed and efficiency Shapley values across these various high-dimensional domains (Covert et al. 202T
Mosca et al.] 2022 Jethani et al.) 2022 Covert et al. 2023 Enouen al. 2024). In recent years, however, some lines of work have identified specific application scenarios where the Shapley value provably guaranteed the Shapley value are instead done in vain (Bilodeau et al.] 2022 Huang & Marques-Silve (003)). Unfortunately, many of these critiques have been made piecemeal without an overall sense of their underlying causes. In contrast, this work takes the perspective that SHAP's issues of explanation speed and explanation power can all be viewed under the same lens through the underlying connec- tion with additive models and feature interactions.
fail, perhaps begging the question of whether such works improving on
In this work, we find that all of the most recent development in the SHAP value, like the practical improvements of FastSHAP (Jethani et al. 2022) and the theoretical advancements of FaithSHAP (Tsa al. 2023), can be unified and more easily understood by using a functional perspective using additive models. In particular, the amortization scheme introduced by FastSHAP (Jethani) ct al. 2022) builds on the least squares formulation of the Shapley value (Charnes et al. 1988] Lundberg & Lee. 2017) by training a global approximator for the Shapley value, which can each be viewed as fitting a pointwise and a global additive model, respectively. Tsai et al. (2023) instead extends this least squares characterization to the bivariate and higher-order interactions to yield richer understanding of the model to explain, drawing the same parallels with higher-order additive models.

---

Published as a conference paper at ICLR 2025
Scenario [tabular dota)
Rastinal
(CV data)
Sconario B:
GAM
GAM
Insta SHAP
Insta SHAP
Leglasation
Brukdke
DNN
Figure 1: The fundamental correspondence between SHAP and GAM used practically to distin- guish two unique scenarios. In scenario A, such as simpler tabular data, GAM models can achieve SOTA performance and their SHAP explanations align with SHAP explanations of blackbox mod- els. In modern ML applications like computer vision, we have scenario B, where there is a gap between GAM and DNN performance in practice. This means that either: (#1) we cannot train GAMs as well as other deep neural networks; or (#2) there is an overcredulous trust of SHAP in these domains.
All of these extensions and more can be though of as special cases of training additive models when considered from the functional ANOVA perspective (Sobol (00T), allowing not only for their sim- ple combination but a greater understanding of the underlying mechanisms overall. We further find that there is a deep variational connection which underscores all such recent developments between the fundamentally interrelated SHAP explanation and GAM model, paralleling the algebraic case studied in [ordt & von Luxburg (2023) and building on various works in the GAM literature study- ing purification of additive models (Hooker 2007 Hart & Gremaud 2018 Lengerich et al. 2020 Sun et al. 2022).
Overall, we set out to emphasize these fundamental connections between SHAP and GAM (as well as RHith-SHAP-k and GAM-k) across all possible correlated input distributions, where previous work only addressed the case of independent input variables (Bordt & von Luxburg 2023). We use these theoretical advances in the variational equation to provide simple check of whether SHAP is trustworthy by requiring GAM model can be trained to the same accuracy (as depicted in Figure [] This development allows for deeper insights and practical checks into SHAP's application in important ML domains like CV and NLP where input features heavily correlated. Further, by developing a new technique to automatically purify additive models to return the corresponding SHAP values, we simultaneously solve a longstanding problem from the GAM literature as well as demonstrate practical advantages of our method InstaSHAP over the existing FastSHAP. We envision our major contributions follows:
Establishment of the variational formulation of Shapley additive models alongside the ex- isting variational formulations of GAM and functional ANOVA, solving the case of depen- dent variables which was left open in Bordt & von Luxburg (203)..
Introduction of practical training method for GAM models via alternative loss function with output masking, automatically solving the problem of GAM purification and allowing for 'instant' access the Shapley values.
Theoretical insights into the real-world application of SHAP and many comparative exper- iments across synthetic and of whether Shapley values are trustworthy for ML in practical environments.
domains of interest, helping to lay bare the question
BACKGROUND
Let F: : - be function representing a machine learning model which maps from input space X' C Rd to output space y R", where there d input features and output features. We will use [d] == (1...., d} to represent the set of input features and C [d] to represent a subset of the input features. We also write the set of all such subsets, the powerset, P((d) = (0, 1)d.
2

---

Published as a conference paper at ICLR 2025
2.1 EXPLAINING BY REMOVING
A very important aspect of removal-based explanations like LIME SHAP (Ribeiro et al.) 2016 Lundberg & Lee 2017) is the method of feature removal (Sundararajan & Najmi 2020; Covert et al, 2021]. We review the three most popular removal approaches: replacing the feature with a reference value (baseline), integrating over the marginal distribution of the feature (marginal), or integrating over the conditional distribution of the feature (conditional). When applied an explanation method like the Shapley value, these result in the corresponding: baseline Shapley, marginal Shapley, or
conditional Shapley (Sundararajan & Najmi 2020; Janzing al. 2020,
Frye et al. 2021).
First, consider an input example one would like to explain E A' and a subset of the features which one would like keep S S [d] as part of the model. We may then compare against a baseline value F E Rd and define the baseline value as B3 as below. We may also choose distribution of baselines p(z) over the input space X, which allows us to define both the marginal projection and the conditional projection, NG and Mp.
[Ba o FI = [V o (r, S) = =Ex_s Bxx->>((Xs))
F(rs, F-s)
Firss X-s)
(2)
[Mp (x, S) = -Ex_ s(xs=xs)
F(Is, F(Is, X-)}
(3)
We write these three operators as functionals mapping F to a new function f to support our analysis from functional perspective. Historically, the baseline value and marginal value have been the easiest to use in practice because we may directly explain our blackbox F without significant modi- fications. However, since the highlighting of the feffthe-mmanfolld problem Frye al. (202I). has been shown that baseline methods B3 and marginal methods No significantly overemphasize the algebraic structure of the model instead of the statistical structure. If one exclusively interested in the algebraic dependencies of their ML model, the correspondence highlighted herein has already been established Bordt & von Luxburg (002)). Otherwise we hereafter restrict our attention to the conditional expectation using Mp and provide details on further considerations in Appendix A.I
We define a feature attribution method + as taking F(x) and returning a local explanation function ( (x:) for each feature i E [d] on each local input X. Similarly, we define blackbox fea- ture attribution method as instead taking a masked function fxx S) and returning a local explanation function for each feature, [Ø, (x)
We now provide one of the typical definitions of the Shapley value as follows; however, we recom- mend the unfamiliar reader instead waits until the more intuitive Equation 13
SHSHAP
+f(())
SHAP (S). (( -f(fff(5S S 1)
(4)
SC
d 1
n SHAAP (S) x
(5)
d ++1
Here, the Shapley value is defined as the addition and removal of a single feature E [d] across many contexts S C [d] according to the distribution (SHAP-uni((S) where the shorthand = |S| is used. Following the discussion in the previous section, we will in this work always consider the conditional Shapley heond.SHAP := SHAA Mp. Alternative black-box explanations to the Shapley value are discussed further in Appendix A.2
2.2 INTERPRETING BY ADDING
We now introduce the interpretable generalized additive model (GAM) of Hastie & Tibshirani (990).. This model is seen as interpretable because each of the input features have a simple ID relationship with their effect on the output. In this work, we also include the 'zero dimensional" nor- malizing constant fe and often use the term GAMI to emphasize a GAM that only has ID functions.
3

---

Published as a conference paper at ICLR 2025
F (21,..., fu f1(11) +... +.............. +f((aa)
= fo +
Σ f((z)) reId
(6)
This further be generalized GAM2 model (Wahba al. 1994; Lou et al. 2012 2013
Chang et al., 2022) which is still seen as an interpretable model because its 2D functions can still be visualized using heatmap plot.
F (II, za) fo + f1(21) + ... ff(aa) + (12(21.02) + +.. fa- aadd,
fo
Σ EE|d]
f((z))
Σ (i))
Recent research has udditionally focused on addressing the practical considerations associated with training increasingly high-order GAMs (Yang et al.) (2000; Dubey et al. [2022] Enouen & Liu 2022).
For some order k: 2 3, we may define the higher-order GAM-k as:
F (I1, xa) fo
Σ f((()) iE(d)
+
Σ
[s(es) E fs(as)
ScId,18|=
SEIgk
where we write Iça= ( [d] :d] k).
This might immediately raise the question of when to stop adding higher-order functions to our GAM model. Multiple practical works have shown that for tabular data, A: does not have to be too large: GAM-1 and GAM-2 are often sufficient to fit the complexities of the data and achieve state- of-the-art performance across many datasets (Chang al. 2022; Enouen & Liu 2022). The same question for CV or NLP, however, has faced little exploration in previous works. In order to answer this question precisely, however, we instead turn to the functional ANOVA decomposition coming from the field of sensitivity analysis.
2.3 FUNCTIONAL ANOVA
In the literature on sensitivity analysis, we may take any function and completely decompose via its functionnal ANOVA decomposition (Sobol 2001 Hooker 2004):
Fz Ra) = Σ fs(#s)
(9)
SCId
Although there are many possible choices of fs which could obey this equation, we may define unique decomposition via the conditional projection from Section 2.1
is(us)
(-1))1S-PI s(x, TCS
(-11))S1- [M, (x. T)
(10)
TC9
Hereafter, we often follow the sensitivity analysis notation of writing fs(Es) rather than f(x,, S).
This specific choice using conditional projection often called the 'Sobol-Hoeffdinn decomposi- tion. In the case of independent input variables, (Sobol (2001) provides complete understanding of what happens to the variational solution of training any additive model. In particular, the vari- ance the mean squared error is able decompose completely via the decomposition of variance formula:
Varx
Varx Varx [Xs)) SCMd
Σ
Σ Vs SC
(11)
where the orthogonal contributions, := Varx, [Xs) are called the Sobol indices and mea- sure the variance which can be uniquely ascribed each feature interaction S.
Unfortunately, this variational formulation for additive models breaks down for the case of correlated input variables. The best existing altermative in the literature is the Sobol covariances (Rabitz) 2010
Hart& & Gremaud, 2018 which is instead defined Cs Covx (F(X), fs(Xx)) However, these

---

Published as a conference paper at ICLR 2025
(a) Synergistic Interaction
Redundant Interaction (d)
X,Y NV (()) 9)) F(a,w) 3x + By + 7.EM
X,Y .V (()) i)) F(x, 8) Bz By
(b) functional ANOVA
(e) Functional ANOVA
frglit D)
Eagtz,, v)
B (py
(x) g (zr + ju
(2)
fol
B (pu
(4) Shapley 02 (oog)
(
(x, By 30y
Co
(x.g))
Shapley
functions
Axr 20
3y
functions
Figure 2: Simple examples (Gaussian input variables and multilinear response variables) which demonstrate each of the two major types of feature interactions: synergistic interactions and re- dundant interactions. Their full functional ANOVA and exact Shapley functions are additionally calculated and shown. Colored by relevant variable. Note we use æ and instead of 2 and 22.
covariances may result in values which are both positive and negative, conflating the synergistic effects between set of features S and the redundant effects of shared information amongst the same set of features S.
We will still say that statistical feature interaction, S [d], is said to exist whenever
Vs Varx, (fs(X)) 0
(12)
however, for the case of correlated input features, importantly need distinguish between the two major types of feature interactions:
1. feature interaction synergy, where V >> 0 and Cs > 0.
2. feature interaction redundancy, where V > 0 but Cs < 0.
Using this functional ANOVA perspective, we can now write the Shapley value complete func- tion for each variable e [d] using the well known alternative via the synergy or unanimity functions (Shapley) 1953):
fs(zs) S
1 f](x) = Σ
(13)
$2(()
Intuitively, the value which is ascribed to each feature interaction S is uniformly divided amongst each of its constituents S. In Figure can see the easily computed Shapley functions coming from computing the functional ANOVA decomposition, dividing the interaction effects in both the synergistic and the redundant setting. In real-world datasets and in the presence of higher- interactions. it easy to imagine how quickly such effects will compound and conflate one another.
REPRESENTATION POWER OF ADDITIVE MODELS
Before proceeding further with the variational GAM methods we introduce, we find is important to characterize the behavior of SHAP in terms of the functional ANOVA decomposition. In particular, we will do this in the form of an "impossibility theorem" to help cement the correspondence which exists between GAM and SHAP. However, unlike previous works focusing the flaws of SHAP, not only exactly characterize all negative results showing when hypothesis tests are impossible, but consequently characterize all positive results showing exactly when hypothesis tests possible.
5

---

Published as a conference paper at ICLR 2025
3.1 SHAP FUNCTION SPACE
Theorem 1. (SHAP=ANOVA-I) SHAP will succeed on any hypothesis for some hypothesis space H. if and only if H. is completely free of feature interactions (H C H ANOvA).
We leave the details of the proof until Appendix C however, for one direction is relatively straight- forward to see from from Equation 13 that SHAP can succeed if all interaction terms are zero. Con- versely, if some true model F E 76 is not representable by an ANOVA-1 model, then F & ANOVA and hence SHAP is instead obscuring the feature interactions. Importantly, we emphasize that this means SHAP cannot distinguish synergistic feature interactions nor can it distinguish redundant feature interactions. We can additionally show the exact same type of relationship is true for Faith- SHAP-K.
For any & E [d]. Faith-SHAP-K will succeed any hy- pothesis test in some hypothesis space H. if and only if H is free of higher-order features interactions of size (k- + 1) or greater (H C H ANovA).
Theorem 2. (Fathh-SHAP-
This similarly implies that even indices measuring feature interactions will still be forced to blur out the higher-orde order interactions and hence remain limited in their representational capacity. Once again, we emphasize that the reliance of these approaches on the functional ANOVA decomposition means it is not possible for them to distinguish between synergistic interactions and redundant interactions.
3.2 GAM FUNCTION SPACE
Let us now contrast these two results with the representation power of GAM models.
Theorem 3. (ANOVA-IÇGAM-I) The functional space of ANOVA-I representable functions is strict subset of the functional space of GAM-I representable functions (H (H S H.GM).
Any function which is representable by a univariate ANOVA decomposition automatically repre- sentable by GAM model by the subset compliance of the ANOVA decomposition. This inclusion strict in the other direction as soon as there a feature correlation in the input data.
Theorem 4. (ANOVA-KÇGAMAN The functional space of ANOVA-k representable functions is strict subset of the functional space of GAM-k representable functions (H ANoVA & H
Once again, the inclusion of k-dimensional ANOVA functions automatically representable by an arbitrary GAM-k model by the definition of the decomposition. The inclusion is again strict as soon as there a correlation between features in the input data. We save proofs and further discussions for Appendix
3.3 PRACTICAL. INSIGHTS
In conclusion, our results show that a practitioner may evaluate the trustworthiness of SHAP on a given dataset by simply training GAM model the same dataset. If GAM easily match the same accuracy as a blackbox model or easily distill the same predictions as a blackbox model, then this is dataset for which SHAP explanations can generally be trusted. On the other hand, if GAM cannot match the same accuracy as the blackbox model, this means that the practitioner needs to be wary about trusting SHAP values on this dataset. In this second scenario, there are two possible resolutions. For the GAM researcher, resolution of Figure is to train better GAM through the use of more efficient training procedures or through an increase in capacity with additional feature interaction terms. For the SHAP practitioner, resolution 2 of Figure iis to admit that SHAP is likely not a sufficient explanation for this model or dataset.
In many real-world scenarios, it is possible that neither of these extremes is completely true, with the lower bound GAM and upper bound SHAP meeting somewhere in the middle. Nevertheless, in the current literature, this gap is extremely large for practical AI tasks including CV and NLP. In our experiments, we highlight this large gap on a high-dimension Ultimately, the key contribution of this theory is that it provides practical test for researchers to understand task- lask-by-task what are the advantages as well as the limitations of SHAP and GAM approaches.
CV task of bird classification.

---

Published as a conference paper at ICLR 2025
4 SHAPLEY VIA LEAST SQUARES
Kernel SHAP
The Shapley value was first given an optimization- definition or 'variational"
characterization in Charnes al. 1988) as the solution to:
[1()
2
arg min $ER"
Es ESS
Σ
1ii 5) -
(14)
where the distribution over the SHAP kernel
1
d (SIAP(S)) (S) x
(15)
s(d s)
where once again 8 = S|| (contrast this distribution with Equation 5 This formulation was origi- nally utilized by KernelSHAP (Lundberg & Lee 2017) fit local linear model according to the SHAP kernel distribution in order to attain sufficient speed to see use in ML applications.
Fast SHAP This was importantly used more recently by FastSHAP (Jethani et al.) 2022) in order create a functional amortization scheme which fits the same SHAP kernel. They then write the Shapley function as the solution to the following equation!
arg min Bewpp(a) ESSNSSIAr(s) (9) [ - ((EE)) -(())))))()) -od
(16)
where they then fit the functions d; : X' R over the entire input space to automatically generate the Shapley values at test time. This dramatically improves the test-time speed with which SHAP explanations can be generated, overcoming what is often the most major practical limitation to deployment.
The summing over multiple functions to create the predicted output should remind the reader of the structure of GAM-1 additive models. As we saw in our impossibility theorems and as we will later show with InstaSHAP, this functional perspective indeed opens up the possibility to connect with training additive models.
Faith SHAP As we highlight in Theorem it is well known that feature interactions are necessary for explaining more complex functions. Accordingly, many works have tried to extend the Shapley value to be able to handle the effects of feature interactions (Grabisch & Roubens) 1999 Sundarara- jan al., 2020 Bordt & von Luxburg 2023 Fumagalli et al., 2023). Recently, there have been theoretical ad advancements which extend the Shapley value directly using the variational formulation in Tsai et al. 2023). They write their higher-order solution, called Faith SHAP, as:
[[(s)) - Σ 1 5T "1
arg min (Sn PER
(17)
TE
Once again, we can see parallels between the GAM-K model from Equation indeed reduces to the original least squares solution in Equation 14
X
The case of I:
INSTANT SHAP
A simple combination of these two ideas (functional amortization and feature interaction) would lead to an explainer which automatically recovers the two-dimensional or higher-dimensional Faith- SHAP explanations of the target function, while maintaining the practical speedups of FastSHAP:
[|rre:s)-
1(T C - or(e)
arg min Exmp(() d.. Sx
ES-
Σ
(18)
TEML
'To keep the notation eleaner and more similar with other existing works, assume throughout this section that fo which is equivalent centering or normalizing the outputs.

---

Published as a conference paper at ICLR 2025
This functional amortization automatically recovers the Faith-SHAP-k values as defined in Tsai al. 2023 but maintains the practical ad advantages and speedups of FastSHAP from Jethani et al. (0022)
Instead, however, propose to adapt the typical variational equations used to fit GAM models to fall under this same variational Shapley framework. We first make explicit the variational equation used to train GAM models as:
(Bemp(s) (S [|F(E::)-
[]]
arg min 6:2 To
E ΦT(ET)
(19)
TEILIZISA
Accordingly, define the Inst--SHAP-GAM additive model which is trained as:
arg min Exnpte) Es-psaN(()) -
||:0,;))-
E
[]]
(20)
TCALTS
where first we introduce the masked training objective where S are drawn from the Shapley kernel and second we add the masking on each additive component of the GAM to only be included all features of that component unmasked in S.
This new formulation is able to bring novel insights to both the literature on SHAP and the liter- ature GAM. For SHAP, incorporate the low-dimensional GAM bias which is able more accurately model SHAP values when compared with FastSHAP. Additionally, by the explicit mod- eling of interactions, which have yet to develop a method for practical deployment. For the GAM literature, we make progress towards the longstanding goal purification of the shape functions of additive models. In the appendix, we further detail how this extends on the existing GAM literature and its various applications towards the selection of feature interactions under correlated inputs.
able improve the practical feasability of methods like FaithSHAP,
TABULAR EXPERIMENTS
6.1 SYNTHETIC EXPERIMENTS
We construct a simple dataset to test the varying effects of synergistic interactions and redundant interactions in a similar spirit to Figure We provide additional details in the Appendix, but we use simple een-dimensional feature space with an algebraic GAM rank of k* and correlation of In Figure 3 we see the approximation results consistently showing InstaSHAP has better inductive bias than FastSHAP for learning the Shapley values.
Model SMMP-1 Eme (te)), peó 1001
Mede: SHAP-1 Ennrr (ke)), ged. pod.3001
Mide: S4MP1 1 Emrr (k*=2, p=6,000
Mode SHMP Emrr (K*=2 po6.667
Nosbend
hoadende Iuta
10-
200 Number Specha
309
US
253
30
56 159
200
259
309
100
200
390 150 21
50
130
156
Number Specht
Number Spochs
Number Epochs
Figure 3: MSE error of approximations of the model's SHAP values. Since both FastSHAP and InstaSHAP are functional approximations, we report the MSE errors across the epochs of training.
6.2 SYNERGY IN BIKE SHARING
In the bike sharing dataset, we find strong evidence synergistic interaction effect. This dataset predicts the expected bike demand each hour given some relevant features like the day of the week, time of day, and current weather. There total of thirteen different input features predicting a single continuous output variable.
In the case of a multi-layer perceptron trained to predict the bike demand, the normalized mean- squared error (R²) results is 6.59%. Although GAM-1 only achieve an R² of 17.4%, a low- dimensional GAM can achieve an R² of 6.23%. It is well-known that on this dataset there a strong interaction between the hour variable and workday variable (since people's schedules change
8

---

Published as a conference paper at ICLR 2025
on the weekend vs. workday.) In Figure 4 captured. In Figure(4p, we can see how if the SHAP value also makes this trend identifiable, but does not completely separate this bivariate interaction from other feature effects in the dataset. Overall, Figure 4 supports the hypothesis that training interpretable models is a better path than explaining blackbox models, especially when the same accuracies can be achieved.
how this feature interaction is adequately
Figure 4: The Spectrum of Interpretability to Uninterpretability. We display the key {hour, work- Jay} interaction for the interpretable GAM, explainable SHAP, and uninterpretable blackbox.
6.3 REDUNDANCY IN TREE COVER
In the treecover dataset, find strong evidence of redundant interaction effect. This dataset consists of predicting one of the seven types of tree species which are covering a given plot of land based eleven input features describing the area. Simple investigation important features for determining the species of tree are the altitude of the land and soil category of the land, Accordingly, we provide their partial dependence plots in Figure and 5p.
determine the most
Species tlaettie
Thes Spedesv Seil Sype
The Specien v. Altouda and soll
Sprucett
Ledpepo Rin
Posterses the
Douplos Krummiiite
200
3ia
Figure 5: ID and 2D Dependence of Tree Species on Altitude and Soil
However, a cursory look the ID dependence such as these plots or SHAP ignores the fact that both the elevation and soil type are additionally correlated with one another. Indeed, some montane type soils can only be found in lower altitudes and, equally, alpine-type soils can only be found at higher altitudes. Looking at the 2D heatmap in Figure 5b, we can see that soil and altitude are correlated with one another and somewhat redundantly predict the joint trend in the species of tree.
Training an MIP on this dataset is able to achieve validation accuracy of 80.4% whereas GAM-1 can only achieve 72.4% accuracy. Alternatively, a low-dimensional GAM is instead able to achieve 82.2% accuracy. This once again demonstrates that although the ID SHAP is unable to accurately represent this tabular dataset, simple low-dimensional GAM is able to as well.
Beyond the datasets we study here, there many existing works showing that both (a) feature interactions are necessary for real-world datasets, and (b) relatively low-dimensional GAM models often achieve SOTA performance on tabular datasets (Chang al. 2022 Enouen & Liu 2022). Accordingly, after the clear demonstration of both categories (synergy and redundancy) of feature interactions in practice, we move to exploring the same phenomenon in higher-dimensional data.
7 HIGHER DIMENSIONAL EXPERIMENTS
We additionally use our methods to explore a bird classification task on natural images. We run experiments on the CUB dataset using a resnet CNN architecture, evaluating not only at the original, fine-grained 200 classes corresponding to each bird's 's species, but also with 37 coarse-grained classes corresponding to each bird's family. For our GAM models we adapt the resnet architecture to only include the influence from 1 x 1, 2, and GAM-9 model. We fine-tune all models with procedures similar to Covert et al. (203)) see appendix for details.
3 set of patches, leading to GAM-1, GAM-4,

---

Published as a conference paper at ICLR 2025
Indigo Runting
Laculi Bunting
Northern Cardinal
Summer Tanagor
California Gull
Western pall
Prothonotary
Blue-winged Warhler
Warkler
CNN
GAM
1x1
GAM 2x2
GAM
3x3
Figure 6: Display of SHAP explanations for multiple images inside of the CUB birds dataset. Ex- planations are provided for the CNN model as well as multiple GAM models. We can see that beyond the gap in accuracy between the CNN and GAM models, there are also discrepancies in the reasoning processes as explained by SHAP. This can be taken as strong evidence of oversmoothing of the interaction effects which are used by the original CNN model.
Fine-tuning resnet-50 model the dataset, 65.0% and a bbarse-grained accuracy of 81.8%. In contrast, the GAM-1 model is only able to achieve 33.2% fine and 53.7% coarse accuracies. This be taken strong evidence that SHAP is oversimplifying the behavior of the CNN on this dataset. In Figure 6 we can see that this also manifests as sizable visual differences in the explanations between the two models. In some sense, at least 20 percentage points of accuracy are being completely thrown away when using the simplifi- cation of SHAP. So although a useful first approximation of the understanding, SHAP once again not sufficient for giving complete understanding of the behavior of feature interactions.
able to achieve fine-grained accuracy of
Extension to the GAM-2x2 and GAM-3x3 models is able give improvements 45.8% and 46.8% fine as well as 66.3% and 66.8% coarse accuracies. Even with some feature interactions, convolu- tional GAM model is unable achieve the same accuracies as a resnet. This points to the fact that either long-range or higher-order interactions are necessary to completely match the performance of the resnet. is also possible that an alternate GAM architecture would be able to further improve upon these accuracy results, helping close the gap between CNN and GAM. Overall, take this as significant evidence towards the oversimplification of SHAP explanations on nigh-dimensional data, domains where the need for explainability also remains the highest.
CONCLUSION
We find that the study of SHAP and GAM from a joint functional perspective allows for a plethora of insights in both domains which were previously possible. We establish the theoretical corre- spondence between the two across all possible correlated input features and discuss the implications terms functional representation power. practical ML datasets where input correlations are abundant, we provide a simple but theoretically grounded method of detecting whether SHAP is providing adeqeuate explanations by means of training a GAM model. We extend on the GAM literature by means of rigorously studying the effect of training on correlated input distribution, as well as introducing a novel masking technique which allows for the recovery purified GAM models. In multiple real-world datasets, we find that the existence of feature interactions as syner- gies and as redundancies is ubiquitous in practical settings. We finally discuss the implications of this fact in the context of interpreting SHAP in high- dith-nniennioall data like natural images. Although SHAP is very useful approximation of the first-order effects, a more careful treatment of feature interactions will be required for complete understanding of blackbox models.
10

---

Published as a conference paper at ICLR 2025
BROADER IMPACT
This work focuses on enhancing the interpretability of deep learning models. Although, broadly, the work of interpretability can help inform all related stakeholders to the reasonings behind decisions made by Al systems to the benefit of everyone involved, ultimately, all interpretations and decisions are made by humans and can hence be used for unfavorable outcomes both intentionally and unin- tentionally. Moreover, interpretability is only one piece of the larger puzzle which is transparency and trustworthiness Al systems.
10 ACKNOWLEDGEMENTS
This work was supported part by the Department of Defense under Cooperative Agreement Num- ber W911NF-24-2-013 The views and conclusions contained in this document are those of the authors and should be interpreted as representing the official policies, either expressed im- plied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

---

Published as a conference paper at ICLR 2025
REFERENCES
Robert J. Auman and Lloyd S. Shapley. Values of Non-Atomic Games. Princeton University Press, 1974.
David Balduzzi, Marcus Frean, Lennox Leary,
Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In Doina Precup and Yee Whye Tch (eds.), Proceedings of the 34th International Conference Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 342-350 PMLR, 06-11 Aug 2017. URL httthhtts://// httpss////roeeeenn...........e.. ba Ibalcuzziln html
John F. III Banzhaf. Weighted Voting Doesn't ' Work: A Mathematical Analysis, volume 19, pp. 317-344. Rutgers School of Law- Law-Newark, 1965.
Blair Bilodeau, Natasha Jaques, Pang Wei Koh, and Been Kim. Impossibility theorems for feature attribution. Proceedings of the National Academy of Sciences, 121(2). January 2022. ISSN
1091-6490. doi: 0tt0733/pna.223440000 URL httptp::/wxx....oo/r/.///// wwoi..org/000 10733//00........... 2304406120
Darya Biparva and Donatello Materassi. Incorporating information into shapley values: Reweight- ing via a maximum entropy approach. In Forty-first International Conference on Machine Learn- ing. 2024. URL http:httss//pperreeete
net. forum?? dHDwniHlwels
Gagan Biradar, Yacine Izza, Elita Lobo, Vignesh Viswanathan, and Yair Zick. Axiomatic aggrega- tions of abductive explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10): 1096-11104, Mar. 2024.
Sebastian Bordt and Ulrike von Luxburg. From shapley values to generalized additive models and back. In Francisco Ruiz, Jennifer Dy. and Jal-Willem van de Meent (eds.), Proceed- ings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 709-745. PMLR, 25-27 Apr 2023. URL pta.eeeddinn.mmrppppsss/////e/ wtw....reesvv/0//eerreee
html
Chun-Hac Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference Learning Representations, 2019.
Chun-Hace Chang, Rich Caruana, and Anna Goldenberg, NODE-GAM: Neural generalized additive model for interpretable deep learning. In International Conference on Learning Representations, 2022. htt.:tttpp::/petttt
ew, net. forum? (tw.:u//APAAAAAAAACCCCCCCCC.
A. Charnes, B. Golany, M. Keane, and J. Rousseau. Extremal principle solutions of games in char- acteristic function form: Core, chebychev and shapley value generalizations. In Jati K. Sengupta and Gopal K. Kadekodi (eds.), Econometrics of Planning and Efficiency, pp. 123-133, Dordrecht, 1988. Springer Netherlands.
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22(209):1-90, 2021. URL http: // r. httpr//aaprr////2222222
h1m.m.m.........mmmmmmm.mmmmmmm. ht.l
Ian Connick Covert, Chanwoo Kim, and Su-In Lee. Learning to estimate shapley values with vision transformers. In The Eleventh International Conference on Learning Representations, 2023. URL
net forummidd JJ.K
Deegan and W. Packel. new index of power for simplen simplen-person person games. Int. J. Game Theory, 7(2):113-123, June 1978. ISSN 0020-7276. doi: 10. 10.1007/BF01153239 URL httpps:///d/. arg/10.1007/B33 001100000/00770000
Abhimanyu Dubey, Filip Radenovic, and Dhruv Mahajan. Scalable interpretability via polynomials. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https::////// /ttp://iepeereiieeeeeee net forum?
Td=TwuCol I
12

---

Published as a conference paper at ICLR 2025
James Enouen and Yan Liu. Sparse interaction additive networks via feature interaction detection and sparse selection. In Advances in Neural Information Processing Systems, 2022.
James Enouen, Hootan Nakhost, Sayna Ebrahimi, Sercan Arik, Yan Liu, and Tomas Pfister. TextGenSHAP: Scalable post-hoc explanations in text generation with long documents. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computa- tional Linguistics ACL 2024, pp. 13984-14011, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguisties. URL https://acl / ogy. .rrg/ 2024. Fin....inninggaaccm 832
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. In International Conference on Learning Representations, 2021. URL https: /ttp///ppeeee....///e/eeee wttpeeewwwwweeeertteeeeeeeeeee wet.t//orrunnn--oPPPrrrrr
Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Eva Hammer. SHAP-IQ: Unified approximation of any-order shapley interactions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL hert/ooum? latp:mmmiddIIEMMMMAIM EMLNF 4GK4
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of Game Theory, 1999. doi: 10.10077:000882055025. URL http::://B...cgg.000000000000110
Joseph Hart and Pierre A. Gremaud. An approximation theoretic perspective of the sobol' indices with dependent variables. International Journal for Uncertainty Quantification, 8(6):483-493, 2018. ISSN 2152-5080.
Trevor Hastie and Robert Tibshirani. Generalized additive models, 1990.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10. 10.1109/CVPR.2016.999
Giles Hooker. Discovering additive structure in black box functions. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 575-580. ACM, 2004.
Giles Hooker. Generalized functional anova diagnostics for high-dimensional functions of depen- dent variables. Journal of Computational and Graphical Statistics, 16(3):709-732, 2007.
Xuanxiang Huang and Joao Marques-Silva. The inadequacy of shapley values for explainability, 2023. URL/
Joseph D. Janizek, Pascal Sturmfels, and Su-In Lec. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22(104):1-55 2021.
URL htt.:neppe//iiiooooop/ppeeeee wwwpor/pppppp/ppaopootpt.eeeeeee
1223.html
Dominik Janzing, Lenon Minorics, and Patrick Blocbaum. Feature relevance quantification in plainable ai: A causal problem. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2907-2916. PMLR, 26-28 Aug 2020.
Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. FastSHAP: Real-time shapley value estimation. In International Conference on Learning Representations, 2022.
2tt22ggg__VVV3333_322 VTV53T
Benjamin Lengerich, Sarah Tan, Chun-Hao Chang, Giles Hooker, and Rich Caruana. Purifying interaction effects with the functional anova: An efficient algorithm for recovering identifiable additive models. In Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference
Artificial Intelligence and Statistics, volume 108 of Proceedings
of Machine Learning Research, pp. 2402-2412, PMLR, 26-28 Aug 2020. URL https: proceedi ngs momr press/ 08/ 0 eeweeenngegeaaaa. html
13

---

Published as a conference paper at ICLR 2025
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 150-158. ACM, 2012.
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 623-631, ACM, 2013.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, Н. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 4765-4774. Curran Associates, Inc., 2017. URL http:: https//s.s.cccccccccccececccpc 7062-a-unifled-eeee-- wttp://......pp.ooooooooeeaaaaaa
Edoardo Mosca, Ferenc Szigeti, Stella Tragianni, Daniel Gallagher, and Georg Groh. SHAP- based explanation methods: A review for NLP interpretability. In Proceedings of the 29th In- ternational Conference on Computational Linguistics, pp. 4593-4603, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics. URL https:
wwwpaathhooo....rg/2//2 htg.://.....g222...oom.111111... coling-1.406
Melvyn B. Nathanson, Alternate minimization and doubly stochastic matrices, 2019. URL https:
//aaxiv.org/aba.t 882. 930
R. O'Donnell. Analysis of Boolean Functions. Cambridge University Press, 2014. ISBN
9781107038325. URL http:::/:
google.
Herschel Rabitz. Global sensitivity analysis for systems with independent and/or correlated in- puts. Procedia Social and Behavioral Sciences, 2(6)77587-5589, 2010. ISSN 1877-0428, doi: https://doi.org/100 http://doooooo/111001116..bppppp 000p000000100.aaap000000000..0.. occ.annce/rrrrlle/////////7711//
URL htsp::/pp// www.sssienccee www.sscceneedreee..ooo com/ Sixth International Conference on Sen-
sitivity Analysis of Model Output.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144. ACM, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206 2256, 2019. doi: 10.
0003/442226-019-004583
L. S. Shapley. A Value for n- r-Person Games, volume 2, pp. 307-318. Princeton University Press, Princeton, 1953, ISBN 400881977 doi: doi: 11111111////1111111777771 URL https:
//doi. 10/0000.1 110.111/97781108111111 0171111100881-0-0- 018
Richard Sinkhorn. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly, 74(4):402-40) 405, 1967. ISSN 00029890, 19300972. URL
http:// /wwp:
LM Sobol. Global sensitivity indices for nonlinear mathematical models and their monte carlo esti- mates. Mathematics and Computers Simulation, 55(1):271-22) 2001. ISSN 0378-4754. doi: URL ht https: /www. scis sciencedirect. The Second IMACS Seminar on
https://doi.org wtt.:////00.11100000007777000010
Monte Carlo Methods.
Xingzhi Sun, Ziyu Wang, Rui Ding, Shi Han, and Dongmei Zhang. puregam: Learning an inher- ently pure additive model. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '22, pp- 1738. New York, NY, USA, 2022. Associa- tion for Computing Machinery. ISBN 9781450393850. doi: 10.1145/3334788.5333555 URL
11w. /1w..11////333311333331 3539255
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9278.
PMLR, 13-18 Jul 2020.
14

---

Published as a conference paper at ICLR 2025
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 9259-9266 PMLR. 13-18 Jul 2020. URL http::://pcoeeddssssm...... wwtps/ms...peeessseeeeeeeesseee. sundarara
jan20a jan20 html]
Che-Ping Tsai, Chih-Kuan Yeh, and Pradeep Ravikumar. Faith-shap: The faithful shapley in- teraction index. Journal of Machine Learning Research, 24(94):1-42, 2023. URL http:
//jmlr. 22wp://:oo//aaprs/////22-2-----2 222.://222022...00 html
Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. arXiv preprint 2.Ki:20006.10665 2020.
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The caltech-uesd birds-200-200 dataset. Technical Report CNS-TR-20 2011-001, California Institute of Technology, 2011.
Grace Wahba, Yuedong Wang. Chong Gu, Ronald Kleins, and Barbara Kle. Smoothing spline anova for exponential familics. In The Annals of Statistics, 1994. URL httph:://// /www
stable/2244 2242772222766
Jiachen T. Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In Francisco Ruiz. Jemnifer Dy, and Jan- Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 6388 6421. PMLR, 25-27 Apr 2023. URL https://
proceedings www. wttppppsssssssasngeeeee
html
Zebin Yang, Aijun Zhang, and Agus Sudjianto. GAMI-Net: An explainable neural network based on generalized additive models with structured interactions, 2020. URL https: htt////axvvv./// abs/2003. 07132
15

---

Published as a conference paper at ICLR 2025
A POST-HoC EXPLAINABILITY
A.1 FURTHER DISCUSSION OF EXPLANATION BASELINE METHODS
Conditional/ Marginal/ Baseline We first reiterate the three main removal baselines which see widespread usage across all domains of machine learning explainability. Those are the three methods introduced in the main text (baseline value, marginal value, and conditional value.) should be noted for instance that in the original SHAP paper (Lundberg & Leel (2017), Equations 9-12), each of the former two were considered as a simplification or approximation to the conditional value. The first assumption of feature independence implies the equivalence of the conditional value and the marginal value. The second assumption of model linearity implics the equivalence of the marginal value and the baseline value. Accordingly, it is perhaps better to think of these two alternatives as practical simplifications whereas the conditional value is the value of theoretical interest. Especially after the highlighting of the tthe-manifodd problem (Frye et al.. [202T), these two approaches have been under higher scrutiny in their application to typical ML pipelines where input data often have heavy correlations existing outside of the control of the ML practitioner.
Integrated Gradients Another common removal approach is Integrated Gradients which is equiv- alent to Aumann- Sumann-Shapley value (Auman & Shapley 1974] (Sundarraaaan et al. 2017 7 Sundararajan & Najmi 2020). In this version, line integral is taken from baseline point the target point 2, rather than the original baseline method which simply takes the difference between the two. Al- though it is a smoother approximation which has had empirical success, its interaction extensions (Janizek et al., 2021) cannot succeed on non-smooth functions like piecewise linear ReLU networks problem. Although incorporating a more general definition of line integrals could be of interest to solving the off-the-manifold and simulta- neously integrating into the discrete masking framework utilize, we envision this out of scope for our focus on the Shapley value.
and it is nonetheless susceptible to the
Stone-Hooker Decomposition Of potentially the greatest interest besides the conditional case which we directly study is the alternative functional ANOVA decomposition proposed by Hooker 2007) and further investigated in (Hart & Gremaud 2018 Lengerich al. 2020, (un et al. 2022).
F(n Flr.,...,
hs(#s) SC(d]
Σ
(21)
where the functions are required to obey a set of 'hierarchical orthogonality conditions'
Mp.0 (gT hs)=0 =0
Vgr, V
(22)
which is equivalent the 'integral conditions'
Mp,(s-n o (hs) =
V.
(23)
Despite its relatively pleasant properties compared to the original Sobol-Hoeffding decomposition, nearly two decades after its introduction it has received relatively little attention when compared with the Sobol-Hoeffdiee that in the case of independent variables, the same solution is recovered.) Amongst its limitations, beyond lack of intuitive meaning behind its prescribed functions, the most severe is seemingly its computational intractability. It is rare to see a calculation of the full decomposition beyond a small number of dimensions or for distributions which are not piecewise constant. Unlike the conditional projection which can be more efficiently approximated from the bottom-up. the Hooker decompo- sition seems to endure the full exponential complexity of constructing a functional decomposition from the top-down (starting with the most complex fyja)))
alternative defined by conditional projections. (It should be briefly noted
Practical approaches to providing solution to the full Hooker problem imitate Sinkhorn approxima- tions via iterative refinement across the different variable axes (Lengerich et al. 2020). Nonetheless, the Sinkhorn algorithm has itself escaped general closed form solution for decades (Sinkhorm, 1967 Nathanson, 2019). and practical application of the original Hooker decomposition has re- mained extremely limited.
16

---

Published as a conference paper at ICLR 2025
In the space of Generalized Additive Models, however, recently some progress has been made. By leveraging the GAM's ability to reduce the exponential complexity of the true function to a lower- dimensional representation, works like Lengerich et al. (2020) and Sun et al. (2022) have found success in purifying the terms of an additive models. However, this success is still limited to two- three- dimensional GAM models, being limited by: discrete variables assumptions requiring his- logramming or kerneling of continuous variables: difficult transformations which do not easily scale to higher orders; and/or the previously discussed Sinkhorn-like approximations without clear guar- antees. We later revisit these considerations more thoroughly in Appendix Elatee the introduction of our novel results for additive models, suggesting how our variational perspective potentially allows to unlock the same advantages for Hooker-type purified models.
Further Alternatives We highly recommend the work Covert et al. (2021) for very compre- hensive review of potential methods for removal; however, we quickly review some of the major flavors for the counterfactual 'removal' of a feature. Some of the important methods yet unmen- tioned include the utilization of surrogate models to explicitly or implicitly mask out the features. This can be done implicitly via the training of a masked surrogate predictor using the projection equations for square error or for KI. divergence as used within this work (Covert et al., [02T] Jethani et al.) 2022] Covert et al.) 2023). There have also been pursuits through a more explicit approach via using a separate generative model (VAE or GAN) as a proxy for removal (Chang et al.) 2019), additionally allowing for more domain-spectie
approaches like image blurring and infilling. Another important set of alternatives is via the language of causality as introduced via vaaaiiing et al. (020). Unfortunately, after the introduction of elegant causal notation, the authors immediately use a simplifying assumption to reduce to the marginal Shapley, which has the aforementioned prob- lems, only considering the engineer-level causality of "causing the model' change its predictions. This is significantly different from the scientist-level causality of 'causing the output' and has only begun to be thoroughly addressed in recent works like Biparva & Materassi (2024).
A.2 POST-HOC FEATURE ATTRIBUTION AND INTERACTION ATTRIBUTION
Notation Let d E N and e E N be the dimensions of the input and output spaces, C Rd and y' R.. Let F X' + y be a function representing a machine learning model which maps from inputs outputs. We will use [d] == [1, .I...., d} to represent the set of input features and S S [d] to represent a subset of the input features. We also write the set of all such subsets, the powerset, as P((d) = (0, 1)dd and use slight abuse of notations including (S+i)= := SU(i and (5-1) := Si)
We will write the function space as some H. = {FF: X y} and a masked function space as H' = (ff : X' x P([d)) + R). For a general feature attribution method, we write H. Hd. taking function F(x) as input and returning local explanation function { (x) for each feature i [d] on each local input J E X. Similarly, we define a blackhox feature attribution method Φ H' + Hdd instead taking a masked function f(r input and returning local
explanation function for each feature, [o, o f](x).
In addition to the notation introduced in the main body, we introduce some notation which are very useful in the domain of feature interactions. We first define the discrete derivative operator:
[6, o f[() = f(T + 1) - f(T - i).
(24)
and its higher-order counterpart:
S(T [ōs o S(T) = (-1)) S W).
(25)
WCS
We note that the decision to add and remove elements instead of only adding elements is not nec- essarily typical: however, we find beneficial not need to restrict the domain of the discrete derivative operator.
We may now define the Mobius transformation or purification transformation as the one which replaces each function with its purified version, μ H¹ + H'.
[n o f[(2, = f(z, S) = Σ (-1)181-1W1 W)
(26)
WCS
17

---

Published as a conference paper at ICLR 2025
We also that the purified functions can additionally be written in terms of the discrete deriva- tive operator.
[(, S) - (ōs A1x, 0)
(-11)81- [(x W) WCS
(27)
Both the discrete derivative operator and the Mobius purification transformation are important tools for being able to more easily study case feature interactions in blackbox explainers. In sections that follow we will define the major feature attribution and feature interaction attribution methods in terms of these operators.
The Shapley Value The most typical definition of the Shapley value is usually its closed form solution as the weighted average of ID derivatives,
SHAPP o A1(()) = Σ ((()))
(o,
Scld-- |S]
although its definition expectation over random permutations,
SHAP
f)
(2,
Sd
TES
has gained popularity in practice due to its susceptibility to Monte-Carlo sampling. We define Sa as the symmetric group or set of permutations on d elements, Sa := { [d] + [d] s.t. is bijective), and we define ST, as the set of predecessors under the ordering л, ST. == {j E [d] s.t. x xi) π (i)).
We also state the alternative formulation in terms of 'unanimity games' which the authors believe to yield more intuitive understanding of the Shapley value.
fs(es)
Sh fl((z)
Σ 82(0)
(28)
[S]
In words, the Shapley value divides the purified interaction (ss) (which is the value created by S and only S) amongst all its constituent features, E S, completely uniformly between them, [S]
We rewrite the four Shapley axioms in terms of the functional notation:
1. Dummy
6 o f1 5) = for all S,
then (o: f[(r) (for that local æ).
2. Symmetry
ा x())) o o f Φ: of
Vi [d] VT E Sa
3. Efficiency
Eielg
f(4 Se
4. Linearity
o (f + = 6o o f bag
It should be emphasized that dummy is truly local property whereas symmetry, efficiency, and linearity can all be realized as properties of the additive functions. Thus, from the functional per- spective, it is more appropriate to call this property 'local dummy' to emphasize its distinction from "global dummy' functions which would be the case when o; f O. We hope this would help eliminate the common confusion we discuss later in Appendix B Another point to briefly note is that linearity condition about the linearity of the operator rather than the linearity of the function.
Other Common Explainers One of the original blackbox explainers is the LIME value Ribeiro et al. 2016), which can also be written in this functional notation as:
[1]]
!IME F](x)
arg min DERd
E (S
|LIME (x, S) - Σ 1(
(29)
1=1
18

---

Published as a conference paper at ICLR 2025
where the distribution is taken LIME distribution pS) and the function fL (a. taken as the semi-local average value according to a data-dependent LIME kernel which is the exponential of some distance function. For further details see Ribeiro et al. (2016) Lundberg & Lee (017).
Another common set of explainers the extremely simple 'leave-one-in' and 'leave-one-out" val- ues, based on including a single feature or removing a single feature:
[ f](x) = (6, o fl 0) p f](x) [6, fl
(30)
(31)
These also have equivalent versions for measuring the interaction effect in the general 'inclu- sion value' or 'removal value':
inc f((x) = (6s [(, 0)
(32)
L0' f](x) (8s [x,
(33)
We note that it is also popular to refer to the difference, f( S--fS)=- 0), rather than the interaction
effect. as the inclusion value. Another important interaction explainer is the Archipelago value (Tsang et al. 2020) which is defined to be the average of these two dighh := 10 + d This
simple estimator is surprisingly robust at detecting feature interactions that di or Uee would each individually miss.
There are also few other theoreti approaches which have attracted attention recently such as the Banzhaf value (Banzhaf] 1965 Tsai et al. [2023] Wang & Jia 2023; Enouen et al.) 2024) and the Deegan-Packe index (Deegan & Packel 1978 Biradar et al. 2024), especially in their application to classification tasks instead of regression tasks.
A.3 SHAPLEY INTERACTION INDICES
Shapley Interaction Indices The first definition extending the Shapley value to try handling fea- ture interactions was already constructed in 1999, mainly by the removal of the efficiency axiom (Grabisch Roubens 1999). This allows for relatively simple extension using the permutation symmetry axiom define the interaction index be present rather than the one. From Table D below, it can be seen how this index divides higher-order interaction effects amongst their constituent lower-order subsets in the same way as the original Shapley value (1/0.
random order value where both features must
Table 1: Shapley Interaction Indices for = 1, 2,3
Equation
= 1
==
B
10
1/2
1/4
1/5
1/8
1/7
1/8
1//
1/ID
1/c 1/4 20
1/2
1//
1/4
1/5
1/6
1/7
1/6
1//
1/2
1/3
1/4
1/5
1/6
1/7
1/8
Equation
t
t =
13
14
15
16
17
18
19
20
1/1
1/11
1/12
1/18
714
715
7/36
1/17
1/18
1/19
1/20
k
2
1/11
1/10
1/11
1/12
1/13
1/14
1/15
1/10
1/17
1/18
1/19
3
1/c
-
1/a
1/10
1/1x
/u2
1/1s
1/34
1/15
1/16
1/17
1/18
19

---

Published as a conference paper at ICLR 2025
Mhapley-Tayloo Interaction Indices The next major advancement Shapley interaction indices came with the introduction of the Shapley-Taylos indices 2019 (Sundararajan et al. 2020). These interaction indices reintroduce the efficiency condition in way which now know reflects the additive model structure summing to the full prediction. However, they achieve this decompo- sition by treating the lower-order additive effects asymmetrically from the maximum rank effects. In particular, they zero out the influence of everything except the purified effect and distribute the higher-order effects amongst the rank k subsets. This can be seen more clearly in Tables 2 and
Table Shapley- Taylor Coefficients for k
Equation
1/4
1/2
1/5
1/4
1/5
1/6
1/7
1/6
1,6
1/10
Equation
16
17
18
19
20
3/1
1/11
1/12
1/18
1/14
1/16
1/16
1/17
1/18
1/10
1/20
Table Shapley- Taalor Coefficients for k:
Equation
3
7
2
0
0
0
0
0
1
1/3
1/6
1/10
1/15
1/21
1/28
1/36
1/45
Equarion
=
11
t
12
8
13
14
15
16
17
18
19
20
k= 2
=
0
0
0
0
0
0
1/68
1/60
1/78
1/01
1/205
1/120
1/130
1/163
1/1m1
Table 4: Chapley-Taylos Coefficients for l: =
3
Equation
== 1
t=
6
7
8
10
1=
8
1
0
0
0
0
0
0
D
0
3
0
0
1/4
1/10
1/20
1/35
1/56
1/44
1/120
Equation
11
12
13
14
15
16
17
18
20
3
0
0
0
0
0
0
0
0
1/105
1/220
1/280
1/804
1/465
1/500
1/680
1/610
1/900
4/1140
20

---

Published as a conference paper at ICLR 2025
n-Shapley Values The D-Shapley values more recent attempt revitalize the original Shap- ley interaction indices to obey the efficiency axiom an alternate way (Bordt & von Luxburg, 2023). They
recursive form that the maximum rank terms (s k) are the same as the original interaction index (Grabisch & Roubens 1999); however, the lower order terms (s < k) are chosen exactly ohey the efficiency terms. This requires the of the Beroulli numbers balance these terms in a recursive expansion. The first few orders can be seen in Tables 5 and 7 Note the similarities and differences with Table
Table n-Shapley Coefficients for n =
Equation
1
1/4
1/3
1/8.
1/7
1/9
1/10
Equation
==
13
17
19
1/1
3/12
1/18
1/14
1/18
1/10
1/20
Table 6: n-Shapley Coefficients for
k 2
Equation
t
1
x
10
8: ===2
=:1
(t-21
21
1
0
1/u
1/4
3/10
1/3
9/24
3/8
7/1s
2/3
1/2
1/6
1/s
Equation ((--))
9/22
9/12
11/20
5/y
7/56
18/84
1/1e
1/11
1/12
1/18
1/54
1/28
1/10
1/57
1/18
1/59
Table n-Shapley Coefficients for n
Equation 1-31--1
t=1
== 2
10
1/30
1/12
1/7
9/24
8/1a
7/2n
(2
-
1/4
3/10
-
5/44
4/4
7/Lu
(0-2)))
0
1/2
1/s
1/4
1/6
1/6
1/7
1/8
Equation
<= 11
= 12
( 13
14
15
16
17
18
19
20
3
14/33
1/3
15/30
88/84
11/15
18/16
91/102
59/30
20/ta
17/15
-//s
9/22
8/1a
11/b
9/8
19/50
7/16
15/84
4/a
-17/38
1/20
1/13
1/14
1/15
1/50
1/57
8/48
21

---

Published as a conference paper at ICLR 2025
Faith SHAP, the Faithful Shapley Index The other most recent attempt at Shapley interaction index is the faithful Shapley interaction index. Instead of leveraging the permutation sampling symmetry of the original Shapley value, this work instead extends the Shapley value by means of its least-squares characterivation. As we discuss extensively in this work, this can be seen as further utilizing the characteriztaion of Shapley values as additive model approximation.
Tsai et al. (002) Equation (16) solves for the faithful Shapey interaction indices from the perspec- tive of the Mobius/purified functions:
S k S
(24-2)
Fiih SHAP k of
f- is (-1)
11
Σ
k fr
(34)
(S)
(1I
T2S/TT>&
K+IS!
In Tables 8 and 10 below, we display what these coefficients are for the Mobius purified inter- action effects. We additionally calculate slightly simpler form of these coefficients in the claim below.
Table 8: FaithSHAP Coefficients for
Equarion
1/4
1/2
1/3
1/9
1/10
Equation 1/4
19
1/11
1/12
1/13
1/14
1/18
1/16
1/17
1/18
1/19
1/20
Table 9: FaithSHAP Coefficients for k =
2
Equation
[=1
cm
6
7
8
10
-21-22
0
1/6
1/5
1/5
5/23
5/2s
1/6
7/46
2054:33
3/10
1/5
3/28
1/12
1/28
2/ea
Equation
17
8/22
- 8/50
13/61
9/98
13/120
- T/6s
9/63
18/571
17/100
5/30
1/22
1/20
4/61
1as
1/00
9/136
1/0
1/5
9/150
1,50
Table 10: FaithSHAP Coefficients for ls = 3
Equation
t=1
=
8
5/30
3/28
7/60
1/38
9/t4
4//33
-//6
-
-7/55
1/3
2/7
5/50
5/ea
1/12
2/53
1/203
11
15
18
20
=3
18/ias
40/004
13,00
33/sa0
19/5500
Pijero
8/1o
0s/usn
203/1100
18/44a
9/61
9/80
13/440
9/4s
18/504
32/00t
37//aa
8/0es
%/61
1/00
10/600
1/114
1/55a
Claim 1. The Faithful Shapley coefficients can be written with the alternative formula:
(t---1)
SHAP- s=
Σ
(-1)+-
(++--1) (-1))-
s-1
7t+
k-s
fr
(35)
T28,TT
k
Proof. We would like to show that:
(-1)a- k- k (
t-s-1
(-1)
k+s-1)
k-o
[+k-1
s-1
[t+-1|
k+t
k
Let us write
8
0-11 t -
-1
8-1 [k+sT_)
(k+
(t==
(t-
+ - -
t
k 1+3
s
b:
k: s
k -
k
22

---

Published as a conference paper at ICLR 2025
k!
(t - 1)!
++ k 1
(k + 1)1 k: (t===-1)
(s 1)1k!
(t (t-s-
(t+ k 1)!
sl( s). k!(t - k - 1)!
k +
k.(t 1)!
(1 1--1] 1)!
(1+ - 1
1++k-1
(8 - +
k-8 )
k 1)1 (1---1)
+ (s - 1)((k s)((t - 1)!
k+8
1+
(1+ -+ 1)! 11 (1-11-1-1)
1: + (k: s)((t
k - 1)
8+8
(k+ 1]!
h: 8-8
1 (t 8 1)((k + s)!
(++ 1)1 (K s)1(t k 1)!
++8 (k- k: 1}1
1)!
+ 1)!
(1 8 1)!
(k+
h++ 1)!
k k+s (t+ k - 1)!
ss+ 1)!
(k: + 1)!
(++ 1)!
(t+ k 1)!
(Ao + 1)!
[1]-[4] [1] =1
Due to the fact that s S l < t, we find this to be a slightly nicer formulation when written as function of
23

---

Published as a conference paper at ICLR 2025
B COMMON PITFALLS IN THE CURRENT LITERATURE
B.1 LOCALLY ZERO VS. GLOBALLY ZERO
Consider the simple function (21, cos (2mx1) + sin (2mzz). Assume the input features uniformly distributed across the space [-1, 1J2 Based on our correspondence, we can see see that the Shapley value functions decompose the same additive model with (æ1, = cos (2 and Φ2 1 T2) = sin (2
When set 31 J. have that Φ1 (2) cos 3 0. Does this suddenly that function Φ1 (21, v2) is not a function of x1 because it is zero at one value? No, it does not.
1)
10.25.1)
(0.25,
(0.00
(0.5.0)
0.5.
Figure Cosine and sine functions.
hoped that after the clarification of the functional perspective on the Shapley value, it can be made clear that the exact same question is being asked when the Shapley value is equal to zero for a single input point. If one is interested in the global importance of a feature, then one should check for being zero nonzero entire function. This can for instance be checked with the variance of the Shapley function:
V/SHAP
Varx SHAP (X)
(36)
B.2 BEESWARM PLOTS INSTEAD OF SHAPE FUNCTIONS
It is common to see beeswarm plots of the SHAP values used as an aggregate summary over an entire dataset. Although an information dense representation of the SHAP values across the entire dataset, we feel the additive model perspective brings some insights into their limitations and what can be improved about them.
Shape Function of SHAP for Workday
43
Beeswarm of SHAP for Hour and Workday
-24
worksaa
weekond
-1.9
www.ee
worksday
Shape Function of SHAP for Hour
Hour of the Tay (b) Shape Function Representation of SHAP
(a) Beeswarm Representation of SHAP
Figure Alternate representations of the SHAP values aggregated over an entire dataset.
First of all, the major limitation when compared to the shape function or additive model represen- tation is simply the compression of information. Each of the shape functions in Figure 8 is com-
24

---

Published as a conference paper at ICLR 2025
pressed horizontally and then its flattened version is rotated and stacked amongst the SHAP values for all other features. Although this information about the feature value is theoretically preserved via the colormap, it is well known from data visualization that this is insufficient to adequately preserve the information. This compression effect is especially pronounced in cyclic or heavily oscillating shape functions.
Surprisingly, these plots are also commonly used with the same red-blue diverging colormap which used in the output space for SHAP. This colormap makes sense for the SHAP values since they are referring to the positive or negative influence on the output prediction; however, when also applied the feature value, we are conflating the y-axis and the x-axis in the shape functions of Figure Bp. Additionally, this is using diverging colormap to represent what are usually sequential features (meaning a sequential colormap should be used instead). Although we again recommend to look at the shape functions to get a clearer overall picture, if one is not willing to inspect all of the shape functions, it is perhaps more effective to remove the color from the beeswarm plot entirely and/ or to just use aggregate statistic such the variance in Equation 36 Shapley values depicted by the beeswarm plot.
measure the spread of the
B.3 LOCALLY LINEAR OR LOCALLY ADDITIVE
The interpretation of a locally lincar model has two key interpretations which are unfortunately conflated in much of the work on explainability. The first is the semi-local interpretation of the coefficients gradient-like coefficients telling the direction of greatest influence. Usually this is done similar to LIME (Ribeiro et al.) 2016) where linear function fit to a weighted neighborhood of points. This is in contrast with the gradient which fits the same linear function to an arbitrarily small neighborhood of points. These are both different from the second interpretation of the linear model structural assumption onto the functional space. In particular, the linearity and efficiency axioms of the Shapley value are made on the Shapley operator itself. The local linearity assumptions do not translate to a global linearity assumption, but rather to a global additive assumption where the height function is dictated to respect the additive structure of the original function.
Hopefully after the clearer connections we make with additive models, it is clear why the height interpretation is the correct one for SHAP and that the gradient-like tests for SHAP are ill-posed after a contextual understanding of the goal of SHAP as the height function rather than a measurement of the local sensitivity similar to the gradient.
Let us also briefly recall that in the case of a GAM, there is still an interesting correspondence between the gradient of SHAP and the gradient of the GAM.
Recall the GAM-1 equation.
F (ai.... va) = f1 +...
.... fa(xa)
follows that the gradient obeys the similar
V F E1 Iz) = (0) (x), (a))(x))
If one is interested in a local sensitivity test, then something like the gradient of the GAM or gradient of the SHAP should instead be used, but is gently reminded that the vanilla gradient will ignore the statistical structure of the manifold (Frye et al. 2021) and could face alternate issues like the shattered gradients problem (Balduzzi al. 2017).
B.4 BASELINE METHOD ON DISCRETE INPUTS
If you have a fimite set of discrete, categorical inputs, then the (mis)usage of the baseline method becomes of great importance for reasons beyond the fif-the-manifole problem. In particular, is common to replace the current input variable by the baseline input variable only to realize they had the same value (e.g. zero). It follows that the counterfactual removal will have no effect, Although extremely rare in the continuous case, when using categorical input variables it is easy to mask out bivariate and even higher-order effects. is especially easy to make this mistake on boolean input variables. Accordingly, it is perhaps recommended to those with lesser familiarity with boolean functions or logical functions to use the {+1} or one-hot encoding instead of (0. 1) to avoid making such mistakes (0' Donnnell 2014).
25

---

Published as a conference paper at ICLR 2025
C IMPOSSIBILITY THEOREMS FOR FEATURE INTERACTIONS
This section will broadly prove the representational power of SHAP and related black-box explainers in the form of 'impossibilityy theorems' or 'possibility theorems'. The main method of proof tech- nique is simply via the proof of functional correspondence between the two spaces, the functional space of interest be taken for hypothesis tests, H. and the functional space of additive models for some sufficiently large order & E N, H.SK.
Let us first proceed by defining the trace of the SHAP function as the object of interest so that we may thus allow direct functional comparison between the two of GAM and SHAP. For simplicity, we will assume a rectangular feature space X' X1
x X2 for some subspaces X, for each i E [d]. It is simple to extend some subset of IR to a rectangular subset in this way by taking the product of the marginal spaces. It is also necessary extend the distribution p(z) which has been assumed on the original space X' to our rectangular space. However, the obvious extension of zero probability on any additional coordinate will be sufficient for our purposes, since will already be required to make all of our statements modulo differences on null set (set of measure zero). Accordingly, we will simply restrict our focus to a rectangular feature space and ignore any concerns regarding differences on null set. Any hypothesis test which is testing zero, although completely ill-posed, will not be answerable under this statistical framework.
set of measure
For given test point x² E X, we define the SHAP trace x= be the object:
TTHAP((" (x*) = (1r [BSHAP F(())) x (zi) x x 1) x X, x (xi+1) x .... x {at)
VE(d)
In words, the trace of the SHAP value is the image under the Shapley function of the set which includes all possible 1D perturbations of a single feature value.
To enable direct comparison with the original functional space H. we will define the completion of the trace as the function which is extended to the entire rectangular fcature space X in the obvious way by assuming that the SHAP trace is the true additive model representing the underlying function F.
CTSHAP (x") (0')= (1. >
ch
SHAP o ((ci,,..,,,$__. F1, .........))).. x E {
Theorem 5. The SHAP trace of a function F will satisfy any hypothesis test Ho v. Hi inside of the functional space H if and only if the functional space H is equivalent to shift which is not superset of HSt ANOVA
Proof. Since we are making a claim across all possible splittings of the functional space H into two possibilites of the null hypothesis Ho and the alternative hypothesis H₁, it is required that we actually show the exact identifiability of the individual function from the SHAP trace alone.
Suppose first the H. truly is a subset of the ANOVA-1 space (or shift thereof). We simply take the difference (Fi - F2) for some arbtirary F1, F2 E H. or otherwise assume we know the functional shift required to center our functional space to be a subset of the ANOVA-1 space. It follows from the exact formula of the SHAP function that for any F E H. we will have that (1(x) = (x,
(z.), which implies that the completion of this trace will immediately recover the original GAM-1 function.
In the other direction, to prove the contrapositive, assume instead that H truly has some feature interaction, which can be represented by (F, F2) and (Fi - F3)) being some different shifts for some FI, F2, F3 E H. For simplicity, let us shift by (Fi - F3) so that one difference is zero and one difference is nonzero. It follows that for the nonzero interaction effect, there is some S C [d] with |S| > 1 such that fs # 0. Since this is true in the statistical sense, there is region of sufficient difference. Via the assumption that our space H is at least as representative as H ANOVA this means we can find two distinct functions which map to the same SHAP trace in the local region of this nonzero measure region. Accordingly, if we take hypothesis test which identifies these two distinct functions, their SHAP traces will still look indistinguishable and there will be no succesful hypothesis test based on the SHAP trace.
26

---

Published as a conference paper at ICLR 2025
Of course, practically speaking. we likely do not have access to priori knowledge about the global feature interactions of some hypothesis space H which would allow for the construction of the isomporhism between our given H. and some which is actually a subset of the ANOVA-1 space ANNVA
Proceeding by defining the trace of Paith-SHAP-4 and GAM-k in the obvious way, we may find similar theorem for the higher-order interactions of & E N. Once again, is practically more useful to say we cannot directly assume the existence of feature interactions across the entire hypothesis space and hence again asssume H S H ANOVA without the caveat of allowing a shift by some oracle assumption. It is also perhaps more interesting that we can make the same statement for any arbitrary frontier I C P((d) as we will introduce the general study of additive models in Appendix E
C.I SPECIFIC EXAMPLES
Although we have shown exact functional equivalence from which the ability to do hypothesis tests follows, we restate some of the tests from Bilodeau al. (2022) to make clearer comparison.
The first hypothesis test is coming from their Proposition 3.5, and although they do not name this hypothesis test, we call it the "almost 8-local Lipschitz" because of its resemblance to the slightly more typical "8-local Lipschitz." test.
[F(x') F(x)
& (F) ==
sup x/E(x: (2(xx-6,,00 6.
(37)
8
[F(x') F(x)]
Ch (F)
sup
(38)
-fE(x;- +,8
|x - x1
They then create the hypothesis tests:
Ho
F E H: Calmoa (F) S
(39)
H1
F E H. almost (F)
(40)
And then identify that the gradient can successfully distinguish these two hypotheses. This is de-
sired since is well known that on compact interval, continuously differentiable functions are automatically Lipschitz functions.
The next two major hypothesis tests they build are for 'local recourse' in Definition 3.7 and 'locally spurious' in Definition 3.8. For recourse, we first assume some counterfactual distribution v(x) which they implicitly assume to have nonzero measure across the left-local and right-local regions (xq - 8, xrj and [ xy + 6). We define the 'value of moving to the left' and the 'value of moving to the right' under the counterfactual distribution, v(z), as the 8-local left and right recourse values:
V (F) Exx.
[EI X , v.) X₂ E [aq
8, sr,]
(41)
V6. (F) (= = Exar Beel....,,,,...... X; E [, + 6]
(42)
They then create the hypothesis tests:
Ho-
F FEH: (hodt((F) (
(43)
H1- F FEH: H: yore (F) S Vó (F) (F)
(44)
The infinity norm over a local interval (from the left and from the right) is defined as:
| 1
sup EE -6,2]]
(45)
{ |mcoss}}
[
sup [[([
(46)
27

---

Published as a conference paper at ICLR 2025
The final major hypothesis tests introduced in that work are the tests for if feature is local spurious:
I =0 =0}
Ho F E H
(47)
H1
r E H: ||
(48)
To reiterate, all of these tests are easily answered via the SHAP value using the additive model trace. All of these hypothesis tests focus on the behavior inside of a local neighborhood for some arbitrary 8, whereas the SHAP value evaluated at single point only describes the behavior at single point. Hopefully, from the functional perspective on Shapley, of these questions can be answered by using the additive model equivalent of the Shapley value. This does not conflict with previous works showing negative results on these same hypothesis test since their focus was on the ability of pointwise indicator's ability to perform these hyptohesis tests, and related to our discussion in Section
now clear how easily all
E and Section B3 3. Moreover, it is hoped that the true limitation of SHAP, its inability to adequately handle feature interactions, is emphasized as something that cannot be shown by any of these existing impossibilityy tests due to their focus on single perturbed feature at time.
28

---

Published as a conference paper at ICLR 2025
D EXPERIMENT DETAILS
D.1 2D SYNTHETIC
First, we look particularly simple example of synthetic data to highlight the two important pects which the Shapley value alone is unable to capture: feature interaction and feature correlation. Hopefully this example will help develop intuition for the Shapley value and highlight its unique challenges in the setting where input variables are correlated.
For some p E [-1, 1], we consider the data generated by
X,Y N KNY~NOO (o.
[(r,, y) = I + ry
It is relatively straightforward to calculate that:
fo
fo = p
fø = x + px
fo + px p
fy = py + Pt
f, PJ) + pi
fay x xy
fzy + ay pr pyy +
Figure 9: Simple Synthetic Dataset using various p E {0.0, 0.1, 0.2, ... 1.0). Each of the five rows corresponds to the learned fa. fy- fzy. 6g. and 6, The third is hence 'top-down' from the z- axis whereas all others have the output as the vertical axis. Visually constrained to 2, y E [-2, 2)2 with colors/ outputs in |-3.3]. A single point is highlighted to emphasize how the Shapley value in the bottom two rows is constructed from the top three rows.
The Sobol covariances are hence:
Elf fal= E[ + px + x²ÿ + p Elf fol = 12
+0 P²
Elʃ }n] = Elpry + pry² + pryz + payy² - pa -0000++++++++++
Elf fayl = E (p²) - 2p²) -3)=-
Hence Cp = p², C. 1 C. 3p² Cay = 4p².
The Shapley functions are also &2 (x, y) = fz + fay and Φy (x, y) = F Pay.
(2(x, v) (x 2a=2 p) { my xy p=² pn² p) -
2 (x² y2 1)
6y(x, y) (py + 2nj - p) + 1 (-py py + xy p.. - pnz + p) [ + प + 2 1)
29

---

Published as a conference paper at ICLR 2025
the learned set of functions across various {0.0, 0.1..........................0 1.0) when learning a GAM with the purified loss function. The Shapley functions can also be calculated to be Oz = fz + Jay and Φy = fy + Say which can be seen in the fourth and fifth rows from the Figure. We furthermore that the additive models using the purified loss align with the true purified ANOVA decomposition and that hence the Shapley value functions can be computed in constant time given the purified GAM model. Further, going left to right, we how the strength of the ID effects (first and second rows) increase, whereas the strength of the 2D effects (third row) decreases as the amount of correlation increases. This corresponds the deterioration of the feature interaction due to the increase in feature correlation. At the halfway mark, the 2D function is no longer positively correlated with the true outcome. This most obvious in the far right (p 1.0) plots where f= f= = --y- meaning du f and Φg = 2.v
In Figure
Interestingly, we can see that for |p > 1. we actually have that Czy < That to say, the interaction term alone is no longer positively correlated with the function are trying to learn. This further implies that the purified interaction is actually negatively correlated with our target, and adding it to the model somehow reduces the performance measured with MSE. This must be juxtaposed with the fact that our function f(r y) = x + clearly demonstrates a feature interaction in the term ay.
This can however be resolved by not thinking of the purified interaction alone but in conjunction with other features when it is added the model. For example, if we started with y and added x, then we could consider Cz + Cay = 2 2p² 2 as the improvement to the model. Conversely, if we started with x and added y. we could consider Cy + Cry the model. Intuitively, it is not the fact that the interaction is detrimental the model performance, as clearly it is necessary for [o] < 1, but rather that is overshadowed by the information which is gained from either or y alone. "The redundant information from knowing x or y outweighs the synergistic information from knowing æ and y."
p² 2 0 as the improvement to
D.2 10D SYNTHETIC
For our major synthetic experiments where we benchmark the ability of FastSHAP and InstaSHAP to recover the Shapley value effects, we create dataset similar to the simple ones in Figures FiguresZJand9
We generate ten features from a correlated pairs structure on the covariance matrix. In practice, this will mean our low-dimensional synthetic target variable will remain having a low-dimensional func- tional ANOVA decomposition because of this simplistic correlation structure. Moreover, this allows us relatively easily calculate the exact Shapley functions even in this 10-dimensional dataset.
P P
1
1
P
Σ
P
1
P
We then take the target variable to be
f(x) = Σ
Bs
II
(49)
SEISK-
IE
for some ke and for some Bs drawn from the normal distribution A/ 1) or the Laplace distribution Laplace(O, 1). We finally divide by a constant to normalize the output response such that the total variance of the output is equal to one.
D.3 REAL-WORLD TABULAR DATASETS
We follow the methods of SIAN (Enouen & Liu 2022 to train GAM models for tabular datasets. After the training of a surrogate model, the Archipelago (Taang et al. 2020) interaction detection method is applied choose the most important feature interactions from the dataset. After small
30

---

Published as a conference paper at ICLR 2025
number of feature interactions chosen from the dataset, neural network which obeys this low- dimensional GAM structure is trained under the same loss function objective as in the main paper.
Bikeshare This dataset predicts the expected bike demand each hour given some relevant features like the day of the week, time of day, and current weather. There is a total of thirteen different input features predicting a single continuous output variable.
There is a gap in accuracy from GAM-1 to an MLP where the GAM-1 achieves an R² error of 17.4%, whereas an MLP achieves an R² error of 6.59%. Using the techniques of SIAN, we select 20 tuples of size three or less to train a GAM-3 model which achieves 6.23% R2 error, closing the gap between GAM-I=SHAP through the usage of feature interactions.
In particular, it is well known that on this dataset there is strong interaction between the hour vari- able and workday variable (since people's schedules change on the weekend vs. a workday.) The ability to capture this particular feature interaction is critical for accurately understanding the dataset, as seen in Figure spectrum along the axis of additive models supports the hypothesis of Rudin (2019) by demon- strating that training an accurate GAM model sufficient to explain SHAP; however, training an accurate SHAP score is not sufficient for training accurate GAM models.
Also in Figure 4 can be viewed how the
The dataset consists of predicting the types of trees covering specific forest area from selection of 7 tree species (Spruce, Lodgepole Pine, Ponderosa Pine, Cottonwood, Aspen, Douglas-Fir Krummholz) in Colorado national park based on 10 numerical features and categorical feature the area.
Treecover
However, this misses the fact that both the elevation and soil type are additionally correlated with one another. Indeed, the soils are grouped according to climatic zone which generally correspond to different altitude climates. For convenience, we keep these soil classes in the same orders as their expected elevation, named: 'lower montane', 'upper montane', 'subalpine', and 'alpine'. : One notes that the Krummholz tree be found high altitudes but also in alpine (often rocky) soil. Similarly, Cottonwoods, Douglas-firs, and Ponderosas are expected to be found at lower altitudes, but also to be found in montane soils. Without an understanding that these two features are correlated with one another, it might priori seem like these are Iwo independent contributions to the prediction. Yet again, it turns out that these two facts are indeed correlated with one another and hence the 1D projections alone may not be sufficient to yield good explanation.
In this case, lot can be gleaned by viewing the 2D shape function which depends on both the soil and the elevation. In Figure 5 we visualize this 2D shape function as scatterplot with colored heatmap. Through the density of points, we can see there indeed a strong positive correlation between the soil type and the elevation. Furthermore, using the colors for each tree species, we can see that there is a lot of redundant information carried by both the soil and the elevation, but also that there is some non redundant information.
We train an MLP on this dataset to achieve 80.4% validation accuracy and we train a GAM-1 1 to achieve 72.4% validation accuracy. This once again shows a gap in the feature interactions which discredits the ability of SHAP to provide an adequate explanation of what is being Icarned by the MLP model. Once again following the techniques of training lower-order GAMs, a GAM-5 on 50 tuples to achieve 82.2% accuracy. This shows that likely there is some information which the low-order GAM with interactions can understand that GAM-I and SHAP are missing.
are able train
D.4 COMPUTER VISION
We perform experiments on the CUB dataset consisting of 200 different species of birds and con- taining over 6000 labeled images (Wah et al.] (001).. In addition to the species level information, we construct a coarser-grained class label out of the taxonomic family each of those bird species. This results in 37 coarse-grained labels for each bird. For our main CNN we train ResNet-50 model (He et al. 2016) on the masked surrogate objective and for our GAM-KxK architecture we train a modififed resnet to only allow for communication between adjacent patches of size 16x further details in code. Both models are initialized with mostly pretrained weights and fine-tuned for 300 epochs on the CUB dataset.
31

---

Published as a conference paper at ICLR 2025
Our vanilla resnet is able to get to 65.0% fine accuracy and 81.8% coarse accuracy. Compare this to the 33.2% fine and 53.7% coarse accuracies of the GAM-1x1. There is clearly sizable gap in performance between the full complexity ResNet and the GAM-restricted Resnet, indicating the importance of feature interactions and emphasizing the potential deceptiveness of SHAP on this dataset. The GAM-2x2 and GAM-3x3 achieve fine accuracies of 45.8% and 46.8% well as coarse accuracies of 66.3% and 66.8%. This once again indicates that even with some feature interactions, the performance of the Resnet is dependent on even higher order feature interactions longer- range feature interactions. As discussed, it is not impossible that these conclusions are only true for the training method used in the GAM-KxK and that some novel GAM architecture would not be able to achieve higher performance. Nonetheless, the issues in training modern neural network will be present also in alternative approaches like FastSHAP, implying that these conclusions are valid regardless.
Another major challenge of applying to domains like computer vision and natural language process- ing is the prolific usage of pretrained models for downstream tasks. As discussed in Covert et al. (2023), this brings up the important question of how to do surrogate-based modeling to compute the conditional expectation M. In principle, would like to also do the pretraining stage with surrogate masking, however, in practice previous works on this domain (Jethani et al. [2022] Covert al. 2023) will instead use the pretrained models which available and do the fine tuning stage with the masked objectives as we presented in the paper.
We briefly discuss the application of SHAP classification and how it is different from its applica- tion to regression. In particular, it is often common to train on the cross-entropy or Dx objective, but still use the SHAP for regression directly on the logits. There are some potential questions raised about how well this address the nuanced differences between the Dxz objective and the || l/2 ob- jective, is the choice made in previous works (Covert al., 2023). Alternatives for classification like Chapley-Shubik (Enouen et al.) 2024) or Deegan-Packo (Biradar et al.) (2024) tend to focus on the one-hot scenario, limiting their application for calibrated prediction. Accordingly, of our explanations are done on the logits or log-pro rogprobaaiilites of the output prediction.
32

---

Published as a conference paper at ICLR 2025
E ADDITIVE MODELS
E. PURIFIED LOss EQUATION
We first reiterate the Fast-Faith- Rastt-aith-SHHAPP SHAP-K, GAM-K, and Inst--SHAP-GAM- equations
arg min (rr)
Bxmp(rr
E SHAPH))((S)
E
1(T _ 5) () [
(50)
TS
arg min (or brefs
Bxpp(())
Es SuNPAM((S)
[(:;;))
Σ
(T((T)
(51)
TCId,ITIk
[|<(E::)) -
E 1( o [2] TEHNTTISR
arg min Bxmpla)) (ør)ress
E (HAPSHAP(S))
(52)
From the Tast-Paith-SHAPP4 perspective, the major modification we make is to remove the point- wise flexibility each of the SHAP-k estimators ΦT(2) and instead replace each functional ap- proximator with GAM-like functional approximator Φπ(*T). This restricts the capacity of the functional amortizer but as discussed extensively this may give a more accurate representation of the true behavior and also is able to demonstrate improved convergence synthetic datasets.
From the GAM-K perspective, the major modification is to replace the typical unmasked distribution p GGAM(S) (S) (or sometimes nontrivial in fitting techniques like backfitting), with the masking distribu- tion coming from the Shapley kernel distribution y ySHAA (S). The second key modification we include the Instant Mask which only allows the additive influence of each function Φr (xT) flow the final output so long as all of its constituents have been included in the observed mask S. It follows that we may easily calculate downstream explanations of interest like the SHAP value because of the automatic purification of such effects.
E.2 EXTENSION TO ARBITRARY FRONTIERS
Although the extension of GAMs to higher-order interactions of size 3D and larger simple to write down as
FS)(1))
Σ fs(xs)) SEIga
(53)
the exploration of these higher-order GAMs delayed because it is typical to be unable to explicitly model all higher-order interaction sets. For instance, the size of Iş* grows like (dd) which is untenable for most practical purposes. Instead, it has recently been proposed to select only portion of these interactions as important enough to be included in the model Yang et al. (2020); Dubey eet al. (2022) ; Enouen & Liu (2022). We consider these more general additive models by first choosing a candidate collection of interactions I C P([d) and then writing the similar equation:
(x Es(rss
(54)
SEI
Once again, we will say the order is the size of the largest subset k max([S]: : Z); however, there is now much richer set of choices compared with the original hyperparameter selection of k. Hence, despite its simplicity by choosing small number of feature interactions, does not provide reduction in complexity unless we can also answer the question of which feature interactions to include.
E.3 SOBOL SOLUTION WITH INDEPENDENT VARIABLES
If we briefly return to the case of independent variables, find that the aforementioned decompo- sition of variance allows a precise answer to the question feature interaction selection. Moreover, this means that the functional ANOVA space and the GAM spaces are exactly connected with each other.
arg min Eample) for Frez
F(x) - > Cor(er))
Σ Vr T'
(55)
TEI
33

---

Published as a conference paper at ICLR 2025
This of course means that if we have good way to approximate the Sobol indices, then we have easy way to select for interaction tuples by choosing the largest Sobol indices.
In the case of correlated imput variables,
not lucky. Although the Sobol covariances (Rabitz) 2010 Hart & Gremaud 2018) are still able to give a decomposition of the variance function
Cs SCM
(56)
where again Cs == (oxx (Xxs) they no longer provide an answer the effectiveness of an additive model with an arbitrary collection of feature interactions I C P([d). In particular, Cs may indeed be negative whereas adding an interaction to additive model can never decrease its represntational capacity. Intuitively, this corresponds to the case where the "constructive" infor- mation provided by allowing a feature interaction is overshadowed by the 'destructive' information created by the redundancies of feature correlation.
follows that we must able to measure the efficacy of an additive model to represent a function under some distribution in an alternative way. In the sections which follow, will take variational perspective to represent the efficacy of the interaction collection for an additive model and to begin to answer the question of how to distinguish the multiple types of feature interactions.
E,4 ADDITIVE MODEL. SOLUTION FOR ARBITRARY FRONTIERS
Because of the need to further granulate to the level of synergistic interactions and dissonant interac- tions, we find that is necessary study the entire set of possibilities for additive models. We find that only then can one distinguish between the synergistic interactions and redundant interactions of Figure For d
we list all representatives ('frontiers') of nontrivial additive models in Figure For example, the function in Figure 2 would be covered by 1, = {0, {1) (2}) whereas the function from would need to covered by = (0, (1), (2). [1,2]}
10
All Possible 3D Additive Models
1,2
12,3
13,2
23,1
1,2,3
12,13
12,23
13,23
GAM-1
13,13
123
GAM-2
GAM-3
Figure 10: All possible frontiers for GAM model when d = 3.
As mentioned in the main text, one of the critical issues for using additive models to learn particular target function, is solving the meta-optimization find an optimal frontier for the additive model. As of yet, there is seemingly no known measurements for the correlated input case paralleling the Sobol indices in the indepdendent input case. particular, for given frontier I S P(dd) and candidate interaction S C [d] not yet in the frontier, there is seemingly work trying estimate the differences in errors between these two learned additive models. Moreover, noted in the main body that the measurements Cs are in general insufficient measure these differences in all cases, and must only be used as an approximation.
Herein, deseribe the solution to the additive model training procedure the solution the Euler- Lagrange equation from calculus of variations. Thereafter, we simplify our solution into a single matrix-operator functional equation defined by the projection operators Ns in the function space
34

---

Published as a conference paper at ICLR 2025
H. We then provide formal solution to the matrix-operator equation and show how it can be approximated through repeated projections.
Theorem 6. Fix input distribution X p(X) and function y F(x). Consider a collection -S. S P([d) and consider the solution to the additive model training equation:
Ex F(X - Σ (x |1x
{of Jr == arg min (grlr
(57)
TEI
Recalling the conditional expectation projection operators
[Ms Fx)x) Ex s~p(X s|X)>ms) F(rs, X-s)
(58)
where we drop the subscript denoting the distribution p(z). The solution to the variational GAM training equation in Equation 57 obeys the matrix equation:
Ms.
Ms. Mse
gss gs.
fs, Ssa
Msz
(59)
Ms
Ms
Is.
f.
Proof. Let the objective functional be defined
27 |[x)) - Σ (r((X)
}}
((grlr) = Ex
(60)
TEI
Recall from calculus of variations the Euler- Lagrange equation:
8J
0
ōgs.
for each possible S; E I, which then implies:
lim
J(loT ter -or))r))) (-r))-(((orh)))) =0
68, ES,
lim Ex 600 8
||(x) Σ gr((X)) - Ss. (Xs:) [[=1|[(() Σ
gr (XT)
11² =0
TEI
TEI
lim Ex 2ōs;(Xs) [F(X) 800
Σ (r((XT)
83 (Xs:)
=0
8
TEI
Ex
28s, (Xs:)
[F(X) - Σ (T()))
= 0
TEI
Exs,
0s. (Xs.) Bx sx
F(X) Σ gr (XT)
=0
S,EI
E Ex (8s, F(X) Σ
gr((T)
=0
Tel
This can simply be rewritten as:
fs,(X) Ms, Σ gx()
fx) = Mss og, (X)+ Σ
Ms, (X)
TET-S S
fs,(z) = gs,(x) +
Σ
[S o gr)((())
TET-8,
fs. = gs. +
Σ
[S grl
TET-S, S
35

---

Published as a conference paper at ICLR 2025
Hence, it can be that each row of the matrix equation corresponds to partial gradient from the Euler-Lagrange equation as desired. Thus, any possible solution to minimization of the quadratic functional J must obey the above matrix equation.
It is moreover the case that we can reduce a collection of feature interactions to its 'frontier' or its solution only on the largest subsets which have no supersets included. From the perspective of the poset P(d]), this corresponds to the set of maximal elements. It can be seen in the above proof that any solution on projection S C S2 must automatically be obeyed by the operator equation for the larger set S2.
Accordingly, identify a frontier I with its set of maximal elements ,i....,7Ta Tr. Following from ensure that it is enough to solve the matrix equation:
Equation 59
(:
e
MT.
gr.
fn
:
:
(61)
MT
91.
ITe
Which then take the formal inverse of the operator matrix yield solution
911
Mn
JT
(62)
9T.
MT
fTu.
so long we take care with the determinant in realizing that matrix of non-commutative elements does not have a well-defined matrix determinant as it does in the commutative case.
Nonetheless, let illustrate the usefulness of such formal inverse in simple case with our synthetic example from earlier.
(M,,
Ma)
-e
Jx
Fy.
=
fzl
(-Mm,
-Mx) e MoM
(
MM²)
(5
(-M,
-Mr)
(sz (v)
My
6
O e MyM
-f
(fr- Mo o fuy My
- M o Fzz
e
9
e - M,MMy - MgMs
Sg =ff, -,M, Sz)
[a - MLM - fy)
3
Fy.
g
May off Mr o ful = - MyMa] ( - - fal
g
We can then use the formal Taylor series expansion of the operator inverse to yield:
9 Σ (MMMM o [ta [a-- -M ful
n=0
9 Σ (MyMa)n [T My o sal
we choose to denote repeated projections with semicolons, we can then write our solutions as
9h = [a - Syg + Sawyz Sypzigu + Saay - Synennann +
96 = fy Jua + Syarr Sawwzy + + Sgaayaa faayanyy
36

---

Published as a conference paper at ICLR 2025
So then we caclulate this to be
g = (a + pz p) (p'x p²x² + p(1 - - - - --- pp)
= [x] 1 + p² 0² +... +...I+ 1p p² Pp ...
z +
1 p² 1r3 -1
9 = (py + pV p) - (py + +ppp pyy²
2 + p(1 - -²) p) (p'y p3 + (1- -
p(1 -p²)
pp)
0 1 + p² p² +.... 1p p² + pd 19.
0 + 1+ 2 w 1]
may be checked that this solution agrees with that of directly solving Equation 59
2 = 1+ 1+ 1x2 9y
P 1 + pa 11² 1]
It should at the very least be cautioned that these operator manipulations, especially that of the inverse are done only in the formal sense. For instance, considerations of the limit point p = are not able to demonstrate local convergence in the inversion; however, the formula still remains true in this case. considered very likely that these matrix equations are, in most cases, easily able to be solved by the suggested formal manipulations but at least some caution should be exercised.
E.5 IMPLICATIONS
We reiterate how the additive model's ability to distinguish between synergistic feature interactions and redundant feature interactions is key strength which has yet be fully utilized in either the literature on SHAP or the literature on GAMs. By the introduction to the characterization of the set of GAM solutions and drawing parallels with where this aligns and misaligns with the ever-popular functional ANOVA decomposition, we provide further set of tools to explore SHAP which goes beyond the 'feature-olly' perspective of functional ANOVA alone, and begins to explore the 'feature interaction' perspective which adequately handles the intimate complexities which are introduced in the case of correlated variables.
This spectrum of additive models which operates over the entire combinatorially large set of frontiers of additive models is able to give a much more nuanced picture of the underlying structure both the underlying statistical manifold of the input X variables, but in conjunction with the mapping to the output Y variables. Compared with existing theory in functional ANOVA which spans the entirety of the exponential feature interaction space, this variational formulation covers the range of structures living in the doubly exponential space of all frontiers. As demonstrated in this work, such structure can be directly accessed with relatively simple machine learning approaches, that is addi- tive models and feature masking. It is envisioned that there yet remains many directions of further theoretical exploration to more suceinctly and understandably represent the underlying structures of statistical mapping between data, while simultaneously there still exists abundant opportunities in the application these learnings directly to machine learning, particularly bridging the gaps from supervised learning to semi-supervised learning and semi-supervised learning to unsupervised learning.
37

---

Published as a conference paper at ICLR 2025
F ADDITIONAL RESULTS
F.1 ADDITIONAL SYNTHETIC RESULTS
Model SHAP-1 Frior ((*=),, pm0.000)
Model SHAP-1 Erior (ktwl, pm0. p50)
Model SHAP-] Iktrl I. pmo p=0.500) 500)
2.03
Samp KamaEHAP
0.05
KomeISHAP
KemelSHAP
3.04
Fast5HAP InstaSHAP
0.04
FastSHAP
MSE Enror
0.04
FastSHAP
Ins
0.08
MSE
0.02
0.02
9.01
0.01
0.01
420
800
1000
200
800
1000
400
500
800
Number of Evals
Number Evals
Number of Evals
Model SHAP-1 Erior (k*w),, 1. pm0. pm0.707) 7077
Model SHAP-1 Erior (ktwl, pm0.841
Model SHAP-1 Error Iit=1, pmo 0.777
Samp
Samp
KSB Broo
3.01
Kernetii (AP
SHA
0.04
KeM
FastSHAP
0.04
FastSHAP
KernelSHAP
InstaSHAP
InstaSHAP
0.00
Insto SHAP
8.02
0.02
9.01
0.01
0.01
0.00
400
800
600
1000
4.00
800
Number of
Number Evals
Number of
Figure 11: Model MSE Error of SHAP values. Comparison with test-time permutation sampling. k. == 1.
Model SHAP- Erior (k+=2, p=0.000)
Model SHMP-1 Error (kf=2, p=0
Model SHAP Error Ikt=2, p=0 500)
0.05
Porm
0.05
0.05
Perm KemelSHAP
MSE Errer
0.04
KemeEHAP
FastSHAP
Error
0.04
KemelSHAP FastSHAP
MSE Error
0.04
FastSHWP
InstaSHAP
SHAP
IHAt
2.02
0.02
3.01
0.01
0.01
200
420
620
300
1000
200
4.00
600
800
1000
200
400
600
800
1000
Number of Evals
Number of Evais
Number of Evals
Model SHAP-1 Error (k+-2, p=0.707)
Modal SHAP-1 Erior (k+=2, p-0.841)
Modal SHAP-1 Ertor p-0.917)
0.05
Samp
0.05
0.05
MSE Broor
3.04
KeMAISHAP
FastSHAP
0.04
KemelSHAP
FastSHAP
MSE Error
0.04
KemmlSHAP
Fast5HWP
InstaSHAP
ShAP
IHAt
0.02
0.00
3.01
0.01
0.01
200
420
620
300
1000
200
4.00
600
800
1000
200
400
600
800
1000
Number of Evals
Number Evais
Number of Evals
Figure 12: Model MSE Error of SHAP values. Comparison with test-time permutation sampling.
k. -==2 2
38

---

Published as a conference paper at ICLR 2025
Model SHAPP-1 Error [kf=1, p=:0. p=0.000) 000)
Model SHAPP1 Error (k*=1, p=0.250)
Model SHAPP1 Error (k==1, p p=00.000
Samp
Samp
Samp KemelSHAP
NSE Ener (100)
10"1
Kem*KHAP
(log)
KemelSHAP
MSE Enor (log)
10
FastSHAP
FastSHAP
FastSHAP
InstaSHAP
InstaSHAP
InstaSHAP
20.11
10
10
10-1
10-3
10-1
200
400
não
1000
200
600
1000
400
6.00
800
1000
Number of Evals
Number Evals
Number of Evals
Model SHAPP-1 Error Ikf=1, p=: 707)
Model SHAPP11 Error (k*=1. o-0.841)
Model SHAPP1 Error (kr=1, p-0.9177
Samp
Samp
Samp KemelSHAP
MSE Error (lg)
KemeEHAP
Error (la))
KemelSHAP
MSE Error (log)
FastSHAP
InstaSHAP
InstaSHtP
InstaSHAP
10-1
10-1
10-4
10-4
10-44
360
600
1000
400
800
Number
Number Evalis
Number
Figure 13: Model MSE Error of SHAP values (logarithmic scale). Comparison with test-time per- mutation sampling. k* =
Model SMAPP-1 Error Ikt=2, pm:0. p=0.000) (0000)
Model SHAP-1 1 Error (k* p= p50:
Model SHAPP1 Error (k*=2, p=0.500
Samp
Samp
Samp KemalSHAP
MSE Erar (l0g)
KeMAISHAP
MSE (log)
KemeiSHAP
MSE Enrr (log)
FastSHAP
FastSHAP
FastSHAP
InstaSHAP
InstaSHAP
InstaSHAP
10
400
800
1000
200
6.00
1000
400
800
1000
Number of Evals
Number Evals
Number of Evais
Model ShrAP-1 Error Ikf=2, p=o. p07) 707)
Model SHAP-1 Error (k* o.18141)
Model SHAPP-1 Error (k*=2, p=9.9177
Perm Samp
Samp
Samp KemeeSHAP
MSE Error (l0g)
KemetHHAP
MSE (log)
KerelSHAP
MSE Ernor (log)
FastSHAP
FastSHAP
FistSHAP
InstaSHAP
InstaSHAP
SHAP
10-1
10-1
10
800
600
1000
400
800
Number
Number Evals
Number
Figure 14: Model MSE Error of SHAP values (logarithmic scale). Comparison with test-time per- mutation sampling. k" = 2.
39

---

Published as a conference paper at ICLR 2025
Model SMAP-1 Error (k*=1, p=00000)
Model SHAPP1 Error (k*=1, p:0.250)
Kodel SHAPP1 Error (k*=1. p=0.500)
10"1
10-1
10-1
MSE Emor (10g)
InstaSHAP
(log)
InstaSHAP
MSE Emor (log)
In ItstSSHAP
20-3
10.2
10-1
50
15.
200
250
300
130
150
203
250
10
50
100
150
250
100
Number Lpochs
Number of Epochs
Number
Epochs
Model SHAPP1 Error (k*=1, p= 707)
Model SHAPP1 Error (k+=1, p= 0.041)
Kodel SHAPP1 Error (K*=1. p= 0.917)
10"4
10-s
RastSHAP
MSE Error (lg)
Insta5HAP
Error (log)
InstaSHAP
MSE Ernor (log)
InstaSHAP
10-3
10-1
50
150
250
300
100
150
200
250
10-1
50
150
250
500
Number
Number
Number
Figure 15: Model MSE Error of SHAP values (logarithmic scale). Comparison with pre-test-time functional amortization. k- 1.
Model SHAP-1 Error (k* p=0 0.0000)
Model SHAP-1 Error (k*= p=0.250)
Model SHAP-] Error (k*=2, p 0.5000
10
10-1
10-1_
MSE Enar (log)
InstaSHAP
MSE (log)
InstaSHAP
MSE Enror (log)
In staSHAP
20-1
10-1
50
15.
200
250
300
100
150
202
250
50
150
250
300
Number Epochs
Number of Epochs
Number
Epochs
Model SHAPP1 Error (kt=2, p=0. 707)
Model SHAP-1 Error (k*= p= 0.041)
Model SHAP-1 Error (k*=2, p=0.917)
10
10"1_
10
MSE Error (log)
Insta5HAP
MSE Error (log)
InstaSHAP
MSE Error (log)
InstaSHAP
30.2.
t0
10
30-1
10-
50
150
250
300
100
200
250
150
300
Number Epochs
Number
Epochs
Number
Figure 16: Model MSE Error of SHAP values (logarithmic scale). Comparison with pre-test-time functional amortization. k* = 2.
40

---

Published as a conference paper at ICLR 2025
Model SHAPP1 Error (kt=1, p=00.000)
Model SHAPP1 1IAP-1 Error (k*=1, p=0.250)
Kodel SHAPP1 Error (k*=1. p=0.500)
10"1
10
10-
MSE Emar (10g)
distil SHAP
MSE (log)
distil InstaSHAP
MSE Emar (log)
distill InstaSHAP
20-3
InstaSHAP
InstaSHAP
10:2
sHAP
10-1
50
15.
200
250
300
130
150
200
250
300
10-
50
100
150
250
100
Number Epochs
Number of Epochs
Number
Epochs
Model SHAPP1 Error (kt=1, p=00.077)
Model SHAPP1 Error (k+=1, p-0..41)
Kodel SHAP-1 Error (KT=1. 9.0.917)
10"4
10tl
MSE Error (lg)
distil SHAP
MSE Error (log)
disbil InstaSHAF
MSE Ernor (log)
distil InstaSHAP
10-3
10-1
50
150
250
300
100
150
200
250
10-1
50
150
250
500
Number
Number
Number
Figure 17: True MSE Error of SHAP values (logarithmic scale). Comparison with pre-test-time functional amortization. k- 1.
Model SHAP-1 Eror (k*= 2, p= p=0.000) 0.000)
Model SHAP-1 Error (k*= 250))50)
Model SHAP-] Error (**=2, pm p=0.500)
10
FastSHAP
10-1_
10"1_
MSE Enarr (log)
distil Ins
MSE (log)
distil InstaSHAP
MSE Enror (log)
distill InstaSHAP
direct InstaSHAP
InstaSHAP
direct SHAP
10-1
20-1
10-1
50
15.
200 Epochs
250
300
202 Number of Epochs
100
150
250
50
150
250
300
Number
Number
Epochs
Model SHAPP1 Error (kt=2, 2, p=0.
Model SHAP-1 Error (,*=2, p= c=0.441)
Model SHAPP1 Error (k*=2, p 0.917)
10
10a1
10"1
MSE Error (log)
distil SHSP
MSE Error (log)
distil InstaSHAP
MSE Error (log)
distil InstaSHAP
direct
30.3
10
10
30-1
10-
50
150
300
100
150
200 Epochs
250
150
300
Number Epochs
Number
Number
Figure 18: True MSE Error of SHAP values (logarithmic scale). Comparison with pre-test-time functional amortization. k* 2.
41

---

Published as a conference paper at ICLR 2025
F.2
ADDITIONAL VISION RESULTS
Indigo Bunting
Lazili Bunting
Northern Cardinal
California Gull
Prothonotary Warbler
Blue-winged Warbler
Tanager
Gull
1x1
GAM 2x2
GAM 3x3
Figure 19: A repeat of Figure 6
Provided for closer comparison with Figure 20 below.
42

---

Published as a conference paper at ICLR 2025
Indigo Buntiig
Summer Tanager
Califoria Gull
Prothonotary
Blue-winged Warblen
Buntting
Cardinal
Gull
Warbler
GAM
1x1
GAM 2x2
GAM 3x3
Figure 20: An alternate version of Figure 6 which includes the contrastive Shapley value. In each panel, similar-looking species directly compared against one another. For instance, the first two columns compare the Indigo Bunting species (more green) versus the Lazuli Bunting species (more magenta).
In Figure 20 we can see some more granular details about the model explanations with respect to certain species. For instance, the bluer and browner feathers of the two Bunting species, especially in the GAM-Ix1 model. Some other characteristics which seem to be picked up by some of the models are the orange beak and black mask of the Cardinal vs. the ordinary beak and face of the Summer Tanager, the yellow feet of the California Gull vs. the orange feet of the Western Gull; and the different upper backs and eye areas for the Prothonotary and Blue-wwinged Warblers.
43

---

Published as a conference paper at ICLR 2025
F.3 ADDITIONAL HEALTHCARE RESULTS
We additionally train on a tabular version of the MIMIC healthcare dataset which consists of thirty features used predict hospital outcomes. We train ensemble of five additive models using the vanilla GAM training procedure and the InstaSHAP masked training procedure. In Figure ( beloww we display all shape functions learned by the 1D additive model. We plot the mean and one standard deviation according to the ensemble of five models.
The vanilla GAM models achieve accuracies of 91.0%, 91.5%, 90.6%, 91.1%, and 91.2% for an average accuracy of 91.1%. The InstaSHAP GAM models achieve accuracies of 91.5%, 91.3%, 91.2%, 91.3%. 91.0% for an average accuracy of 91.3%. Generally, the InstaSHAP models have more consistent interpretation of the dataset and achieve tighter confidence intervals than the typical training procedure. It can then be assumed a significant amount of the variance between the vanilla ensemble is due to overinterpretaion sensitivity to the natural correlations of the dataset.
44

---

Published as a conference paper at ICLR 2025
Van la GAM
Variaa GAM
Var GAM
1.0
Incossse GAM
1.0
0.0
0.0
4.0
1.0
50
65
160
120
163
2.0
56
60
40
162
Age
max Heart Rate
min Heart Race
tre 6/M
Voi CUN
Ian 64M
Vanits 6AM Intidinn CAN
Irxubrse CAR
Intus se CA2
-1.54
-1.24
170
140
140
76
2.2
-/4
max BRood Presuuee
min Bood Presuue
max Thmmpesstug
min Temperature
Marila CAM
Varily CAM
Varia CAM hruudes CAM
Vardaa Inlidee CAN
hrubbree CAR
320
6000
nih Pa02 F102
0a F.02 mobsured?
Unin Output
Unine missured?
Veri CAM Intadlipp up CAN
Varily CAM hrsss- GAM
Varla CAM hesso-ip GAN
Werla CAN Inttdii CAM
-2.0
BUN
max SUN
WBC
max WBC
Van GAN
Varills GAN
Vani la Instdeee CAM
Var GAM
11500-i5 GAM
1o0lsteep GAM
15s
mih Polasaum
max Porassium
mi8 Sod
Sedium
Insta/14 GAM
3.2
Inbidw GAM
Werih Got
ToleshiP
Birabonnte
Bilnnin
2.0
1.0
haludiie CAP
1.
4.0
Mas la CAM
Vuri G&M
Int
Instadi GAM
Sitrubin
Bii measurdd?
2.5
2.2
2.
2.2
3.0
fress GAM
3.5
3.7-
hessiiip GAN
Intio-i0 GOM
0.3
3.2
Virl ls GAM
3.3-
V1z000-4P UAM
-22
20/002
prssent
predern Malignancy
3b6ere
preser
v9d.
poy suggy
Mctastok Conder
Admission
Figure 21: Shape Functions for the MIMIC dataset
45

---

Published as a conference paper at ICLR 2025
F.4 ADDITIONAL FINANCE RESULTS
We train on the census income dataset which consists of thirteen features used to predict whether or person's income surpasses certain level ($50,000 annually). We train ensemble of five additive models using the vanilla GAM training procedure and the InstaSHAP masked training procedure. In Figure below, we display all shape functions learned by the ID additive model. We plot the mean and one standard deviation according to the ensemble of five models.
The vanilla GAM models achieve accuracies of 82.1%, 84.6%, 85.1%, 84.4%, 84.7%. for an average accuracy of 84.2%. The InstaSHAP GAM models achieve accuracies of 82.1%, 85.4%, 84.4%, 84.7%, 84.7%, for an average accuracy of 84.3%. We again find that the InstaSHAP models have more consistent interpretation of the dataset via tighter confidence intervals over the ensemble. Once again, assumed that the variance in typically trained GAMs is coming from the inability to consistently interpret the correlations which exist in the dataset.
Vellls Lndk
Tall Laf
1.4
28.3 15
Age
Capta Calta Chouardd calans
Capoal Loam moouard dilags)
Boon Worned Wact
Wall Lnb
2.5
220
11t mág
Nowe
Figure 22: Shape Functions for the Income dataset
46


Reviews:
[{"review_id": "lpvjfZCIKC", "reviewer": "Reviewer_pfXM", "summary": "The paper introduces InstaSHAP, a method that leverages the link between Shapley values and Generalized Additive Models (GAMs) to create a more efficient model for interpreting ML models. The authors argue that while SHAP explanations are widely used for model interpretability, they have inherent computational challenges and limitations in representing feature interactions. The authors propose a variational approach that unifies Shapley values with additive models via functional ANOVA. InstaSHAP provides an approach to explain models in a single forward pass, and incorporates a new loss function with output masking to automatically purify GAM models. Experiments demonstrate that InstaSHAP outperforms baselines in approximating Shapley values on synthetic, tabular, and high-dimensional data.", "questions": "1. Currently, InstaSHAP is mainly compared with FastSHAP. Would you consider adding comparisons with other popular Shapley-based methods like KernelSHAP or TreeSHAP?\n\n2. Given that InstaSHAP uses purified GAM models to approximate Shapley values, are there cases where this might result in different Shapley attributions compared to traditional methods?\n\n3. Shapley-based methods, including InstaSHAP, are associational rather than causal. How do you see this gap affecting the performance of InstaSHAP? How can you deal with sensitivity to at least observed confounders?\n\n3.  The experiments are useful, yet most real-world evaluations focus on high-dimensional image data. Did you consider expanding InstaSHAP’s evaluation to real-world datasets where correlated feature spaces play a more critical role?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper introduces InstaSHAP, an approach that unifies Shapley value explanations with Generalized Additive Models (GAMs) to improve computational efficiency and representation for model interpretation. This work is original in its use of functional ANOVA to connect Shapley values and GAMs, and in addressing limitations in handling correlated features—an area where Shapley methods traditionally struggle.\n\nThe paper is technically sound, presenting a useful theoretical framework with a loss function for automatic GAM purification. The experiments are well-designed, testing InstaSHAP across synthetic, tabular, and high-dimensional datasets, with results (at least on synthetic data) that convincingly support the method’s advantages over existing approaches like FastSHAP in representing complex feature interactions.", "weaknesses": "1. The experiments could benefit from more varied, real-world datasets that reflect the complexity of correlated feature spaces in practical applications. For example, exploring datasets in domains like finance or healthcare—where feature correlations are known to impact interpretability—could showcase the utility of InstaSHAP in settings beyond the selected synthetic and high-dimensional benchmarks. Additionally, while the authors compare InstaSHAP to FastSHAP, further comparisons with other recent Shapley-based methods, such as KernelSHAP or TreeSHAP, could provide a more comprehensive evaluation of the method’s relative strengths and weaknesses.\n\n2. While the InstaSHAP method addresses the efficiency limitations of SHAP, it would be helpful to clarify any potential trade-offs in accuracy or interpretability when using the purified GAM approach. Addressing when and why InstaSHAP might yield different Shapley values from those generated by traditional methods could enhance readers' understanding of the practical implications and limitations of this approach.\n\n3. The paper could benefit from a discussion on how InstaSHAP’s approach relates to causal explanation methods. For instance, the authors could consider highlighting situations where InstaSHAP may not capture causal relationships due to feature correlations that stem from confounding variables rather than true causal links.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sCvNcP63NY", "reviewer": "Reviewer_kSd9", "summary": "The paper \"InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantly\" introduces InstaSHAP, an efficient method for calculating Shapley values to explain machine learning model predictions. InstaSHAP leverages a novel connection between Shapley values and Generalized Additive Models (GAMs), allowing Shapley values to be computed in a single forward pass. This variational approach addresses computational bottlenecks in traditional SHAP methods and improves accuracy in scenarios with complex feature interactions.\n\nKey contributions include a unified framework that connects GAMs and Shapley explanations, the development of InstaSHAP for fast Shapley computation, and extensive experiments showing its practical advantages over other methods like FastSHAP. This work makes Shapley-based explanations more accessible for real-time applications in high-stakes areas such as healthcare and finance which advancing the field of interpretable machine learning.", "questions": "Questions:   \nValidation: Would you include experiments with more varied data types (e.g., time series, medical)? Have you considered comparing with Integrated Gradients or LIME?\n\nScalability and Performance: How does InstaSHAP scale on large datasets compared to FastSHAP? Can you share latency and runtime benchmarks?\n\nHandling Complexity: How does InstaSHAP manage complex, high-order dependencies? Have you tested it in scenarios where GAMs typically underperform?\n\nClarity Enhancements: Would you add intuitive explanations alongside complex proofs? Could you improve figure captions for clarity? Any real-world cases showing InstaSHAP's advantages? Are there plans for easy implementation or integration with SHAP tools?\n\nSuggestions\n\nExpand Empirical Analysis: Include experiments on more varied datasets to showcase broader applicability and compare InstaSHAP with methods like Integrated Gradients or LIME for a comprehensive evaluation.\n\nProvide Scalability Insights: Add benchmarks on training times and real-time performance metrics to support the paper’s scalability and real-time claims.\n\nClarify Theoretical Scope: Discuss how InstaSHAP handles high-order, non-linear interactions and elaborate on its limitations with complex models.\n\nEnhance Clarity: Add intuitive explanations for theoretical sections and improve figure annotations to make the paper more accessible.\n\nHighlight Practical Use: Present real-world case studies demonstrating InstaSHAP's benefits and share plans for making it user-friendly (e.g., open-source tools).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper uniquely links Shapley values and Generalized Additive Models (GAMs), presenting an innovative approach to model interpretability. Introduction of \"InstaSHAP\" enables real-time Shapley value computation through GAM-based training which stands out as an original contribution.\n\nQuality: Strong, rigorous proofs link SHAP and GAMs can provide a detailed understanding of representation limits and power. Validated on synthetic, tabular, and image datasets showcase clear improvements over existing methods like FastSHAP and FaithSHAP.\n\nClarity: In this paper, complex concepts are made accessible with formal mathematical definitions and visual aids. Comprehensive context on Shapley values and GAMs ensures readers understand the motivation and methods.\n\nSignificance: The paper addresses gaps in traditional SHAP methods, especially for correlated and high-dimensional data, with real-time applicability. InstaSHAP’s instant computation is highly relevant for fields needing quick and accurate model insights, like healthcare and finance.", "weaknesses": "Empirical Analysis: More diverse datasets (e.g., time series, medical) would better demonstrate robustness. A broader comparison with methods like Integrated Gradients or LIME is needed.\n\nScalability: The paper lacks details on computational efficiency for large datasets. Quantitative analysis of runtime and latency is needed to validate real-time claims.\n\nTheoretical Depth:  More discussion on handling complex non-linear dependencies would clarify limitations. The method's performance in non-GAM-friendly tasks needs clearer boundaries.\n\nClarity: Intuitive explanations alongside proofs would improve accessibility. More detailed captions and annotations are needed for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhHHFU437U", "reviewer": "Reviewer_qEEw", "summary": "The paper titled \"InstaSHAP: Interpretable Additive Models Explain Shapley Values Instantly\" presents a novel approach for computing Shapley values efficiently, aimed at improving both the efficiency and interpretability of machine learning model explanations. Though Traditional SHAP methods are powerful, they are often struggle with computational overhead and limited representation capabilities, especially in the presence of feature interactions and correlated inputs.\n\nThe authors address these limitations by establishing a theoretical connection between Shapley values and Generalized Additive Models (GAMs), which form the basis for InstaSHAP. InstaSHAP enables the instant calculation of Shapley values in a single forward pass which addresses key computational bottlenecks and making it highly practical for real-time interpretability tasks. The paper’s theoretical contributions unify recent advancements in Shapley-based explanation methods (e.g., FastSHAP, FaithSHAP) through a variational framework that highlights the power of additive models for feature interaction.\n\nThe primary contributions of the paper includes, developing a variational formulation that aligns Shapley values with GAMs, providing a new perspective on feature interaction and introducing InstaSHAP, a method that achieves efficient Shapley value computation by purifying GAM models, delivering explanations with significantly improved efficiency and accuracy. \n\nThrough extensive experiments on both synthetic and real-world datasets, the paper demonstrates InstaSHAP’s advantages in capturing complex feature interactions and delivering reliable, interpretable explanations at scale. This work contributes to advancing interpretable machine learning by addressing the practical constraints of existing SHAP-based methods, which makes the way for broader application of explainable AI in high-dimensional and high-stakes environments.", "questions": "1. Additional High-Dimensional Datasets in Evaluation\n\n    Question: Could you provide more insights into why the high-dimensional evaluations were limited to the CUB bird dataset? Was it a matter of computational resources, dataset availability, or other considerations?\n    Suggestion: Including results on a broader set of high-dimensional datasets, such as CIFAR-10, ImageNet for computer vision, or popular NLP datasets (e.g., SST-2 or IMDB), would help demonstrate InstaSHAP’s scalability and effectiveness across a wider range of real-world scenarios. If adding these evaluations is not feasible, a discussion on InstaSHAP’s potential limitations or expected behavior on these kinds of datasets would be beneficial.\n\n2. Intuitive Explanation of the Theoretical Framework\n\n    Question: Some sections, particularly those with dense variational equations and functional ANOVA decompositions  may be challenging for readers without a strong theoretical background. Could you clarify or add a more intuitive explanation of the variational framework and functional ANOVA, perhaps with a real-world example or simplified illustration?\n    Suggestion: A high-level summary or visual aid, either in the main text or supplementary material, could make the theoretical foundations more accessible. This could help readers, especially practitioners, grasp the core concepts without needing to follow every technical detail.\n\n3. Handling of Feature Correlations\n\n    Question: InstaSHAP relies on certain assumptions about feature correlations, particularly with respect to how it handles conditional expectations in feature interactions. Could you elaborate on how InstaSHAP might perform in cases where correlations differ significantly from these assumptions? Are there any specific conditions under which InstaSHAP’s accuracy or interpretability might be compromised?\n    Suggestion: A more detailed discussion on how InstaSHAP manages datasets with varying levels of feature correlation would be valuable. It would also help to provide guidelines on how practitioners can assess whether their datasets align with InstaSHAP’s assumptions, or whether InstaSHAP might be less effective under certain correlation structures.\n\n4. Comparison with Non-SHAP Explainability Methods\n\n    Question: The paper mainly compares InstaSHAP with other SHAP-based methods. Could you provide more context on how InstaSHAP would perform in comparison to other interpretability methods, such as LIME, Integrated Gradients, or counterfactual explanations? For example, what are its expected advantages or limitations relative to these non-SHAP approaches?\n    Suggestion: Adding experiments or a comparative analysis with non-SHAP methods could offer a fuller understanding of InstaSHAP’s relative performance and strengths. This would also give readers insight into when InstaSHAP might be preferable to these other methods and when it might be more limited.\n\n5. Scalability to Higher-Order Interactions and Complex Dependencies\n\n    Question: The paper mentions challenges in scaling InstaSHAP for datasets with higher-order interactions or complex dependencies, which is especially relevant in tasks with spatial or sequential data. Do you see potential ways InstaSHAP could be adapted to handle these scenarios more effectively?\n    Suggestion: Exploring architectural modifications or possible future improvements for handling higher-order interactions, such as multi-scale or hierarchical structures, would be helpful. Even speculative ideas on how InstaSHAP could evolve in this direction could be useful for readers interested in expanding its applicability.\n\n6. Practical Guidelines for Real-World Application\n\n    Question: Are there specific types of datasets or problem settings where InstaSHAP might be particularly well-suited or less effective? Do you have recommendations for practitioners on the best practices for applying InstaSHAP in real-world tasks?\n    Suggestion: Including a section with practical guidelines, detailing ideal use cases or specific considerations for InstaSHAP’s application in real-world scenarios, would provide valuable context for practitioners. This might include advice on dataset types, feature characteristics, or model architectures that align well with InstaSHAP’s strengths.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: This paper brings a fresh approach by combining Shapley values with Generalized Additive Models (GAMs) to create InstaSHAP, which allows Shapley values to be computed in a single forward pass. This is an impressive alignment of two core ideas in model interpretability—using Shapley values for feature importance and GAMs for creating interpretable models. By introducing a variational framework that connects GAMs with Shapley explanations, the authors present a conceptually rich and practical model for instant interpretability. This work also does a great job addressing and combining limitations of methods like FastSHAP and FaithSHAP, enabling it to tackle feature interactions even in high-dimensional datasets.\n\nQuality: The paper’s quality is high, with robust theoretical analysis backed by thorough empirical validation. The authors prove several key points, including the link between Shapley values and functional ANOVA decompositions, and specify conditions where Shapley explanations hold up. They support these theoretical findings with extensive experiments on synthetic and real-world datasets, showing how InstaSHAP outperforms traditional methods both in speed and accuracy. Overall, the paper is well-researched, thoughtfully designed, and clearly accounts for the limitations and assumptions behind their approach, giving a well-rounded view of InstaSHAP’s potential.\n\nClarity: The paper is generally well-written, guiding the reader from theoretical underpinnings to the practical use of InstaSHAP. Key ideas, like the interplay between Shapley values and GAMs, are explained in detail, making it easier to understand how the method works even for readers less familiar with the nuances of either technique. Visual aids and detailed breakdowns of the mathematics further aid clarity. However, some heavily mathematical sections could benefit from a more intuitive explanation to make it even more accessible, especially for readers focused on practical applications.\n\nSignificance: This work could make a meaningful impact on the field of interpretable machine learning. InstaSHAP’s efficiency makes Shapley-based explanations much more practical for real-time use in areas like healthcare, finance, and other high-stakes domains. Its ability to handle feature interactions through the GAM framework could inspire further development in interpretable models that are both fast and powerful. The theoretical contributions here also lay a solid foundation for future work on GAMs and functional ANOVA in model interpretation, which researchers and practitioners could build on.", "weaknesses": "Limited Scope of Experiments\n\nThe experiments on synthetic and tabular datasets give a good picture of InstaSHAP’s strengths, but the evaluations on high-dimensional data, like images or text, feel a bit limited. Given that InstaSHAP is designed for complex tasks in fields like computer vision (CV) and natural language processing (NLP), it would be helpful to see its performance on a broader range of real-world datasets in these areas. Right now, the CUB bird dataset is the only high-dimensional example, so including more CV and NLP datasets could provide a stronger case for InstaSHAP’s versatility in real-world applications.\n\nSuggestion: Adding evaluations on popular high-dimensional datasets in CV (like CIFAR-10 or ImageNet) and NLP (such as SST-2 or IMDB) would better showcase InstaSHAP’s performance on tasks that feature complex interactions and correlations.\n\nComplexity of Theoretical Explanations\n\nThe paper does an impressive job with the theory, but some sections—especially those that dive into the variational equations and functional ANOVA—might feel overwhelming for readers who aren’t well-versed in the math. It would be helpful to have more intuitive explanations in these areas, so readers can grasp the main ideas without needing to go too deep into the technical details.\n\nSuggestion: Including a high-level summary or simpler examples to illustrate the variational formulation and ANOVA concepts could make these sections more accessible. Even a short summary or diagram could help clarify the main points for readers who are looking for an overall understanding rather than technical details.\n\nAssumptions About Feature Correlations\n\nInstaSHAP’s approach assumes certain patterns in how features are correlated, especially in how it handles feature interactions. While this is touched upon, the paper could go further in explaining how InstaSHAP might perform if the correlations between features differ from these assumptions. This would be valuable for practitioners who want to apply InstaSHAP to datasets with different correlation structures.\n\nSuggestion: Providing a discussion or some analysis of InstaSHAP’s performance on datasets with varied correlation patterns could be helpful. Even some general guidelines on when InstaSHAP might face limitations with unusual or weaker correlations would offer more context for its practical use.\n\nLimited Comparison with Other Explainability Methods\n\nThe paper primarily compares InstaSHAP with other SHAP-based methods like FastSHAP and FaithSHAP, but it would be helpful to see how it stacks up against non-SHAP explainability methods like LIME, Integrated Gradients, or counterfactual explanations. This would give a more complete view of InstaSHAP’s strengths and where it might still have limitations.\n\nSuggestion: Adding comparisons with a few well-known, non-SHAP explainability techniques could give a more rounded picture of where InstaSHAP fits into the broader field. Showing InstaSHAP’s performance alongside these other methods on selected benchmarks could highlight its unique advantages or challenges.\n\nScalability to Higher-Order Interactions\n\nThe paper points out that InstaSHAP has trouble scaling with higher-order interactions in high-dimensional data, which impacts its performance on tasks with complex dependencies, as seen in the bird classification experiment. Since handling these kinds of interactions is important for CV tasks with spatial relationships or NLP tasks with sequential data, it would be great if InstaSHAP could be adapted to handle these better.\n\nSuggestion: Exploring modifications to InstaSHAP’s architecture, like using hierarchical or multi-scale structures, could potentially improve its handling of complex dependencies. Even some suggestions for future work in this area would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)