Tackling the XAI Disagreement Problem with Adaptive Feature Grouping
Authors: Gabriel Laberge
Ola Ahmad
Venue: ICLR 2026 Poster (2026)
Keywords: Explainability
Disagreements
Functional Decomposition
Feature Groups
Primary area: interpretability and explainable AI
TL;DR: We consider feature as groups in order to increase agreement among post-hoc explainability methods.

Abstract: Post-hoc explanations aim at understanding which input features (or groups thereof) are the most impactful toward certain model decisions. 
Many such methods have been proposed (ArchAttribute, Occlusion, SHAP, RISE, LIME, Integrated Gradient) and it is hard for practitioners
to understand the differences between them. Even worse, faithfulness metrics, often used to quantitatively compare explanation methods,
also exhibit inconsistencies. To address these issues, recent work has unified explanation methods 
through the lens of Functional Decomposition. We extend such work to scenarios where input features are partitioned into groups 
(e.g. pixel patches) and prove that disagreements between explanation methods and faithfulness metrics are caused by between-group 
interactions. Crucially, getting rid of between-group interactions leads to a single explanation that is optimal according to all faithfulness metrics. We finally show how to reduce the disagreements by grouping features on tabular/image data.

Published as a conference paper at ICLR 2026
TACKLING THE XAI DISAGREEMENT PROBLEM WITH ADAPTIVE FEATURE GROUPING
Gabriel Laberge'& & Ola Ahmad Thales cortAlx Lab
Montreal, Qc, Canada (gabriel. Ggabriel.laberee laberge, ola. www.oolaaaaaa........ atamaa@thhalessgrouupcoomm
ABSTRACT
Post-hoc explanations aim understanding which input features (or groups thereof) are the most impactful toward certain model decisions. Many such meth- ods have been proposed (ArchAttribute, Occlusion, SHAP, RISE, LIME, Inte- grated Gradient) and it is hard for practitioners to understand the differences be- tween them. Even worse, faithfulness metrics, often used quantitatively com- pare explanation methods, also exhibit inconsistencies. To address these issues, recent work has unified explanation methods through the lens of Functional De- composition. We extend such work to scenarios where input features are par- titioned into groups (e-g. pixel patches) and prove that disagreements between explanation methods and faithfulness metrics are caused by between-group in- teractions. Crucially, getting rid of between-group interactions leads to a single explanation that is optimal according to all faithfulness metrics. We finally show how to reduce the disagreements by grouping features on tabular/image data.
INTRODUCTION
With the rise in complexity of Machine Learn- ing models, there has also been a rise in con- cerns regarding the black-box nature of the most performant models. As a result, the field of eXplainable Artificial Intelligence (XAI) has rapidly grown and now proposes myriad of techniques to "explain" model predictions (Molnar 2025).
Longlaie
Medee
Hosniig
One of the main roadblocks XAI is the called Disagreement Problem (DP) (Krishna al. 2022). which refers to inconsistencies between explanation methods. Due to lack of ground truth in XAI, practitioners cannot de- cide which explanation, if any. is the correct one when they disagree. To address this issue, methods like Shapley Values (Lundberg & Lee 2017) and the Integrated Gradient (Sundararajan et al., 2017) have been motivated as the unique explanations satisfying a set of theoretical properties. As such, they are advertised as form of ground-truth. Still, in the case of Shapley Values, it was demonstrated that their "Dummy" property can be violated in practice (Sundararajan & Najmil 2020). Regarding the Integrated Gradient, its properties were proven to be insufficient at specifying unique explanation (Lerma & Lucas 2021
(b)
Figure I: California Housing. (a) PFI, SHAP, and PDP disagree the importance of Long i tude. (b) By grouping features, we increase agreement between the techniques.
Alternatively, some works benchmark explainability methods using Jaithfullss metrics for example the F-score (Tomsett et al. 20200, JiFiellity (Bhatt et al.) (020), INFD (Dai et al. 2022), and Shapley-Weighted Weighted Fidelity (SWF) (Muschalik et al.] 2025). Unfortunately. (un)faithfulness metrics were previously shown be inconsistent an explanation can be ranked first by metric and ranked last by another (Tomsett et al. 2020).
Mainn Correspondance

---

Published as a conference paper at ICLR 2026
Image
Arch
Occ
IG
RISE
LIME
SHAP
L=
L=
L= 0.50
Figure 2: Explaining the "White Wolf" prediction of a ResNet18 using several saliency map meth- ods. (Top). When considering 14 14 patches, the saliency maps highlight different parts of the image. (Middle) To minimize a disagreement objective function, AGREED fuses the various patches. For example, the initial patches covering the wolf's eyes and nose are fused. (Bottom), Eventually, AGREED leads to large patches where agreement among saliency map techniques is increased.
Recent work has unified the various explanation techniques through the lens of Functional Decom- position (Fumagalli al., 2025) Deng al., 2024). We extend these efforts to scenarios where features are partitioned into groups (e.g. pixel patches for images), we identify the root cause of disagreements between explainers and (un)faithfulness metrics: between- group interactions, and we minimize it by adaptively grouping features. Our framework, called Adaptive Grouping to RE- duce Explanation Disagreements (AGREED), is empirically assessed on tabular and image datasets. Figures and]] illustrate AGREED on the California and ImageNet datasets. To resume, our contri- butions are
1. Unifying group importance methods and funtfaithfulless metrics through Functional De- composition and demonstrating that disagreements caused by neween-ggrup interac- tions.
2. Proposing the AGREED algorithm to discover feature groups and assessing its performance tabular and image datasets.
2 BACKGROUND
2.1 FUNCTIONAL DECOMPOSITIONS
All notation used throughout the paper is enumerated in Appendix A.1 We let [d] == (1....,d] d} be
set of d features, X C Rd be the input domain, E X' be an arbitrary input, f X + R be model, and 'D be the data distribution of inputs (æ ~ D). Given a feature subset u C [d]. we denote
its cardinality by |u]. Functional Decomposition aims represent f as a sum of 2d sub-functions
f(x)
Σ fu(x).
where fu only depends on (xz)jEu- The term fo is constant, the terms fa for |u| 1 are called main-effect while the terms ju| 2 are referred to as Jul-way interactions. Functional Decomposi- tions not unique and their definition is often based heuristic for removing feature Tz. One
heuristic consists of freezing 2; at a baseline value by using the replace-function T., X - X defined as
Tu (b. x), = I if j E u otherwise by.
(2)
Treating the bascline ~ B as random leads to the Marginal Decomposition (Fumagalli et al. 2025).
2

---

Published as a conference paper at ICLR 2026
Definition 2.1 (Marginal Decomposition). Given distribution B, the Marginal Decomposition is
fu, = E (r,,((((,, - - Ev,(s)).
(3)
bwg
MC
When B 5g is Dirac delta centered the decomposition falls back the so-called Anchored
Decomposition (Kuo et al. 2010). If B Bind == 11 -i=1 B; (i.e. input features are independent), then becomes the ANOVA Decomposition (Hooker 2004) (see Appendix (A.2). Since the Marginal Decomposition is relative to distribution B, it cannot explain model prediction [(x) in isolation.
Rather, the main effects and interactions explain the Gap f(x:) [s-s([(()] between a specific prediction and the average prediction. For tabular data, it is common to let B D be the data
distribution (Lundberg & Lee (017). while for images it is common to use a single bascline image b: the average color (Ribeiro al. 2016) 0 (Petsiuk et al. 2018).
2.2
EXPLAINING FEATURE GROUPS
Many explainability methods explain the joint effect of feature groups, instead of their individual effects. This the case for saliency maps that first group pixels into super-pixels (Ribeiro et al, 2016 Tasng et al.) 2020) or square patches (Zeiler & Fergus 2014 Petsiuk et al.] 2018). Assuming features are fed to the model, consider a partition
[d] into disjoint groups. This partition can be described with function P [d] [D] that associates each feature E [d] to its group index P(j) E [D]. We will employ the mapping P(u)= :- (P() ; E u} for u [d] to enumerate all groups indices within a |u| tu|-way interaction and the inverse map P (U) == {j E [d] P(j) E U} for U [ D to list all features that are part of certain groups. Finally, P is super-uartiine partition of P
P()) = P(k)= P'(j) P'()k..
When investigating the effect of feature groups on the gap f(x) EewBss the ideal scenario is that of Groupwise Additive Model.
Definition 2.2 (Groupwise Additive Model (Sivill & Flach 2023). Let R C X be hyperrectangle region. A model f X - is called Groupwise additive in R wrr P if there exists D functions 9p ((()) that each only depend on features in group i and such that
D
f(x) = wo +
Σ gp (gp)((1)())) 1=1
Vo E R.
(4)
In this ideal scenario, the contribution of group toward the gap ((x))EE-s(f(0) is unambiguous: Op ((=))(fa))(ee (x) Ex-B(gp -())))pp))))((((((((()))))))))))
(((:)(()]] (b)]. If Equation does not hold however, there is no longer unique group attribution. This has caused the development of myriad of post-hoc explainers: Φ(f. x B, P) E Rº that estimate the contribution to each group i E [D]. Many methods are conveniently defined in terms of coalitional game.
Definition 2.3 (Grouped Coalitional Game), Define the coalitional game Vs.
VU = b-B (f(r [w){b,x))] CUCDD
that applies the replace-function simaltaneously all features within the same group.
Various explainability methods can be expressed weighted sum of marginal contributions for group
8f B.
Σ
((U)Vrr, ( (.
(6)
UCDDN
with (t) E R,, such that
Eucioioo m(U) 1. The joint-PDP (Friedman 2001 and ArchAt-
tribute Tsang al. 2020) 2022 and Patch-Occlusee occlusion wise. Finally, the SHAP LLnddeerg & Lee 2017) uses (uU) (Pu "/D, while RISE (Petsiuk
both employ (1(0) and zero otherwise. The joint-PFI (Au al.)
(Zeiler & Fergus 2014) both consider [([D] {i}} = 1 and zero other-
al. 2018 defines (²U)
2--1
3

---

Published as a conference paper at ICLR 2026
The Integrated Gradient (IG) (Sundararajan et al. 2017) is alternative feature importance that is not easily expressed terms of a coalitional game. Nevertheless, it naturally extended to feature groups
0s ((1 t)b tx:) (zy (g) bj) daz
E b JEP-)((E)) (-
E
by summing over all features within group (Taag et al. 2020). IG, unlike the other methods, is not black-box since it requires access to output gradients instead of model outputs. Recently Cai Wunder (2024) have proposed estimates of the IG that relax these assumptions, making truly black-box.
The calculation post hoc explanations can require evaluating expectations involving B. Through- out experiments, all expectations were estimated via Monte Carlo (MC) with N samples.
2.3 DISAGREEMENT PROBLEM
Many post-hoc explainers have been proposed, each with different motivations. So, it is not surpris- ing that they often disagree in practice (Krishna et al. 2022). To characterize said disagreements, we use the L2 metric (Laberge et al. 2024)
(1, Φ') = E 2,
B,
(8)
2~D
since will see shortly that this loss has many desirable properties when used minimization objective. Other disagreement metrics proposed in the literature, egg. the spearman correlation Schwarzsehild et al. 2023).
(o(f.,, 0' U 2 (00660)):1 E anD
[ ,
do share these properties.
The current literature tackles disagreements between explanation methods by benchmarking them using Janffaittfulness metrics. Many such metries take the form
F(d,S.w)
E
Σ w(U) US(D)
Σ IEU
ar (v((D) ((D))
(10)
F(Ø. f,
Σ w(U) UC(D]
Σ NEU
o. (0(U)
(11)
anD
using = Vf..B.Br. and specific weight w+U) E R4 for any U C [D]. Metrics that take the form of a weighted squared error include Sensitivity-n (Ancona et al. (017). INFD (Yeh et al. 2019
u-Fidelity (Bhatt et al. 2020), and Shapley-Weighted Fidelity (SWF) (Muschalik et al. (025). ). We refer to Appendix A.3. for the complete definition of each metric. Note that the choice of weights impacts which explanation is considered optimal: minimizing F with w(U) 2 D as done in LIME (Ribeiro et al.] 2016) leads to the Banzhaf index (Tsai et al. 2023), minimizing F with w(U) & 1/(0-)) leads to SHAP (Lundberg & Lee 2017). Accordingly, unless we can unan- imously agree on which weighting function w the correct one, (un)faithfulness metrics simply postpone the disagreement problem instead of fixing it.
The unfaithfulness metrics 10 & both assume that features are partitioned into disjoint groups. This contrasts with recent metrics that allow for overlap between feature groups (You al. 2025). Differences between metrics with/without overlap are discussed in Appendix A.3.2 In this paper, we focus on faithfulness without overlap since the explainability methods studied require feature partition.
3 METHODOLOGY
Because of the aforementioned inconsistencies, practitioners do not have a mean of determining which unfaithfulness metrie, and by extent which explanation method, is correct. Each method
4

---

Published as a conference paper at ICLR 2026
employs different weighting scheme μ aggregate marginal contributions (cf. Equation 6 different weight w to aggregate errors of additive reconstructions (cf. Equations TO I1 of proposing optimal choice for either μ w, next objective will be to lessen the impact of these choices.
Instead
Theorem 3.1. The Arch/Occ/LLE/SHHPPPLII attributions can be expressed via the Marginal De- composition
(rr (((d)(1))P(w))
E fLb()) +
E
((P(()))(ss())
(12)
where h is different for each μ. Different explainability methods disagree on how to redistribute between-group interactions (fu,(( with (P(u) 2 2) among the groups involved.
Theorem 3.2. Let R S X' be hyperrectangle region such that supp(D) S R and supp( C R. Let P be a feature partition. Whenever the model f is groupwise additive in R w.r. P, (un faithfulllsss metrics that follow Equations [0 17 M are all simultaneously optimized
F( a)
(13)
for any weight function w and attribution o²"
The proofs are presented in Appendix B.2 Both theorems imply that if there were no between- group interactions (i.e. fu,B = O whenever [P(u)] 2 2), then all explanation methods would agree on the group importance and all (un)faithfulness metrics would be minimized. Therefore, the Disagreement Problem can potentially be tackled by searching for partition P regarding which is group-wise additive, or "almost" group-wise additive. This is trivially achieved by considering single group containing all features, we must trade-off explanation agreement with group sizes. Similarly to recent work, we frame this search as an optimization w.r.t P using a special class of loss functions.
Definition 3.1. A partition loss function Ly(,,B,P) R., should respect:
1. If f is groupwise additive in R. w.r.t P. then for any distributions DD and B such that supp(D) S. R and sBpp(B) S. R. CD P)
2. If additive w.rl to group (i.e. f(x) = gp ((5)) + +. (p- fusing group with another does not impact the loss.
3. Pf is super- partitiinnnn of P, Ly Bind: P') S (B Bind P).
These properties ensure that C is sensible objective minimize w.r. P. Property 1 guarantees convergence once the optimal P is found. Property 2 encourages the minimization algorithm to only fuse groups that interact with some other. Property suggests that, on tabular data, where D, an iterative algorithm monotonically decreases its loss if features are independent.
Theorem 3.3. The L2 disagreements ( (Ee d')
between Occlusion and the
explainers respect Definition 3.7 3.7
Theorem 3.4. The L2 disagreements between explanation pairings that do not include Occlusion (e.g. LIME vs SHAP) break property of Definition disagreement Da breaks property 2.
3.7.
Moreover, using the Pearson correlation
The proofs are presented in Appendix B.3 These theorems highlight that only the La disagreement metric and certain explanation pairings should be considered when infering feature groups. Our algorithm, AGREED, partitions features using the Le disagreement between the PDP/ArchAttribute and PFI/Occlusion
C CGRRED (D.B.P)
E R~D
PDP/Arch
(f, 0 PFUOce
(f,,,
Σ (1) (14)
which are the cheapest explanations to compute. Our minimization of equation 14 w.r.t P follows: 1) start with granular partition {{1)..... {d}}; 2) select the group with the highest potential V (i): compute its pairwise interactions with of candidate groups f!; 4) fuse group with the
5

---

Published as a conference paper at ICLR 2026
group 1' that yields the maximal pairwise interaction; 5) repeat until the objective (cf. Equation 14 falls below c. We refer to Appendix C for the technical details.
Solving minp L CACREED (D, B3 P) yields feature partition that is valid over the support of the distri- butions D and B passed as parameters. In the tabular setting, we set D and B to the data distribution so solving minp L LAGREED (D, D, P) leads to a partition P useful to explain any data point. For image data, we set DD 6n and B ôg to Dirac measures over an image of interest and the base- line. Accordingly, solving minp f LAGREED ( ōb; P) leads to partition P that is only valid for this single image æ. The partitioning algorithm must be run separately on each image.
Regarding scalability, AGREED has complexity O(22N2) (see Appendix C where N is the number of samples used to estimate expectations. Our quantitative experiments on tabular data employed various models, random seeds, and reported explanation disagreemenns/uffattftllee various stages of AGREED. This required subsampling 50 < N I 100 instances to ensure reasonable runtimes. Our qualitative examples on tabular data, however, used N = 1000 samples since they did not require using multiple models/seeds and logging metrics at each step of the algorithm. For images, N = 2 samples are always used (the image and baseline) so the bottleneck is the ( factor, which we control by first partitioning images into W x W patches and subsequently fusing them.
metries at
4 RELATED WORK
There are multiple existing methods for partitioning input features. The PAIRWISE algorithm ad- vocated by Tsang et al. (2020) groups features in three-steps: 1) computes all d(d 1)/2 pairwise interactions between features; 2) retains only the interaction whose strength is above threshold e and organizes them in graph; 3) defines groups the cliques of said graph. This algorithm was applied to tabular and image data. The RECURSIVE algorithm proposed by Sivill & Flach (2023) was demonstrated scale as (d log d) on tabular data. Finally, the IGREEDY algorithm introduced by Xu et al. (2024) for tabular data starts from granular partition { {d}} and progressively fuses pairs of groups until convergence.
Laberge et al. 2024) recently identified feature interactions as the root-cause of disagreement be- tween PDP/SHAP/PFI explanations and minimized them by restricting the baseline distribution B to rule-based regions. Although promising for tabular data, rule-based regions do not work on pix- els. AGREED, in contrast, minimizes disagreements by partitioning features into disjoint groups, methodology that works for tabular data and images.
Prior work has already unified most explainability methods through Functional Decomposition. (Deng et al.] 2024) have previously unified 14 saliency maps while (Fumagalli et al.) 2025) de- veloped a categorization of explanations along three axes: conditional conditional-margiaallaoo marginal-ancchree decom- positions, pure- part-ppatall-lllll explanations, and individuul-oonnt-inenattooo effectss. Unlike ours, these existing frameworks do not consider feature partitions. Moreover, while prior frameworks also highlight interactions as the root cause of disagreements, ours is the first to propose practical methodology to minimize said disagreements.
Contemporary studies propose explaining prediction in images using overlapping pixel patches. To extract patch importance, the model architecture is constrained f(x) = E ( where G C [d] is the ith patch of pixel. Pixel groups could be the receptive field of neurons in a CNN (Brendel & Bethge 2019). or the result of a sparse attention mechanism (You al.) 2025). This differs from AGREED, which is forced to consider disjoint feature groups to remain model-agnostic. As a result, AGREED might be forced to merge many features together, hindering interpretability compared to model-specific methods that allow for overlap.
EXPERIMENTS
An open-source implementatior is provided to reproduce the experiments.

---

Published as a conference paper at ICLR 2026
5.1 TABULAR DATASETS
All tabular experiments were run on a laptop with an 11th Gen Intel(R) Core(TM) i7- 18-1H CPU,
16 threads, and 32 GiB of RAM.
5.1.1 KNOWN GROUND-
SYNTHETIC DATA
We compared AGREED with IGREEDY, RECURSIVE, and PAIRWISE on synthetic data where the model is known to be groupwise additive over X w.r.t some ground-truth partition The exact and estimated partitions were compared for any discrepancies. We set = 'D as the data distribution. For various numbers of dimension d, MC samples N, and five random seeds, generated random data/models with correlated features (see Appendix D.I). Since an optimal partition exists, we set 10-10
All algorithms obtained the ex- ground-truth partition except for IGREEDY. We suspect that this method does not always converge because its termination criterion as- sumes that the loss monotonically de- creases as features are grouped. This can only guaranteed when features independent.
1006
2006
(Nounbo of Iaforasd)
127k
saresd
2004
pirrear
71k
138-
5000
75
300
08
30
20
10 30
To compare the scalability of each algorithm, we compared the number of model calls across different val- Fig-
(a)
(b)
Figure 3: Comparing AGREED baselines synthetic tab- ular data. (a) Scalability of each method w.r. the number of features d. (b) Scalability of each method w.r.t the number of Monte Carlo samples N. Confidence bands range from 5th percentile the 95th
ues of d and N. According ure AGREED and PAIRWISE have a similar runtime, but is the recursive method that scales best with and N.
5.1.2 REAL DATASETS
When studying real-world black-boxes, we no longer know the optimal partitions. We instead eval- uate how much can AGREED reduce PDP/SHAP/PFI disagreements and minimize the Sensitivity-l, INFD, and SWF (un)faithfulness scores.
16), Default-Credif (dd = 23), SPAM (d = 57), and NOMAC (d 118) datasets were investigated as they contain a large number of features, which renders computing all pairwise interactions intractable, We split each dataset into train/test sets with ratio 0.9:0.1, and we trained Explainable Boosting Machines (EBM) (Nri et al. 2019) from the InterpretMI library and Histogram Gradient Boosted Trees (HGBT) from the Scikit package (Pedregosa et al, 2011). Hyperparameters were tuned using 5-Fold CV and random search.
The Marketing
AGREED requires a random subset of N data samples to approximate expectations w.r. D. For the Marketing and Default-Credin we considered N 100 samples and for the other two, we employed N = 50 samples. To account for the randomness that arises from subsampling large datasets, we repeated the subsampling of N points five times using different random seeds, ran the partitioning algorithms, and reported the L2 disagreements and Sensitivity-l, INFD, SWF infidelity scores. The runtimes for AGREED ranged from 10 100 seconds. Note that the partitioning only needs to be done once and can used to compute local explanation any data point.
presents the tradcoffs between disagrcements/((n) faithfullnsss and number of feature groups when rumning AGREED on the Marketing dataset. Other datasets exhibit similar trends, see Appendix D.2.1 From Figure 4 (a), we see that AGREED reduces disagreements between any pairing of ex- plainers although the algorithm is designed to minimize differences between PDP and PFI. This
Stres: pabos://archiverr ics, uct,
wct..cu edu http///caattttt//eeeooooeooeeeet /ttp
htips: httpss//archieeee uns net wdt /der aset /350 fac lofteredi Hicaritel lenta
https //wrs/////oiiioo www.oo.eeee........ nest wwwp/errsstt///

---

Published as a conference paper at ICLR 2026
Hepunuuke peponnt Hagpoigh
130
1004
133
00001
0000
0300
Groage
135
230
Prtonge
1.30
0.53
1o0
Staber Ratur
Narsor Duster Googg
Stadec: Parrax
Stader Ratte Garge
(a)
(b)
(c)
(d)
Figure 4: Running AGREED the marketing tabular dataset: tom row). (a) L2 disagreement between any explainer pairing is reduced as AGREED groups more features. (b-c-d) Three inconsistent (un)faithfulness metrics collectively converge as AGREED groups features.
EBM model (top row) and GBT (bot-
highlights the role of between-group interactions in the disagreements between explainability meth- ods. Moreover, Figures 4 (b-c-d) highlights the inherent inconsistencies between the ranking of (un)faithfulness metrics. For instance, when no features are grouped (D = 16), Sensitivity- claims that PFI is the most faithful explanation, while the other two metrics claim that SHAP is most faith- ful. Also note that INFD and SWF disagree on the ranking between PDP and PFI. By reducing the number of groups (i.e. grouping more features together), all faithuuufulnness metrics collectively converge to zero for either PDP, SHAP, or PFI.
Grouping features reduces the inconsistencies between explainability methods and (un)faithfulness metrics. Nevertheless, the resulting explanation must be interpreted with care since the joint- attribution of group maltivariate function of all features involved. Appendix D.2.2 presents practical examples of how to interpret the joint influence of groups containing at most three features.
5.2 IMAGES
Convolutional Neural Networks (CNN) remain a strong baseline across a wide range of image do- main tasks. Since these models involve the composition on multiple non-linear spatial filters, is unrealistic to find single partition P w.r.t which is groupwise additive across all images from the dataset. Instead, by fixing an image of interest æ, a baseline b, and setting D 8 and B = ôb, AGREED will search for partition w.r.t which the model is group-wiss additive in the region 11==
X. While this could leads explanations with reduced disagreements and increased faithfulness, this also implies that AGREED must be rerun for each individual image leading to a partition that is only valid for this image.
Moreover, since pixels have inherent sense of proximity, further restrict the partition to de- scribe path-connected patches: any two pixels within a patch must be connected via path span- ning said patch. To produce such patches, group fusion is only performed in AGREED if two patches share a boundary. Image experiments ran on a NVIDIA GeForce RTX 3090 GPU with 25GiB of RAM.
5.2.1 KNOWN GROUND-TRUTHE ON SYNTHETIC DATA
We first experimented on synthetic image dataset for which ground-truth partitions are known. The dataset consisted of W W black and white images with a random rectangle drawn into them.
Each image is labeled as = the rectangle is tall and y = it is wide. This classification problem can be solved exactly with CNN that composes convolution layer (with 2 2 filters that detect left, right, up. and down edges), ReLU non-linearity, and linear layer (see Figure [5] (a)). Although this model is not groupwise additive its whole domain, it is groupwise additive in
8

---

Published as a conference paper at ICLR 2026
(M602))
LAGKE (MAE 1e-66
pairnée
Avd
1500
DdZ Condl
100000
08
20
20
(a)
(b)
Figure 5: Synthetic image example. (a) A toy CNN detect vertical and horizontal edges to perfectly classify rectangles while being locally group-wise additive. (b) Comparing the scalability of AGREED and PAIRWISE the image size W increases. (c) AGREED finds four patches to increase agreement between Arch and Occlusion.
the region I1 10, æyy] and the optimal partition p* considers each of the four rectangle edges as separate group. Fixing the haseline = O, we ran the AGREED and PAIRWISE methods (IGREEDY and Recurse partition for any image T.
developed for images), and confirmed that they both converge the optimal
However, the PAIRWISE approach is much more computationally expensive than AGREED, as ev- idenced by Figure (b). Indeed, PAIRWISE scales poorly w.r.t W compared to AGREED. This is because the number of pairwise interactions to consider (W+).. AGREED avoids this complexity by making the assumption that pixel interactions in CNN model are local. This assumption leads efficient convergence on this synthetic data and model, see Figure for qualitative example comparing Arch and Occlusion. Appendix D.3 illustrates how other explainability methods also reach perfect agreement.
5.2.2 SALIENCY MAPS ON MINTÍMAGENET
We studied the VGG16, ResNet18, and ConvNext models pre-trained on ImageNet and explained their predictions on the MinilmageNet subset containing 100 classes and 600 images per class (Ravi) & &arochelll|] 2017). Given the lack of ground-truth partitions, we compared pixel partitioning algo- rithms terms of their tradeoff's between disagreement((un) farrrremmennnnnnffffffullleee faithfunesss and patch size. The PAIR- WISE grouping algorithm was not investigated because, on realistic CNN, the resulting patches are no longer guaranteed to be path-connected. Instead, we compared AGREED with two algorithms previously used ImageNet: the QUICKSHIFT image segmentation algorithm used by LIME and ArchAttribte, and W x W SQUARE patches implicit to Occlusion and RISE. To accelerate AGREED, started with a partition of small 14 x 14 patches. A null baseline b was used to mask out pixels. AGREED took 1-7 seconds per image to generate partition for VGG16/ResNett and 4-30 seconds for ConvNext.
To compute a single explanation disagreement scores that can be compared across pixel grouping methods, we averaged disagreements over 100 randomly chosen test images and over all pair- ings of the Arch/Occ/IG/LLIME/SHH LIME. The (unhuaithfulness metrics were also estimated using the same 100 test images. Results of the experiments on the three models are presented in Figure 6 We see that AGREED offers the most competitive tradeoffs: lower disagreements/(umg (anggrrremenssuuuaaaifuuuleeee faithuuuuns for smaller patches on average. In the case of ConvNext, AGREED is the only method that is able to consistently reduce disagree- ffant//uuunaaathuullesssss For the other two methods, these quantities can either stagnate or even increase pixel patches grow in size. We suspect this occurs because stronger feature interactions are involved ConvNext compared to the other two models. Grouping the pixels involved in said interactions appear to be necessary to obtain saliency maps that agree with each other and that more faithfull.
explainers. RISE was excluded because it is equivalent to
Appendix D.4 presents saliency maps yielded by AGREED, which tend to generate large impor- tant patch that covers the object being classified, although there are exceptions. This suggests that, when explaining the function, that maps pixels class logit, all pixels that cover the object interact

---

Published as a conference paper at ICLR 2026
40
030
INTO
191
6000
2040
800
3001
840
3801
(08
Groap
Zomri Goog Sax
Armpp Goop
Gran
yitk
3.5
200
DXPT
100
Ancigs Groap Skot
Jorra Goop Six
Aomp Goog Si
Annas Groez She
IVP
2:30 Aunuae Sia
360 Avergp Goop Sbc
430
20:0
201
Anuage Goup
Aearae Grous
(a)
(b)
(d)
Figure 6: Running AGREED, QUICKSHIFT, and SQUARE on CNNs pre-trained on ImageNet: VGG16 (top row), ResNet18 (middle row), and ConvNext tiny (bottom row). (a) The average ArchOOc/LLME/SHAPIG L2 disagreement versus the average size of pixel patches. (b-c-d) The INFD (un))aithfullnss metric of Arch, Occ, and IG for various sizes of pixel groups. AGREED offers the most competitive trade-offs: lower (issgreements/(un)faifiiiii
for smaller patches.
strongly with its neighbors. Since AGREED is constrained to output partition of pixels (no overlap), pixels covering different semantic elements (e-g. differrent object parts) must be fused no interactions are remaining. In future work, we envision extending AGREED to return overlapping pathces by applying the orignal algorithm on each the functions f that map input pixels concept activations in hidden layers (Fel et al. 2023). Explaining the input-concept mapping (instead of the input-logit one) might also allow for more semantically meaningful explanations when the related concepts are interpretable by humans.
that
CONCLUSION
We unified feature groups explanation methods through Functional Decomposition. We identified the culprit that prohibits agreement among the methods and (un faithfulless metrics: between- -gruup features interactions. The strength of these interactions AGREED, that iteratively fuses feature groups whenever they interact. On two data modalities, AGREED was demonstrated to efficiently reduce inconsistencies between explanation techniques and (un)faithfulness metrics. AGREED is broadly applicable to tabular and image data although both structures are treated differently in the algorithm. Future efforts could extend other modalities where groups are natural, for example time-series and text.
minimized using novel algorithm,
For tabular data, the remaining challenge is to provide an automatic methodology for visualizing the joint attribution of feature groups involving more than three features. We envision combining AGREED with regional based explanations (Laberge et al. 2024) to interpret these high-dimensional functions regionally. For images classification, we managed to increase agreement betwee the Ar- Stt.riiuueeOccuuuuo//////AELLLLL
saliency maps. In most cases, the pixel grouping al- gorithm returns single patch of upmost importance that covers the whole object. This clarifies "where" " the network looking but not "what" semantic elements the network seeing. Combin- ing AGREED with concept-basee based techniques (Fel et al., 2023) could potentially make explanations more semantically meaningful and might also lead to smaller pixel patches since overlap is allowed between patches related to different concepts.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Marco Ancona, Enea Ceolini, Cengiz. Öztireli, and Markus Gross. Towards better understanding of gradient-basee based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06. 1.06 2017.
Quay Au, Julia Herbinger, Clemens Stachl, Bernd Bischl, and Giuseppe Casalicchio. Grouped feature importance and combined features effect plot. Data Mining and Knowledge Discovery, 36(4):100111400, 2022.
Umang Bhatt, Adrian Weller, and José MF Moura. Evaluating and aggregating feature-based model explanations. arXiv preprint arXiv:2005.00631, 2020.
Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-featuree models works surprisingly well on imagenet. In International Conference on Learning Representations, 2019.
Yi Cai and Gerhard Wunder. On gradient-like explanation under a black-box setting: when black- box explanations become good as white-box. In Proceedings of the 41st International Confer- ence on Machine Learning, pp. 5360-5382, 2024.
Jessica Dai, Sohini Upadhyay, UIrich Aivodji. Stephen Bach, and Himabindu Lakkaraju. Fair- ness via explanation quality: Evaluating disparities in the quality of post hoc explanations. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 203-214 2022.
Huiqi Deng. Na Zou, Mengnan Du, Weifu Chen, Guocan Feng, Ziwei Yang, Zheyang Li, and Quan- shi Zhang. Unifying fourteen post-hoc attribution methods with taylor interactions. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 46(7):4625-4666 2024.
Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2711-2721 2023.
Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pp. 1232, 2001.
Fabian Fumagalli, Maximilian Muschalik, Eyke Hüllermeier, Barbara Hammer, and Julia Herbinger. Unifying feature-based explanations with functional anova and cooperative game theory. In In- ternational Conference on Artificial Intelligence and Statistics, pp. 5140-5148. PMLR, 2025.
Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. journal of Com- putational and Graphical Staristics, 24(1):44 65, 2015.
John C Harsanyi. A simplified bargaining model for the n-person cooperative game. International Economic Review, 4(2):194 1200220--22222 1963.
Giles Hooker. Discovering additive structure in black box functions. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 575-580, 2004.
Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner's perspec- tive. arXiv preprint arXiv:2202.01602, 2022.
F Kuo, I Sloan, Grzegorz Wasilkowski, and Henryk Wožniakowski. On decompositions of multi- variate functions. Mathematics of computation, 79(270):553-66. 2010.
Gabriel Laberge, Yann Batiste Pequignot, Mario Marchand, and Foutse Khomh. Tackling the xai disagreement problem with regional explanations. In International Conference on Artificial Intel- ligence and Statistics, pp. 2017-2025. PMLR, 2024.

---

Published as a conference paper at ICLR 2026
Miguel Lerma and Mirtha Lucas. preseeviin paths in integrated gradients. arXiv preprint arXiv:2103.13553, 2021.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference neural information processing systems, pp- 4768 4777, 2017.
Jean-Luc Marichal, Ivan Kojadinovic, and Katsushige Fujimoto. Axiomatic characterizations of generalized values. Discrete Applied Mathematics, 155(1):26 43, 2007.
Christoph Molnar. Interpretable Machine Learning. 3 edition, 2025. ISBN 978-3-911578-03-5.
URL hattps::/chriittptttttt hwtpth/wpoo.iit........... github. io/ http:/iioottteeeta..e.ee...
Maximilian Muschalik, Hubert Baniecki, Fabian Fumagalli, Patrick Kolpaczki, Barbara Hammer, and Eyke Hüllermeier. shapiq: Shapley interactions for machine learning. Advances in Neural Information Processing Systems, 0111130024--333555 2025.
Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv: 1909:09990092,, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-28363 2011.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv: 1806.0721, 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International conference on learning representations, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference 2016.
on knowledge discovery and data mining, pp.
Avi Schwarzschild, Max Cembalest, Karthik Rao, Keegan Hines, and John Dickerson. Reckoning with the disagreement problem: Explanation consensus as training objective. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 662-678, 2023.
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, pp. 307-317, 1953.
Torty Sivill and Peter Flach. Shapley sets: Feature attribution via recursive function decomposition. arXiv preprint arXiv:2307.01177, 2023.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Inter- national conference machine learning, pp. 9269-9278 PMLR, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Pro- ceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328. PMLR, 2017.
Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 6021-6029, 2020.
Che-Ping Tsai, Chih-Kuan Yeh, and Pradeep Ravikumar. Faith-shap: The faithful shapley interac- tion index. Journal of Machine Learning Research, 24(94):1-42, 2023.
Michael Tsang, Sirisha Rambhatla, and Yan Liu. How does this interaction affect me? interpretable attribution for feature interactions. Advances in neural information processing systems, 33:6147- 6159, 2020.
Sascha Xu, Joscha CÂžppers, and Jilles Vreeken. Succinet interaction aware explanations. arXiv preprint arXiv:2402.05566 2024.
12

---

Published as a conference paper at ICLR 2026
Chih- Kuan-Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar. On the (in) fidelity and sensitivity of explanations. Advances neural information processing systems, 32, 2019.
Weiqiu You, Helen Qu, Marco Gatti, Bhuvnesh Jain, and Eric Wong. Sum-of-parts: Self-attributing neural networks with end-to-end learning of feature groups In Forty-second International Con- ference on Machine Learning, 2025.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. $18-833. Springer, 2014.
13

---

Published as a conference paper at ICLR 2026
A
EXTENDED BACKGROUND
A.1 NOTATION TABLE
Notation
Definition
Sets and Partitions
d
Number of input features. Set of all features indices.
[d] (1,2....,d]
J E [d]
is an input feature index. u is a subset of input features indices.
S [d]
D
Number of features groups.
[D] := (1,2.....
Set of all groups indices. is group index.
[D
U is a subset of groups indices.
P: D
Partition of [d] into D disjoint groups Partition mapping of feature subset
PPu): - P(j U3
Partition inverse map of the group subset U.
Explanations and Cooperative Game
X Rd
Input domain.
X
Input to explain.
D
Probability distribution for æ D.
b x
Baseline used as reference.
Probability distribution for b ~ Sometimes equal to D.
f: R
Model to explain.
Tu Tu: x X +
Replace-function.
f(x) Eugid fu(a:)
Functional Decomposition.
fu,(())
Marginal Decomposition.
6(h, B,P) E RD UJa.8.P
Group Importance toward the gap f(x) - [x--s((())
Coalitional game.
A A
Harsanyi Dividend.
PD,
Partition loss that depends on D, B, and P.
Table 1: All notation used throughout the paper.
A.2 ANOVA DECOMPOSITION
When the baseline distribution represents independent features (i.e. B Bind -i=1 B3), the
Marginal Decomposition (cf. Def [21l] falls back to the ANOVA decomposition (Hooker 2004).
This functional decomposition enjoys additional theoretical properties that do necessarily hold for the Marginal Decomposition. Notably, the components fu,B.. are zero mean and uncorrelated
+ 0
=
[Su,Brus (x:) 0.
(15)
u + u
E (x) frr (x)) = O.
(16)
Letting 02 : Eer [... (ar) 2 the total variance of the model can be decomposed
E (f((x) - Jo. = Σ B-a
(17)
W [d]
1u|21
This property is where the terminology ANalysis Of VAriance comes from. Note that this elegant decomposition of model variance is not guaranteed to holds if features are non-independent. In that case, Equation 17 potentially involve negative/positive correlation terms between fu and fo- Since feature independence is unlikely to hold on realistic Machine Learning datasets, the ANOVA decomposition will be of solely theoretical interest: it is used derive Property 3 of Definition 3.1
14

---

Published as a conference paper at ICLR 2026
A.3 UNFAITHFULNESS METRICS
A.3.1
NO GROUP OVERLAP
Many unfaithfulness metrics proposed in the literature can be framed as
F(o.f.w)
Σ
w(U)
Σ
Φ (f.,
(18)
2nd
UCIDI
İEU
or
Φi x. B, P) (r,,,,,(()) ( ( ]
F() = E Σ w(U)
Σ REU
USIDI
(19)
using a distinct weight w(U) for every coalition U C [D] The Sensitivity-n metric proposed by Ancona al. 2017) and the j-Fiddelity of Bhatt et al. (020)) report the F metric using the weight w(U) = (r) for [U] = n and w(U) = otherwise. The INFD metric Ych et al. (2019) also employs the F metric but it weights the various coalition uniformly: w(U) -
The whighey- Fidelity (Muschalik et al. 2025) reports the F score using the weights w(0) = ((D = oc and w(U) & (1D1-1) -1 The infinite weights can be seen as a hard con- straint that the attribution should respect the efficiency axiom: they should sum up to the model prediction (Tsai al., 2023). Nevertheless, for the purpose of comparing different explanations quality, we will be ignoring the edge cases U = 0), D when computing this faithfulness score.
A.3.2 WITH GROUP OVERLAP
There is on-going research on how to define Self-Attibuting Neural Networks (SANN) that have build-in notion of feature group importance. This is typically done by restricting the model chitecture f(z) Σl ( where G; S [d] the ith group (You et al. 2025). The proposed build-in patch importance is then d:SANN (f, a:) == fi(rc.). ). Since groups can overlap (i.e. G n G1 * 0), the unfaithfulness metrics discussed in the previous section are not applicable to these explanations.
The literature on SANN uses alternative measures of unfaithfulness. Fixing B = 6g to null image, letting C |d] be subset of features interest, and letting (GiG2,, G2,..... Gn) be D feature groups G C [d] that possibly overlap, the Insertion and Deletion errors are defined as (You et al. 2025
(nEEm((f,,,S) = ((rs((0 ( - f(0)
Σ igD:G.Cs
of(fxx)
(20)
DelEmrff = - f(r((), iSD
Σ
x)
(21)
These two definitions are generalizations of Equations 19 & 18 that allow for overlap between groups. Indeed, when the groups G; happen to define a partition of [d]. then these new definitions fall back to the previous ones.
This generalization does not come for free though. Theorem 3.2 highlights that being group-wise additive w.r.t to partition of [d] is sufficient to reach optimality w.r.t Equations 19 & I8 Attaining an unfaithfulness
0 according to Equations 20 & 21 requires different assumptions about the
model. You al. (2025) proved that the InsErr and Deltir of SANN explanations are whenever the model takes the form
f(x) Σ p((a,,), with p=(
Σ ask k=1
II
higk T
(22)
JEG.
where a:y E R are coefficients the polynomial and bizk E 21,,,3,... 3... is the exponent for feature j in group in the polynomial term indexed by k. It is crucial that all xj within group j G,
15

---

Published as a conference paper at ICLR 2026
have non-null exponent. This structure is respected by the SANN f(x) 1132 124A with built-in groups G1 = {1 and G2 = {2,3}. The corresponding InsErr and DelErr scores using S = 21 and æ (1,
InsErr(f, SANNN S) =I(((1,,0) - f((0,0.0)
(f. -(00001)))) oSANN
1/=0. 0.
(23)
DelEmr(S, SSANN, S) = ((0.0) - SSANN - 0. (24)
This confirms that SANN explanations are the most faithful explanations of f according to both metries. However, if the model does not respect Equation 22 egg. f(x) = sin(x; + x22 + sin +
43), using the same groups 1 21 and G2 32 no longer leads to null InsEr
InsErT(f. SANN .)
sSANN (f. x) (inn()) sin * 0.
(25) In short, whether not one should consider overlapping vs non-overlapping feature groups may depend on the model. Non-overlapping groups may be preferable when f is assumed to be (ap- proximately) group-wise additive. Overlapping groups may be more pertinent if the model can be (approximately) expressed in the polynomial form outlined in Equation 22
16

---

Published as a conference paper at ICLR 2026
B PROOFS
B.1
HARSANYI DIVIDENDS
Before unifying the various post-hoc explainers proposed in the literature, we must introduce an additional definition.
Definition B.1 (Harsanyi Dividends (Harsanyi, 1963)). players, its Harsanyi Dividend A is defined recursively
Given coalitional game involving D
=(U)==((U) -000)-An A(V,, VCU
(26)
for any coalition U - [D]
The base cases are (0) = (0) and ((()) ((()))
Harsanyi dividends (U) can be interpreted as the excess gain of coalition ((0) that cannot
be explained by any cooperation between strict non-empty subsets C U. We can reorganize Equation 26 to express any coalitional game in terms of its dividends
v(U)=
Σ A(V) for any U C [D] VCU
(27)
Similarily, we can reexpress the definition of the Marginal Decomposition (cf. Definition 2.1
E (r,(((x))))
Ee.s(z) for any S [d].
(28)
b
vC
The ressemblance between Equations 27 and 28 is striking and suggests deeper connection be- tween the Marginal Decomposition and the Harsanyi Dividend. Our goal with this subsection is to highlight the link between these two related (but different) concepts.
Lemma B.1. Given a partition P [d] + [D], the following holds
u C p- PU) > P(u) CU,
(29)
where u S [d] and U C [D] are subsets of features and feature groups respectivelly.
Proof.
We start from this simple consequence of the definitions of P
j j JEP = P() E.
(30)
The goal of the lemma is to translate this equivalence to feature subsets u and not just a single feature Letting C [d], we have
u C P (U) = ViEu: = viEw P()) EU = P(u) CU
(31)
Lemma B.2. The Harsanyi Dividend A B...B.P for the coalitional game of Definition expressed in terms of the Marginal Decomposition
2.3 can be
A1,e,B,(UU)
Σ
fu,((z),
(32)
under the convention that P(0)= = 0.
Proof. The proof proceeds by induction. The base case covers all dividends A A,,,, such that |U] < 1. Indeed, have:
A (,,,8(((1)) (.,, := b-B
[f(b] fo,s(2).
(33)
17

---

Published as a conference paper at ICLR 2026
and
Afa.,,r((i)):
b-B b- =1Bg((rp---0))) x)) b-8 [(()
(cf. Definition 2.3
E [uB((r) - (0)
(cf. Equation 28
"CP ((0))
Σ
[u,B(1) S0.
(cf. Lemma B.1
= E
fub(()).
Now fixing U [D] and assuming the premise holds for all V c U, have
A (U,) 1',,a,B.P(U)- - Σ AA,,BB;((
(cf. Definition B.1
VCU
E (115(r=-40)) (b, 16 -A (A,BB,((
(cf. Definition 2.3
VCU
E fu,b(x)- - Σ As
(cf. Equation 28
"CP (U)
VCU
= E SuB(w)) Σ A.
(cf. Lemma B.1
VeU
Σ Sub(a))
Σ
Σ
Su,)(
VCU
(u)):
(By the recursion Assumption)
E
SL,((
Σ
[us(3)
(C
NC(4:P(W)CU
Σ
(a)(()))
concluding the proof.
18

---

Published as a conference paper at ICLR 2026
B.2 UNIFICATION
Theorem B.1 (Theorem 3.1). The ArcHOOc//LME/SSHPPPRSSSS Atriibutions following can be pressed via the Marginal Decomposition
f. P)
Σ f.,B(.).+ + (C(d)()))(()))
E
((P(())))..())
(34)
where h is different for each μ.. Different explainability methods disagree how to redistribute between-group interactions (f), with P(u) 2 2 among the groups involved.
Proof. The proof of the theorem consists of expressing the attributions in terms of the Harsanyi Dividend A B (cf. Definition B.1 2) Use Lemma B.2 to express the Dividend in terms of the Marginal Decomposition.
ArchAAtrribute/Joont--DD
can be expressed in terms of the Harsanyi Dividend
of Auach (f.x, = A,,,8,,((()))
(35)
Following Lemma B.2 its holds that
Arch (f. x,
Σ
f,,((x).
(36)
" [(():P())(()
Patch--Oclusion//oonn-P11 is also easily expressed in terms of Harsanyi Dividends
0 0cc
B, [D)) V. (i))
Σ A (,,,,,P((U)
Σ
A (U,
(cf. Equation 27
(E(D)
U
Σ
(,
UE(D]aet
Σ
E fu,)(a))
(cf. Lemma B.2
US(D)
P(U))
Σ
fu.B(æ).
(C()))P(()
Shapley Values known to redistribute the Harsanyi Dividends evenly between all players in- volved (Shapley 1953)
0 SHAP
f,
Σ
(U)/UI
Shapley 1953
U UID
Σ
|0|-
Σ fa,8(x)
(cf. Lemma B.2
UCDD
c(dF:P(u)=U
Σ
Σ
Su.s(a)/(((u))
(Since [U] = IP(u))
UCD) lus(et)/\Pw))
Σ
vCId):E(())
RISE computes
RISE
1
Σ
(U DfF,,,,(UU,
(37)
2D-I
UCP
19

---

Published as a conference paper at ICLR 2026
which is actually the Banzhaf Index (Marichal al. 2007). is well-established that this attribution method assigns a score to each player by sharing Harsanyi Dividends using a power-of-two rule
RISE B,
Σ
A (1)),_
(See Page 8 from (Maricall et al. 2007
UCID)iEU
Σ
1/2101-1
E
fu,B(æ)
(cf. Lemma B.2
UgiD
"CId:P(u)=U
E
E
(Since [0] (P()))
US(D]iEU uCld)p(u)=U
= E
/
LIME advocates fitting linear model on the function output evaluated masked inputs
(Wo, W1, w2, WD) argmin WERDI1 2D
Σ (V),,,,P(UU) wo Σ w. e UI) UCID)
(38)
and reporting the coefficients (Wi, W2,,, wp) as the local feature-groups attributions. This mini- mization problem is alternative formulation of the Banzhaf Index (Tsai al. 2023) LIME is equivalent to RISE.
We have thus proven that ArchhOce/LLI///SS//SHAA can be expressed
Σ
fu,b(x)+
Σ
M(P(W))a,,B()))
(39)
where h((P(w))) ArchAttribute, M(P(())) for Occlusion, M(((()))) 1/(P(u)) for SHAP, and h()P(u))) 1/2 for RISE/LIME.
Before highlighting the critical role between-group interactions the (un)faithfulness metrics, first recall the Minimality property of the Marginal Decomposition.
Corollary B.2 (Corollary A.1 from (Laberge et al. 2024 Ler R C Rd be hyperrectangle and let f Rdd be a function that can be written f(x) Eugid Ia(x:) Va R, where Ju only
depends on au- Also, assume that subsel v C [d] exists such that
2 Vx gx(xx 0.
Then, for any probability distribution B such that supp(B) C R. the Marginal Decomposition re- spects
" 2 = Vo E R fa,(()) = 0.
Minimality implies that the Marginal Decomposition will not contain interactions that are not present in the model in the first place. This theorem induces an important corollary.
Lemma B.3. Let R be a hyperrectangle region and let f be groupwise additive in R w.r. partition P. Then for any distribution B such that supp( (3) C R, it holds that
IP(b) 2 2, x E R + fv. = 0.
Proof. Let of be groupwise additive in R w.n. P. Thus there exists rectangular region R. such that supp(B) C R, and f can be written as
D
f(x) = wo + Σ
9p-1(1y)(2))
9p-
Va e R.
i=1
20

---

Published as a conference paper at ICLR 2026
Thus, can be written in the form Eugld [d] 9u where [P(u) 2 9u 0.
Now, letting æ E R an input and v S [d] be any feature subset such that [P(o)] 2 Any superset
p o respects [P(u)] 2 [P(x)] 2 2 and so u 2 v + gu(x) = 0. By minimality (cf. Theorem
B.2 this implies that (v, and so
IP(x) 2 2. R f,,8(x) O.
With this corollary now proven, can demonstrate how group-wise additive models have faithful explanations according to the various (un)faithfulness metrics.
Theorem B.3 (Theorem 3.2 Let R C X be hyperrectangle region such that supp(D) C R and supp(B) C R. Let P be a feature partition. Whenever the model f is groupwise additive R w.r. P. (unIfaithfulness metrics that follow Equations 170 and are all simultaneously minimized
F(6", f. w)
(40)
for any weight function and attribution d²
Proof. Fix the set C [D]. The metrics F and F are simply weighted aggregates of the difference between Eien of (J, B, ) or 8, (U)) ",,,,,,.7 (0) Vj...BB.((D) ", \U).
By Theorem 3.1 and Lemma [.3 the first term is equal to
Σ iEU
(J.
Σ
Σ
Su,a(æ))
iEU (I)
Σ
lus(()).
By Corollary B.2 the other two terms are also equal to this quantity V1,,,B,(UU) - V1,,,,BP(0))
Σ
AI..B - AI.0,8P(()))
(cf. Eq 27
VCU
Σ
A5æBB,((
vÇU.V*
E
Σ
fu,((2)
(ef. Lemma B.2
VÇU.V uCL))
Σ
fu,((z)
Σ
fu,8(x)
(cf. Lemma B.3)
((,)),,,p(DD) ((D) = Σ
A
Σ A (ef. Eq 27
VC
VCLD
Σ
AL
VCDDvDU+0
Σ
Σ fu,((x)
(cf. Lemma B.2
VCID)VUU ((d):
= Σ
fu,((x)
Σ
[u,((
(cf. Lemma B.2
Σ
fu,(()).
PCCC: P(W)==
21

---

Published as a conference paper at ICLR 2026
B.3 PARTITION LOss FUNCTION
In this section, we demonstrate the various properties of partition loss functions, which will be used infer feature groups. We start by presenting a link between the cardinalities [P(u)] and (P(()) when P is a superpartition of P
Lemma B.4. Let P' be superpartition of P. then for any subset u S [d] is holds that
IP(u) 2 (P)(.)
(41)
Proof. Recall the definition of superpartition: P a superpartition of = P()) = P(()= P'()). Conversly, (i))) * P'( = P(t) + P(j) must hold. This implies that [P(u) [P'(u) for any u. C [d]. To prove it, assume the opposite holds: [P(w) < [P)(,)] This implies the existence of (u) points i, E such that Pi) o P'(j). However, by the definition of superpartition, P(i) + P()) must also hold for these (P'(u) points, which contradicts the assumption that P(u)
has smaller cardinality than P'(u).
Lemma B.5. Define the objective
L,(
E wnD
Σ M
(42)
for some function w such that w(P(u), 2 w(P((u), 2 for any interaction u. Then
C respects Definition 3.7
Proof. We prove the function Ly from Equation 42 respects the three properties of Definition 3.1
Property
Let be group-wise additive in R w.r.t P. Lemma B.3 states that, for any probability distribution such that supp(B) S R. supp(D) S R, the following holds
(P(t) 2 2, x E R fr,5(x) = 0.
Sampled inputs a: ~ D are guaranteed to land R (since supp(D) C R2) and so
(D, E
Σ a.CLd
" ((WW)),)ll)ffaott)f)dud
E [0] 0.
PP
Property Given a partition P of [d] into D groups, assume w.l.o.g that f is additive w.r.t group D. Also, assume we wish fuse group D with group D which will lead super partition p' such that
P'(1== = P(i) ViEP 10-P-1(11, 21,, D- - 13),
(43)
but D-1 ((D)) The loss under partition P can be written
C, aND
Σ a.eCld IP(x) 22)
E
Σ
m
G.C Ptu]2211P())22 DEP
(44)
Σ a.C |P())22, >2 DgP( and D$P(()
((P()))
22

---

Published as a conference paper at ICLR 2026
We prove that (, (, by rewriting both summation terms of Equation 44 For the first term, we exploit the fact that f additive R w.r. group D, implying the existence of rectangular region R such that
f(z) = 9p 90-90---10000 ()))(())(()))-)))))))))))))))))) + ((I))..D-I))(())) Vo R.
Hence, f can be written in the form Engid (x( where [P((u)] 2 2 and D P(u) and e
R] = (x())) 0. Letting C [d] be some feature subset such that (P(o) and D P(r). Any
superset 2 respects (P(u) [P(v)] > 2 and D E P(u), thus 2 and æ E R = (u( = 0.
By minimality (cf. Corrolary B.2 the component fu, (x) 0s null
(P()] 2, D E P(u), x R fu.b(z) =0
(45)
Accordingly, the first summation term of Equation 44 is null
Σ
E 00) 0. xn
(46)
PP DEP(u) DEP(u)
By Lemma B.4
we also have that (P'(u) 2 implies (P(u) 2 2 and so
E
Σ LEC
m (PP((u), (1)))))))))))).()))5)
E [0]
(47)
PP(W)22
(2)22 DEP(u) or DEP(o)
As result, the left-most terms of Equations 46 and 47 are equal.
Now tackling the second term of Equation 44 By Equation 43 it holds that P(u) = (u)).. So for any æ we have
for any S [d] such that D P(u)
Σ
Σ w.Cld P'()222((()) and
w(P'(u), (())),,
M.CLId
[P(n)]22, P(2)222 DdP(U) and DeP(e)
(48)
Thus we have proven that -C
Property 3 When features are independent, the Marginal Decomposition falls back to the ANOVA decomposition (see Appendix A.2). In this case, the functional components become zero-mean, uncorrelated, and have variance oi Using Equation 16 and setting Bund D
(B Bind P)
((P(u),P(u)) of
(49)
"CId:P(M))22
and
(B Bind, P')
Σ
w(P'(u), P'())
" (u.)22
We define the sets
Sp fu S [d] IP(u) 2 2} and Sp fu C |d] IP (u)) 2).
By Lemma B.4 we have that Spr C Sp and so
(Bii Bund P) (Bins Bund P')
Σ
w(P(u),(uu)
Σ ((P((t),P'u))) 02
HESp
RESp
Σ
SS
Σ
(u(P()),P(u)) - P'u)))
2 0 (Since ((P(u), P(w)) > (w), P' (u) 0)
23

---

Published as a conference paper at ICLR 2026
The following Theorem highlights sufficient condition to define partition loss functions using explanation disagreements.
Theorem B.4 (Theorem 3.3 The L2 disagreements Da. (o o') between Occlusion and the
Arch/LIME/RIS//SIAA explainers respect Definition 3.7
Proof. We demonstrate that the L2 disagreements between the post-hoc explanation methods respect the premise Lemma B.3
D
- (u Su.s(2)) (us((()) )
Dr. x')= =
Σ
1=1
M(P())
D
Σ
E
a~D
1=1
a d HEP(a),(P(u))2 tEP()))P())2
E x~D
Σ
PWWNFF())PPP())))P0P)))) f..
is The corresponding interaction Pla P(v)) PU) Now, obviously w(P(u), P())) [P(u))b((Pu))) 2 0 but (u) P(u)) 2 (u), only holds for any superpartition P' we compare certain pairs of explainers Oce-Arch, Oce- SHAP, Occ-LIME, Occ-RISE.
While the Theorem highlights a sufficient condition for explanation disagreements to respect Def- inition 3.T one might wonder whether these conditions are also necessary. The following result partially answers that question by demonstrating that alternative measures of disagreements do not respect Definition 3.1
Theorem B.5 (Theorem 3.4). The L2 disagreements between explanation pairings that do not
include Occlusion (e.g. LIME SHAPP) break property 3 of Definition 3.1 Moreover, using the Pearson correlation disagreement Dy breaks property 2.
Proof. To prove the two statements of the Theorem, we simply need to come up with a single counterexample where the measures of disagreement break one of the properties of Definition 3.1
(Counterexample for L2 distance) Let f(x) IT T; be the model and let B = D U[-1 1)yd be distribution with independent features. We show that L2 disagreements that do not involve Occlusion break property 3. First, we have that fu,b(x) = 0 for any u * [d] and
f14,,()) I Zy. If we group the features into D groups, by Theorem 3.1
(f, P) (D)ff(...t
(50)
Recall that I(D) for ArchAttribut, I(D) 1/D for SHAP and I(D) 1/2 for LIME/RISE, Thus, we have
D
Dis (12(0,00) = =ND
Σ 1=1
- (N(D))
D(((D)) - -1(D))² 2nD
E.
[14),6(2)
D(K(D) - 2D))2
x
aND
3d
Since all variables are symmetric this toy example, explanation disagreements depend only on the number of feature groups D and not the groups themselves. Now, property 3 of Definition 3.1 immlies that this function should not increase as is reduced (i.e. as we fuse groups together). However, this is not true when comparing Arch-LIME, Arch-SHAP, and LIME-SHAP.
24

---

Published as a conference paper at ICLR 2026
(Counterexample for Dş) Let f(x) 11 + D2 + 2y24 be the model whose prediction at a compared to the one at = 0. When each feature considered as separate group, the post-hoc explanations take the form ff x, 2, P) [1, 1, (2),, h(2). Note that RISE/LIME and SHAP will yield the same explanation since the largest interaction is of order 2. By rephrasing Equation 9 as agreement metric (instead of disagreement), and setting D 6R and B 8_ the distributions,
we get
(6.0" | ||'1||
2 (2(2)
+ (22)
(51)
V2 + 2h(2)² /2 (2)2)
VI (2) / 22(2)
Now, since the model is additive in features 21 and 22, an ideal loss function should remain un- changed whenever these features are fused into single group. Otherwise, reducing the loss does not guarantee that we have found relevant interactions between the features. If we define a new partition P'' where the first two features are fused, we get the explanations Φ x. 2, P') = [2. (2),, The corresponding cosine agreements are
( Φ²) O ||
4 (22
2 (22))(
(52)
4 + (2) 4 (2
/2 + 122 /2 (22
For any pairing between Arch (h(2) = 0) Occlusion (h(2) = 1), and SHAPILIME/RISE (h(2) = 1/2)), Equation 52 is larger than Equation 51 As result, agreement can be increased between explanation methods without infering meaningful groups.
25

---

Published as a conference paper at ICLR 2026
C
AGREED ALGORITHM
The AGREED algorithm aims at minimizing the L2 disagreements between the joint- PDP/ArchAttriuute and Join--Fii/occluusoon explanations
1 LAGREED (D,B,P)
Σ (i) with
T
E
"0 (f, B,
(53)
Starting from granular partition {{1).... {d}}, greedily minimize Equation 53
1. Select the group with highest potential (i).
2. Compute its pairwise interaction with other groups j.
3. Fuse group i with the group j of maximal pairwise interaction.
4. Repeat until the disagreements fall below 6.
The implementation details of the algorithm depend whether = 'D not.
C.I CASES WHERR B = D
common to set = D on Tabular data so that æ and b can both be interpreted as a random sample from the dataset. This assumption introduces symmetry between and b that makes statistical estimates more efficient.
Step 1. Computing the Group Potential The building blocks of AGREED are the following D x x N tensors
Definition C.1. Let be partition of [d] into D disjoint groups, D be the data distribution and (x(0) N be N points sampled from it. We define the x N N tensor G
G. = f(ro (()))( x (1))) f()
(54)
These tensors are of interest because averaging them along their second and third axis leads to consistent estimate of the joint-PDP/ArchAttribu and joint-PF1/Ocellsioon respectively
N
Gi 3n (k) D.P)
(55)
N
G.ae 3 ( (z) 19), P).
(56)
Given G, efficiently compute the potential of group i
N
N
Ginee GieN)) Gien+GGen)) 2 (1).
(57)
2. Computing Interaction Between Groups At any point in the iterative algorithm, we will have access to the G tensor of the current partition. Thus, a good between- netween-group group interaction score should leverage this precomputed tensor to avoid unnecessary model inferences.
Definition C.2. Let P be a partition of [d] into D disjoint groups, E [D] be a group that want extend, DD be the data distribution, and (k) j be N points sampled from it. We define the (D - x N x N matrix I such that
== = (11)))(())))) (x)) - ffrff(rpp ((1 (= (0)))) (x))) 210) f((x)))
(58)
26

---

Published as a conference paper at ICLR 2026
Crucially, I can be computed efficiently by querying G j,K = f(rp *((65))(x)) x _x(())))
f(x!0) ) Gi.kee Gj,ee
(59)
Averaging this tensor along the third axis yields
Σ J.Ke
Σ
(a,
(60)
pure measure of interactions only involving features from groups i and j. Averaging the tensor along its second axis leads to
1 N
J.
3
E
fu,D(x(0),
(61)
full measure of interactions involving features from groups i. j. and possibly other groups. To compute the strength of the interaction between two groups and j. we could report the pure inter- action, the full interaction, weighted average. Like (Tsang al. 2020), we average the pure and full interactions with weights 0.5.
Fusing Groups After having identified two groups and j that interact, we define a superparti- tion P' of size D - 1 where j are fused into single group. The D 2 groups & * i,j are simply re-indexed from 1 to - 2, while and j are considered the (D 1)th group. Since the G tensor relative to the current partition, it must be updated to G' when performing group fusion. For the D 2 groups that were not fused, we copy their Gk...: values. For the 2 groups that were fused, we store the joint effect
GDI,, f(rp- f(rp-1((s))) xx )))
f(x(()
IIN. Gi + GyA.e
(62)
that is computed without additional model inference. Here the pseudocode for updating the parti- tion.
Algorithm Update Partition P by fusing i,j E [D] into a new group
1: procedure UPDATE_ PARTITION(P, G, i,
2: Initialize new partition map P
3: G' 2eros 1, N);
4: % Count Variable
5: c+1
6:
for E [D] such that k + j.j do
7:
Define P'(
VR E p- ((k));
8:
Ge... + GR...
9:
c+ c+++1;
10:
% Groups and j are fused
11:
Define P'(() ;= D 1, Vl E P- ({i,)));
12:
GD
1.. - I,::: + G... + Gzasi
13:
return P', G"
This partition update requires no model inference since all relevant computations are queried from the G and I tensors. Putting everything togheter, we end up with Algorithm 2 AGREED requires (d2N2) model inferences since line calls f d.2 times, then for each iteration of the while loop (of which there are at most d 1), I is computed which does not call more than dNN2 times.
C.2 CASES WHERE B + D
When computing saliency maps for Image Classification, it is common to use a single baseline B = & (typically image with information). Also, there is need to find partition P that works across all images æ from the dataset. Finding partition that works on a single image æ is a more realistic goal set D 8g.
27

---

Published as a conference paper at ICLR 2026
Algorithm 2 Adaptive Grouping to ReducE Explanation Disagreements (requires D B).
1:
procedure AGREED(F, (x(1) JK=1 k=1 DA
2:
% Initialization
3:
Initialize partition P such that P()
Vj E [d];
4:
5 d:
5:
Compute the X N
N tensor G (cf. Equation 54 Compute the potentials (i),, = 1, ..... from G (cf. Equation 57
6:
Obj + D 1 (i);
7:
8:
while Obj do
9:
Which group to extend i <- ar.. (1);
10:
I zeres 1, , N)
12:
% Find the fuse candidate
for j [D] such that j * do
14:
Compute Between- Group Interaction L, (cf. Equation 59
j a12.ma IN Σx [(+ Σx I,x,))² ( 2e )²ex))]
Fuse groups and j leading to a superpartition
17:
P. G <
i,,,
18:
D+D
19:
Compute the potentials (i), i == 1.2,... ... D from G (cf. Equation 57
Obj -E (i);
return P. G.
Since AGREED expects B DD introduce mixture distribution Q (6 + 66) and note that
CARRED (5z, 56 P) = 4 x L
AGREED
Q P)P.
(63)
Thus, we can apply AGREED on images by feeding Algorithm 2 with two samples (æ and b) from fictional data distribution Q. This will lead to D x 2 G tensor and D 1 x 2 2 I tensor.
28

---

Published as a conference paper at ICLR 2026
pepongt Maeponggh
0007
1..00
go2
130
0001
008
030
Rotaa
Gooepe
Droiee
(a)
(b)
(d)
Figure 7: Tabular Data: EBM fitted on default credit. (a) Explanation L2 Disagreement are reduced as AGREED groups more features. (b-c-d) Three inconsistent unfaithfulness metrics collectively converge to 0 as AGREED groups features.
8.38
0:03
2,30
0.00
Stader Potter Crnge
Nactord Festrey Ghange
Stxtee el Pariee Grorge
Stader Hatter Corge
(a)
(b)
(d)
Figure 8: Tabular Data: GBT fitted on default credit. (a) Explanation L2 Disagreement are reduced as AGREED groups more features. (b-c-d) Three inconsistent unfaithfulness metrics collectively converge to 0 as AGREED groups features.
2,000
502 uuleiakkm
1000
10:00
130
Gloege
(a)
(b)
(d)
Figure 9: Tabular Data: EBM fitted on SPAM. (a) Explanation L2 Disagreement are reduced as AGREED groups more features. (b-c-d) Three inconsistent unfaithfulness metries collectively con- verge to 0 as AGREED groups features.
D EXTENDED EXPERIMENTS
D.I TABULAR TOY DATA
Tabular synthetic datasets are generated by first sampling N samples of features ~ W/O, B) with a block-diagonal covariance matrix. The feature groups forming each block are chosen ran- domly. Then, another random feature partition P is generated to define a group-wise additive model. Each component (p-1((s)) of the model is generated from the following list
1. 9p ((())) (x) = I ((a) I (())) ((t) Tz
2. 9p- (1p11(())(a))) = exp 1000--
3. (p-1(()))(m) O(jE ((0)) w/a?) where is Sine, Cosine, Tanh, ReLUU.
D.2 TABULAR DATA
D.2.1 ADDITIONAL. QUANTITATIVE RESULTS
Figures I show the trade-offs between explanation
and feature group
sizes for EBM and GBT models fitted on Default-Credit, Spam, and NOMAO. The insights identical
29

---

Published as a conference paper at ICLR 2026
2 (0:03
1.200
1.000
502
130
(a)
(b)
(d)
Figure 10: Tabular Data: GBT fitted on Spam. (a) Explanation L2 Disagreement are reduced as AGREED groups more features. (b-c-d) Three inconsistent unfaithfulness metries collectively con- verge to 0 as AGREED groups features.
1200E
0080
30
1010E
6.02
0000
0003
20.00
0000
3.300
Cooage
Rittes Dirige
(a)
(b)
(d)
Figure 11: Tabular Data: GBT fitted on Nomao. (a) Explanation L2 Disagreement are reduced AGREED groups more features. (b-c-d) Three inconsistent unfaithfulness metrics collectively converge to 0 as AGREED groups features.
to those discussed in the main manuscript: as we group features together, disagreements between PDP/SHAP/PFI are reduced and the Sensitivity-1, INFD, SWF unfaithfulness metries also collec- tively converge to zero.
The positive effects of feature grouping most prominent EBMs compared GBTs. This is because EBM are restricted to only model interaction order 2, while GBTs with depth-T trees can model interaction whose order at-most T. Apparently, the GBTs trained on the largest datasets (Figures 10 & 11 have learned very high-order interactions that are extremely hard to minimize. Although AGREED fails to find agreement those two settings, the algorithm still useful to warn the user that the model might be too complicated to be explained with feature-based explanations. Hence, it might be best rely on EBM if faithful and unambiguous explanations are desired.
30

---

Published as a conference paper at ICLR 2026
Local Pocture Attritation
4-
2
pood
ap Iney FiE Ju
celluarr conturt
telephone
10
0.0
1.0
day
(a)
(b)
Figure 12: Marketing with no grouping. a) The global feature importance according to the PFI (opaque), SHAP (semi-transparent), and PDP (transparent) explainers. b) The PDPs of three dis- agreeing features (thick black line) along with the ICE (thin lines).
Contact
Contact-oelblat
Contact stelephheee
Month
2
Day
Day
Deg
(a)
(b)
Figure 13: Marketing with grouping. a) The global feature importance according the PFI (opaque), SHAP (semi-transparent), and PDP (transparent) explainers. b) The joint-PDP of the month: day: contact feature group.
D.2.2 QUALITATIVE RESULTS
The challenge in interpreting joint-importance scores is that, when features j. and e are treated group their joint-PDP of (h, æ. B, P) becomes multivariate function of æz. 2y and zr. Humans notoriously bad at visualizing high-dimensional functions, so advocate selecting 1-2 features to plot while conditioning the remaining ones. When features are binary or categorical, condition on their unique values. For numerical features, condition on their quantiles.
Marketing The Marketing dataset describes the marketing campaign of a Portuguese banking in- stitution. Each instance corresponds to a distinct phone call and the binary label encodes whether the client subscribed to term deposit. We explain EBM fitted this dataset using the PDP/SHAP/PFI explainers. From Figure 12 (a), the three techniques attribute very different global importances the features month, day, and contact. Figure 12 (b) shows the PDP of these three features along side their Individual Conditional Expectation (ICE) (Goldstein et al.) 2015). The ICE curves can be interpreted as the local trend when varying xy while the PDP is the average trend. It is apparent that the average trend is very different from local ones, especially at the value contact=?. These disagreements induced by strong feature interactions within the model.
To reduce the disagreements caused by feature interactions, we ran AGREED using = 1000 random samples for MC estimates and e 0.25 as the stopping criterion (it stops when the loss reduces to 25% of its original value). In 30 seconds, AGREED discovered the group month: day: contact. According to Figure [3 (a), there are now almost no disagree- ments between the global group importance reported by PDP/SHAP/PEL The joint-PDP of month: day: contact can visualized using a scatter plot along month : while condi- tioning different values of contact E [?, celllll, te elephone]. This is presented in Figure (b). (b). When contact=?, there is a significant drop in model output during June and July compared to other values of contact. Moreover, the trends along month and day hardly appear to be additive: there is a sharp drop in late January that does not occur in other months. Therefore, it is better interpret them jointly single "date" feature group
31

---

Published as a conference paper at ICLR 2026
Local Featuun Attrilattion
BAN
5g
Falee
200000 10000
Falee
0.2
0.4
0.6
Delay-Sep
Bill-Sop
Delag-Aug
(b)
Figure 14: Default-Credit with no grouping. a) The global feature importance according to the PFI (opaque), SHAP trasppraan,,,,,,,, and PDP (transparent) explainers. The PDPs of three important features (thick black line) along with the ICE (thin lines).
BilleSegeDiiee-- Pae
Linil
Local Group Attrilattion
Dill Jul
Delly-hil
Apc
DellyJunn
Daby
Dow May
Bill Ang
Por Ape
D03 Ape
Taby Ang wilkdy Soy
Marigg
Pac-Mhy
Gender
Edbcatin
Age
25000
50000 Bill-Sep
75000
0.00
0.25
0.50
(a)
(b)
Figure 15: Default-Credh with grouping. a) The global feature importance according the PFI (opaque), SHAP transparran,,,,,,,,,, and PDP (transparent) explainers. b) The joint-PDP of the Bi See feature group.
Default-Credit The Default-Credit dataset aims at predicting if clients of a Taiwanese bank will default on their credit. The data contains records of 30K individuals and 23 features related to past nayments/bills/delays and demographic characteristics. We explain an EBM fitted on this dataset using the PPP/SHAP/PFI explainers. Figure 14 (a) demonstrates that the three explainers provide different global importances the features Delay aaiaaySepp Sep, Delay Aug, and Bil1-Sey, Figure 14 (b) shows their PDP and ICE local attribution. By comparing the PDP and ICE, it clear that the PDP of Delay -ug is misleading since having delayed payment in August sometimes increases the model output and sometimes decreases it. This suggests that the effects Delay-Aug cannot be faithfully described using single feature importance score.
To faithfully explain the effects interacting features, we ran AGREED using N = 1000 random samples for MC estimates and when the loss reduces to 10% of its original value). After about a minute, AGREED yielded the group Delay Aug-Aug::0 Delg::eeleSSS:BBBii. Shep:Bill-Ses PDP/SHAP/PFI now agree on the global importance of each feature group. The joint-PDP of Delay-Aug: Delay-Sep: Bill-See and numerical one. This function can be visualized by plotting four line charts w.r.t B Bill-Sep (one line for each configuration of the remaining two binary variables). See Figure T3 (b) for the results. Interestingly, the impact of August delays depends on whether there was a September delay. Comparing the yellow curve to the green one, and the red/blue curves, the effect of De Aalay-Aug completely reversed depending
0.1 the stopping criterion (AGREED stops
According Figure 15 (a),
multivariate function involving two binary features
Delay-Sep Sep. The effect of Bill-Sa Sep also depends De Delay Sep. We are sure why these trends are happening in the data, but least AGREED warns us that trends involving Delay-Aug, :ell:::eelypppppp Bi 111 11-Sep are inherently high-dimensional and should be visualized accordingly.
32

---

Published as a conference paper at ICLR 2026
Arch
Occ
1G
RISE
LIME
SHAP
LUR00 03
Loone 1c06
Figure 16: Example of applying AGREED on the toy rectangle classification dataset. The plots show the image overlaid with the patch importance. Blue and red refer to negative and positive attributions, respectively. As AGREED converges (larger patches are found), the various explainability methods increase in agreement.
Arch
Occ
IG
RISE
LIME
SHAP
0
[]
[]
0
0
0
0
0
0
0
0
0
0
0
Figure 17: Another example of applying AGREED on the toy rectangle classification dataset. The plots show the image overlayed with the patch importance. Blue and red refer to negative and positive attributions respectivelly. As AGREED converges (larger patches are found), the various explainability methods increase in agreement.
D.3
IMAGE Toy DATA
17 show qualitative results for the toy image dataset consisting of predicting whether rectangles are tall (y 1) or wide (y = 0). Once each edge of the rectangle becomes its own group, all explainability methods agree on their importance.
Figures 16
33

---

Published as a conference paper at ICLR 2026
Image
Arch
Occ
IG
RISE
LIME
SHAP
1.00
==
L=
0.75
==
L=
Figure 18: Explaining the "House Finch" prediction of a ResNer18. AGREED yields partition with
increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
= 1.00
0.75
=
0.50
Figure 19: Explaining the "House Finch" prediction of ResNet18. AGREED yields partition with increased agreement between the various saliency map methods.
D.4 MINIIMAGENET
Figures 18 to 27 present the saliency maps resulting from the AGREED partitions. The model under study a ResNet18 pre-trained on ImageNet. We see that, in general, AGREED identifies a large patch of great importance that covers the animal. However, there are exceptions: in Figures 20 & 23 various patches cover specific parts of the animal.
34

---

Published as a conference paper at ICLR 2026
Image
Arch
Occ
IG
RISE
LIME
SHAP
L=
== 0.75
== 0.50
Figure 20: Explaining the "House Finch" prediction a ResNer18. AGREED yields partition with
increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
== 1.00
0.75
L=
L= 0.50
Figure 21:
Explaining the "Toucan" prediction of ResNer18. AGREED yields partition with increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
== 1.00
0.75
=
0.50
L= L=
Figure 22: Explaining the "Toucan" prediction of a ResNet18, AGREED yields partition with
increased agreement between the various saliency map methods.
35

---

Published as a conference paper at ICLR 2026
Image
Arch
Occ
IG
RISE
LIME
SHAP
1.00
==
0.75
L=
0.50
==
Figure 23:
Explaining the "Toucan" prediction of a ResNer18. AGREED yields partition with increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
1,00
0.75
L=
L=
L= 0.50
Figure 24: Explaining the "Malamute" prediction of ResNet18. AGREED yields a partition with
increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
== 1.00
1=0.3
0.50
L=
Figure 25: Explaining the "Malamute" prediction of ResNet18. AGREED yields a partition with
increased agreement between the various saliency map methods.
36

---

Published as a conference paper at ICLR 2026
Image
Arch
Occ
IG
RISE
LIME
SHAP
1.00
0.75
L=
0.50
L=
Figure 26: Explaining the "White Wolf" prediction of ResNet18. AGREED yields partition with
increased agreement between the various saliency map methods.
Image
Arch
Occ
IG
RISE
LIME
SHAP
Loss = 0.50 Loss 0.75 Loss 1.00
Figure 27:
Explaining the "Lion" prediction of ResNet18. AGREED yields partition with in- creased agreement between the various saliency map methods.
37


Reviews:
[{"review_id": "7r9OoQwezs", "reviewer": "Reviewer_EP8L", "summary": "This paper addresses the issue that many explanation methods often disagree with each other. They formally prove that when the model is group-wise additive, using the ground truth groupings, then unfaithfulness metrics will all reduce to 0. They then prove that if we minimize the difference between Arch and Occlusion, then this will minimize the disagreement between all groups. And they achieve it by iteratively merging groups with the highest pairwise interactions. They repeat until the disagreement falls below a threshold. They conduct experiments to show that when they merge the groups progressively, the difference between different explanation methods consistently decreases, and that each method is changing to become more agreed as the loss decreases, and that the proposed method AGREED can get much lower disagreement when merging not many groups than other methods, meaning that it is merging the groups that will reduce disagreement the most.", "questions": "1. You et al. 2025 also considers group-based attributions and categorizes insertion and deletion style errors (Definition 2.2 of Sum-of-Parts is similar to Equation (10) in this paper, and Definition A.1 is similar to Equation (9)), although they compute differences with respect to the prediction contributed by a group, rather than differences in attribution scores produced by two methods for that group. They show that when there are between-group interactions (e.g., polynomial correlations), these errors can grow exponentially (Theorem 2.3), while using the correct grouping drives the error to zero (Theorem 2.4). The theoretical results here (e.g., Theorem 3.2) appear closely related, as both hinge on the vanishing of between-group interaction terms. It would be helpful for the authors to explicitly discuss the similarities and differences with You et al. 2025, including the choice of disjoint partitions in this paper vs. allowing overlapping groups in You et al 2025.\n\n2. In Section 5.2.1 and Figure 7, only Arch and Occlusion are shown. Since the paper's claims concern reduced disagreement across a broad set of methods, could you clarify whether the qualitative improvements also generalize to other methods like LIME/SHAP etc.? Including additional visual examples in the appendix would strengthen the qualitative evidence.\n\n3. In functions with overlapping interaction structure (e.g., where a feature participates in multiple strongly interacting cliques), disjoint partitions may not be expressive enough. In such cases, the optimization may be forced to merge many features together, reducing interpretability. It would be helpful if the authors could comment on this tradeoff and whether and how overlapping or hierarchical groupings could be supported in future work.\n\n[1] You et al. \"Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups\" ICML 2025", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper proves a novel and important theorem that if we reduce the disagreement between Arch and Occlusion, it will reduce the disagreement for all methods.\n2. Their proposed method empirically finds better groups than other methods.\n3. The authors conducted comprehensive experiments to demonstrate their advantage in different aspects.", "weaknesses": "1. Some of the empirical figures (e.g., Figures 4 and 5) are not very intuitive\nin their current presentation, and the takeaways are not made explicit in the\ncaptions. Clarifying the intended interpretation in the captions or main text\nwould significantly improve readability for non-expert readers.\n\n2. One prior work also proves how one can achieve zero unfaithfulness when using the correct grouping structure. It would be helpful to cite this work to situate the current contribution within a more comprehensive body of related theory [1].\n\n3. The paper considers only non-overlapping partitions, while there could be cases where groups overlap with each other a lot and only considering non-overlapping groups could make it less expressive.\n\n[1] You et al. \"Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups\" ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Tsj9gANog", "reviewer": "Reviewer_MwSX", "summary": "Summary\nThis paper addresses the disagreement problem among post-hoc explanations in explainable AI by proposing an adaptive feature grouping method (AGREED) that reduces interactions between feature groups, aiming to bring explanation methods into agreement. The work builds on prior frameworks of functional decomposition and attribution methods, extending them to disjoint feature groupings. Empirical validation on tabular and image datasets shows improvements in explanation consistency and faithfulness metrics.\n\nSoundness\nThe methodology is mathematically sound and supported by formal proofs linking disagreement to between-group interactions. The algorithm is intuitive and empirically demonstrated to reduce disagreement. Results are robust on multiple datasets, though assumptions like groupwise additivity in regions and sampling baselines could affect results in practice.\n\nPresentation\nThe paper is well-organized but the presentation is somewhat standard and occasionally dense. Some of the core ideas could be clarified with more intuitive or visual illustrations, especially around interpretation of grouped attributions. The empirical section is comprehensive yet could better emphasize practical significance. Comparisons to prior grouping methods on images are limited by scalability and implementation constraints, leading to incomplete benchmarking in some cases.\n\nLiterature review\nLiterature review is a bit outdated, check these e.g. for attributions methods such as IG and SHAP with stronger mathematical content:\n\nA General Feature Attribution Framework under a Black-box Setting\n\n-Y. Cai, A. Thibaud, G.Wunder, International Conference on Machine Learning (ICML'25), Vancouver, Canada, 2025\n\n-On Gradient-like Explanation under a Black-box Setting: When Black-box Explanations Become as Good as White-box\nY. Cai, G. Wunder, International Conference on Machine Learning (ICML'24)\n\nContribution\n-Proposes a practical adaptive grouping algorithm to reduce disagreement in feature attributions\n-Unifies explanation methods under a functional decomposition framework for groups\n-Extends interpretability to handle feature group interactions explicitly\n-Demonstrates empirical gains on tabular and image data\n-Provides theoretical insights into when explanation agreements are possible", "questions": "-How does AGREED compare to prior grouping methods that are not feasible for images, when run on smaller image subsets?\n-How practically interpretable are grouped attributions for non-expert users, especially with large feature clusters?\n-Can the algorithm support global grouping for an entire dataset rather than per-sample in images?\n-Would integrating semantic/region-based features improve grouping and interpretability further?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-Sound theoretical grounding\n-Addresses a key challenge limiting trust in explainability\n-Novel focus on attributing disagreement to feature interaction", "weaknesses": "-Novelty incremental given prior functional decomposition frameworks; core insight of grouping to minimize interactions is expected\n-Presentation could benefit from more accessible examples and clearer motivation for practitioner\n-Limited direct comparisons with some existing grouping algorithms on images\n-Interpretation of grouped attributions remains difficult without actionable visualization\n-Image domain grouping requires per-sample runs, limiting real-world scalability", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtI7BLwanl", "reviewer": "Reviewer_5SdD", "summary": "The paper aims to address the problem of XAI disagreement. The aauthors extend Functional Decomposition and parition the input features/pixels into groups and prove that the primary reason of disagreement between different explanation methods is the between-group interactions.Further, they advocate eliminating between-group interactions and show that the disagreements can be reduced by adaptively grouping features/pixel.", "questions": "I request the authors to address the weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The paper deals with an important aspects of XAI: disagreement between explanation methods and the unreliability of fidelity metrics\n\n2) The theoretical sections are formally presented, with clear conditions, properties, and proofs.\n\n3) The proposed method was shown to work in tabular data as well as images.", "weaknesses": "1)  The authors use L2 disagreement between attribution vectors to define their partition loss. While this choice yields a differentiable and additive objective but it emphasizes only magnitude alignment and misses out on rank alignment. Rank-based measures such as Spearman, Kendall correlation etc could capture rank consistency between explanations but are non-differentiable for the groupwise optimization. A brief justification of this design choice would enable the readers to understand the paper clearly.\n\n2) The partition loss formulation is conceptually appealing but it requires computing multiple attribution maps per input, severely limiting scalability. This limitation should be discussed more explicitly as it constrains the applicability of the proposed framework to high-dimensional or real-time settings.\n\n3) The authors provide the algorithm details of AGREED in supplementary and also the computation cost of O(d^2N^2). The paper would benefit from a detailed analysis of this computation cost.\n\n4) AGREED seems to be prohibitively expensive as the cost O(d^2N^2) increases rapidly with number of features and samples. Even for 100 features and 1000 samples the number of model  inferences would be too high. Hence, there is limited scope of appying AGREED on high dimensional and real-world settings. I request the authors to clarify this limitation in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p800rDUXE6", "reviewer": "Reviewer_Dpg6", "summary": "The paper addresses disagreement problem in explainable ai, where different post-hoc explanation methods and metric often produce inconsistent results. The paper extend the framework of functional decomposition to analyze explanations over groups of features. It has the theoretical contribution to prove that the disagreement between various group-based explainers and high scores on common metrics share a common root cause. Therefore the paper proposes AGREED, an algorithm that adaptively finds feature partitions that minimize these interactions.", "questions": "Look at the weaknesses", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The paper has rigorous theoretical contribution.\n3. The paper clearly demonstrates that both disagreements between explanation method and (un)faithfulness as measured by standard metric are functions of the same underlying between-group interaction terms.\n4. The paper validated the theoretical claims in diverse modalities both synthetic and real world dataset.", "weaknesses": "1. The problem statement in the introduction is very broad. The described challenge is fundamental to almost all xai research.\n2. The readability of the figures are poor, also the main text needs to provide guidance on how to interpret the results in the figure (e.g., what the takeaway is).\n3. Need to cite all the explanation methods that are introduced in the paper (e.g., arch, occ).\n4. The paper's argument appears to equate 'faithfulness' with 'agreement/consistency' between explainers. While the theory links common (un)faithfulness metric to interaction-driven disagreement, the author should specify the definition of 'faithfulness'.\n5. The vision experiments rely on VGG16 and ResNet18, which are architectures from 2014 and 2015 which is outdated.\n6. The proposed algorithm does not optimize the true interaction loss, which captures all interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)