Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Authors: Weiqiu You
Helen Qu
Marco Gatti
Bhuvnesh Jain
Eric Wong
Venue: ICML 2025 poster (2025)
Keywords: explainability
interpretability
faithfulness
self-explaining models
feature attribution
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: We develop Sum-of-Parts (SOP), a framework that transforms any differentiable model into a self-attributing model neural network whose predictions can be attributed to groups of features.

Abstract: Self-attributing neural networks (SANNs) present a potential path towards interpretable models for high-dimensional problems, but often face significant trade-offs in performance. In this work, we formally prove a lower bound on errors of per-feature SANNs, whereas group-based SANNs can achieve zero error and thus high performance. Motivated by these insights, we propose Sum-of-Parts (SOP), a framework that transforms any differentiable model into a group-based SANN, where feature groups are learned end-to-end without group supervision. SOP achieves state-of-the-art performance for SANNs on vision and language tasks, and we validate that the groups are interpretable on a range of quantitative and semantic metrics. We further validate the utility of SOP explanations in model debugging and cosmological scientific discovery.

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Weiqiu You'
Helen Qu
Marco Gatti
Bhuvnesh Jain"
Eric Wong'
Abstract
GroupGen r (trained)
GroupSelect e (trained)
Self-attributing neural networks (SANNs) present potential path towards interpretable models for -isheniiennional problems, but often face sig- nificant trade-offs in performance. In this work, we formally prove lower bound on errors of per- feature SANNs, whereas group- based SANNs can achieve zero error and thus high performance. Mo- tivated by these insights, we propose Sum-of-Parts Parts (SOP), a framework that transforms any differen- tiable model into group-based SANN, where fea- ture groups are learned andtto-end without group supervision. SOP achieves state-of-the-ee perfor- mance for SANNs on vision and language tasks, and we validate that the groups are interpretable on a range of quantitative and semantic metrics. We further validate the utility of SOP explanations in model debugging and cosmological scientific discovery.
E/Or
Cross Attn
Self Attn
logils
hiddden
score
Aggregated Prediction
Modal Backbone (frozen)
Σ
Figure I. Sum-of-Parts (SOP) linearly aggregates outputs from multiple feature groups. This mainlains performance while en- suring interpretability. SOP first generates groups using a group generator I, predicts with pre-trained backbone h, and aggre- gates the group predictions with group selector
Self-explaining neural networks (SENNs) were proposed as way to create neural networks with guaranteed linear inter- pretations (Alvarez Melis & Jaakkola, 2018). Specifically, SENNs express predictions linear combinations of inter- pretable atoms scaled by learnable coefficients, natural generalization of the statistical interpretation of the classic linear model. These interpretable atoms represent semantic notions such as object segments in images or example image prototypes (Alvarez Melis & Jaakkola, 2018).
Introduction
Machine learning (ML) models are powerful at complex tasks, but also notoriously opaque. In high-stakes domains such as science (Li al., 2021; Zednik & Boelsen, 2022) and medicine (Reyes al., 2020; Tjoa & Guan, 2021), experts need explanations to trust the models' decisions. For instance, physicists use interpretable coefficients to validate an ML model's 's rediscovery of Kepler's first law (Li et al., 2021), while physicians require explanations to trust ML- driven diagnostic decisions (Klauschen et al., 2024).
A common class of self-explaining neural networks, which refer as Self-Attrbutting Neural Networks (SANNs), use embedded feature subsets as interpretable atoms (Bren- del & Bethge, 2019; Jain et al., 2020; Agarwal et al., 2021). SANNs have predictions faithfully decomposable as linear combinations feature subset contributions. On the other hand, post-hoc feature attributions fail to pass the sanity checks for faithfulness (Sundararajan al., 2017; Adebayo et al., 2018).
Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, USA DDpaarmeent of Physies and Astronomy, University of Pennsylvania, Philadelphia. PA, USA. Correspondence to: Weiqiu You <weiqiuy @ww.s..peee..duuu.... Eric Wong <ex- wong www.saaac.......u......
However, SANNs often exhibit performance trade-offs and rely on specific components such as per-feature modules in NAM (Agarwal et al., 2021), convolutional layers Bag- Net (Brendel Bethge, 2019), or attention mechanisms in FRESH (Jain et al., 2020). This architecture require-
Proceedings of the 4rnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Ouu code is available Braghiolab/see scp
com/

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
ment hinders SANN from leveraging pre-trained models that achieve high performance on target tasks.
2 Overcoming Self-Attributing Neural Networks' Limitations with Groups
To understand these limitations, we first amalyze a theoreti- cal barrier that limits SANN's performance. Specifically, we prove a lower bound on the error of a class of self-attributing neural networks that we refer to as per-feature SANNs. This result shows that it impossible for per-feature SANNs to achieve high performance when features are highly corre- lated, critical limitation in high-dimensional vision and lan- guage data. In contrast, we further prove that group-based SANNs can achieve high performance in these settings, but require a careful selection of feature groups.
In this section, we first review self-attributing neural net- works (Section 2.1), and prove that the poor performance of previously explored per-feature SANNs is theoretically limited due to correlated features in high dimensional data (Section 2.2). In contrast, we further prove that group-based SANNs can overcome these fundamental limitations, moti- vating our proposed framework for learnable group-based SANNs (Section 2.3).
2.1 Self-Attributing Neural Networks
To overcome these provable limitations for per-feature SANNs and inflexible group-based SANNs, we propose Sum-of-Parts (SOP), flexible framework that transforms any differentiable model into a group-basee SANN (Fig- ure 1). Specifically, given backbone model and an input, the framework (I) identifies feature groups with a learned attention module, (2) encodes each group using a model- agnostic backbone, and (3) aggregates predictions with a second sparse attention module. This framework can then be learned end-to-end with only the final prediction labels, notably without the direct supervision of feature groups. Learned feature groups can capture the dynamic correla- tions in high- -ibbenmienii data, enabling SOP to overcome the theoretical limitations of per-feature and fixed-group- based SANNs.
Self explaining neural networks model predictions as a lin- car combination f(x) E, (()),,(z); where ((x) are linear coefficients and (ar) are referred to as interpretable atoms (Alvarez Melis & Jaakkola, 2018). A common strat- egy for creating self-explaining neural networks is to use feature subsets as interpretable atoms. For example, Bag- Net (Brendel & Bethge, 2019) or Neural Additive Mod- els (Agarwal al., 2021) decompose predictions into a linear combination of terms, where each term is directly computed from and attributed to subset of input features. We denote such models, which combine the interpretability of linearity with guaranteed attributions to input features, as Self-Attributing Neural Networks (SANNs).
Definition 2.1. self-atribbtting neural network ʃ R + E given input x e IR decomposes predictions as follows:
Our main contributions are as follows:
1. We propose Sum-of-Parts (SOP), a model-agnostic framework which transforms any model into group- based SANN. The groups in SOP are end-to-end leam- able without the need for group label supervision.
m f(x)= = Σ ((x).((GG.)
(I)
where 0(x) E R" are linear coefficients, and h(xG.) E R. are embeddings of the feature subset TG, corresponding to the subset G; S [d]. Note that m can be different from the number of raw features d.
2.
We formally prove that groups are essential for SANNs to achieve low errors for highly correlated features. In contrast, we prove lower bound on per-feature SANNs' errors, which grows as the number of features increases.
The resulting linear combination constitutes a faithful- by-construction explanation for the decision process of the model (Lyu et al., 2024). SANNs are only as inter- pretable as the underlying feature subsets (Alvarez Melis & Jaakkola, 2018; Zytek et al., 2022), and different SANNS have explored various feature subsets TG,-- For example, NAM (Agarwal al., 2021) uses individual features, Bag- Net (Brendel & Bethge, 2019) relies on large patches, and FRESH (Jain et al., 2020) selects single subset using at- tention scores. However, across these subsets. SANNS have consistently exhibit significant trade-offs in performance in exchange for interpretability. In this section, we theoreti- cally analyze the underlying cause for this trade-off and how SANNs can overcome these barriers.
3. We show that SOP achieves state-of-the-at art perfor- mance among SANNs on vision and language tasks as informed by the theory, with learned interpretable groups validated on range of quantitative and seman- tic metrics.
4. We validate the utility of SOP in debugging if cor- rect/incorrect model predictions rely more on the back- ground/objects, as well as a scientific discovery setting within cosmology by using the groups and scores to uncover new insights about galaxy formation.
2

---

Sum-of-Parte Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Min Total InsErr
60
Min Total DelErr
6000
40
4000
20
2000
5
10
15
10
15
Size of Polynomial d
Size of Monomial d
(a) Minimum total insertion error for binomials.
(b) Minimum total deletion error for monomials.
Figure 2. Errors for per-feature SANNs grow fast unavoidably. The minimum (a) total insertion error of monomials of size d and (b) total deletion errors of binomials of size the minima all possible per-feature self-explaining models. The dots are the lower bounds computed by the solver, while the line is hest-fit exponential function.
2.2 Lower Bounds on the Error of Per-feature SANNs
For brevity of presentation, in this section we focus on insertion, and present analogous definitions and theorems for deletion to Appendix A. We note that the insertion and deletion procedures are analogous to insertion and deletion tests for post-hoe explanations (Petsiuk et al., 2018; Samek et al., 2017), but used here to capture the error of SANN.
One class of SANNs uses individual features as interpretable atoms (Agarwal et al., 2021), where each feature sub- set G; = {i} and the corresponding interpretable atom h(a..) = hxx;) is the encoding of exactly one feature Z4- We begin by analyzing the error of per-feature SANNs.
Error Lower Bounds for Data with Correlated Features. We now prove that it is impossible for per-feature SANNs to perform well when the data contains correlated features. Specifically, in Theorem 2.3, we show that when estimating polynomials function with correlated features, per-feature SANNs have non-trivial lower bound on the total insertion error.
In order for such models to be accurate, the contribution of a feature to the true label should match the change in prediction when the feature is removed or added. Similarly, the contribution of a subset, Eg.cs a; should capture the change in prediction when the subset is excluded or in- cluded. We formalize this difference between a SANN prediction and the ground truth when inserting or deleting subsets of features insertion and deletion errors, respec- tively.
Theorem 2.3 (Lower Bound Insertion Error for Bino- mials). Lei p {0, 1] 1,21 be multilinear bi- nomial polynomial function. Furthermore suppose that the features can be partitioned into (S1, S2, S2,Sa, Sa) of equal sizes where p(z) = -IIES, It + IIJESJus, Iz.- IJ. Then, 2 (ino(A), where Dine(A) () X2)T To)ceh i the lower bound, À a dual feasible point, and is constant as defined in (16).
Definition 2.2. (Insertion Error) Let o; = 0(2): be the total contribution of the ith feature group to the predic- tion of SANN. Then, the insertion error self attribut- ing neural network fz) Ei ((x),((xG;) Σ, a for target function /" Rd IR when inserting subset
Esçia InsErr
of features S to input is
InsErr(G.a,5) a, (xs) -f
Σ a G,SS
To derive this lower bound, formulated the minimum total error of any SANN a linear program, and used dual feasible point to compute a lower bound. The proof and an analogous theorem for deletion error on monomials (Theorem A.2) are presented in Appendix A.
2z if jjeS otherwise
where (xs);
The total insertion error over all possible insertions is Esçia InsFrr(G,,,,S),
Lower Bounds Grow Rapidly with Dimension. In Fig- ure 2, calculate the lower bound using ECOS (Domahidi et al., 2013) and plot the lower bound for total insertion and deletion errors as the feature dimension grows (Figure 2), and observe that these errors increase exponentially with d. Altogether, these theoretical lower bounds and empirical trends suggest that per-feature SANNs are fundamentally in-
The insertion error captures the difference between the ground truth effect of inserting subset of features S and the corresponding change in the SANN. If these two quantities are close, then the error is low. The insertion error of a per- feature SANN a special case where G {()...., {d}).
3

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Algorithm 1 The Sum-of-Parts Framework
Require: Backbone Pred h, Backbone Encoder hns Embedding Model he, Class Weights Cy
Require: GroupGen r, GroupSelect 0, Number of Groups m, Input
(g1..... (..) < [(x) + o0 (he(a))
{Group Generating (trained))
for i 1 m do
Vi <- h(g o x)) end for
(Predicting with frozen Backbone)
+ (1 ① (r))...,,,(.... Ox)) (cI,...,Cm) Cm) f x)) + SparseCrossAttn (Cn, 2)
[GroupSelect (trained))
1i=100 Citii
capable of modeling high-dimensional data, as they cannot model even simple polynomials.
offs. For example, SANNs that rely rigid patterns (Bren- del & Bethge, 2019), use groups that are too small (Agarwal et al., 2021) or use too few groups (Jain et al., 2020) are un- likely to align with the ground truth correlations. In contrast, a high-performing SANN requires the ability to use many groups of flexible patterns to capture the diverse signals in high- gigh-dimensional data. These criteria motivate a new type of SANN that can overcome this theoretical performance barrier: the Sum-of-Pant framework.
2.3 Can Overcome the Performance Barrier
Group-based
Neural Networks
The fundamental limitation of per-feature SANNs comes from its choice of interpretable atom: individual features are unable to capture correlations between multiple features. If we allow SANNs to use more expressive interpretable atoms composed of feature groups, can we get past this limitation? To answer this question, we carry out an analogous analysis for more general, group-based SANNs.
3 The Sum-of-Parts Framework
In this section, we introduce our main technical contribution Sum-of-Pare -frrtst (SOP), a framework that transforms any dif- ferentiable model into group-based self-attrbbutngg model.
In this section, we summarize our main thcoretical result in Theorem 2.4: we prove that there exist group-based SANNs that can not only perfectly capture the carlier settings in Theorem 2.3 (and Theorem A.2 in the appendix), but also far more complex, general polynomials with zero error. In other words, the right groups can enable SANNs capture correlated signals and overcome the performance barrier.
Suppose we have input x E R and backbone model h Rd - R that makes prediction with the input, and we hope to convert h to SANN. A SANN requires components to generate and encode feature subsets and another component to assign them coefficients.
Theorem 2.4 (Informal: 7ero Group Insertion and Dele- tion Error). For any general m-nomial polynomial hav- ing at most m groups is sufficient for a group-based self- attributing neural network to achieve zero insertion and deletion error. See Theorem A.5 for full theorem and proof.
SOP therefore naturally consists of three parts: a group generator T : Rd {0, 1]mxd that generates groups 91..... 9m E 13 backbone predictor h : Ed + E that makes prediction with each group of features, and a group selector 0 : {0, 1 pmdd x Idd + [0, 11m that assigns scores to the groups:
Intuitively, SANN can achieve low error if its groups align with the correlations in the data. Specifically, con- sider data generated from polynomial with multiple terms p(x) = q1 (xaz) ) +... Im (fTG. ), where each qi (TG;) ) is a multiplicative term that depends on the group of features in G¹ Then, a group-based SANN can achieve low error if each correlated feature group G², aligns with SANN group G;. On the other hand, misaligned or insufficiently many groups lead to nonzero errors, highlighting how group- alignment is key to SANN performance.
m f(x) Σ O(T()),)) Mg Ox);
where gi E (())
group selector
backbone predictor
group generator (trained)
(trained)
(fozen)
Here we consider single predicted logit, while the process can be repeated in batch for multiple logits classes. Our a algorithm is illustrated in Figure and Algorithm 1.
While this theorem demonstrates that SANNs can model highly complex polynomials, it also provides insight into why existing SANNs have suffered major performance trade-
Group Generator T Rd > {0, 1pmxxd takes in an input x E Rdd and outputs m binary group masks gl Im.

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
{0, 134, where Øiy if and only if the feature j is included in group gi.- 2 It uses multi-headed self-attention mod- ule (Vaswani et al., 2017) to assign scores to features, and threshold each attention distribution to include top r = 20% features to each group.
In summary, the learnable group generator dynamically cre- ates correlated groups needed for high-performing SANNs, the model-agnostic framework supports arbitrary backbone predictors, and the sparse group selector assigns contribu- tions for a small number of groups. Together, these compo- nents are essential for SOP to be a high- perfooren SANN, as informed by theory in Section 2.
r(x)= = (g...... gm) = SoftSelfAttu4-00% (h)(
where the encoder he _d + Rdxe, which we typically take up to the penultimate layer of the backbone model, embeds each feature H; into a vector with embedding di- mension e. The learnable group generator dynamically creates feature groups for each input, enabling better corre- lation compared to fixed groups (e.g., patches). Moreover, it has no specific architectural constraints on the backbone such as attention mechanisms.
4 Empirical Evaluations of Sum-of-Parts
We conduct experiments using our framework on image and text tasks to see if our theory-informed framework actually uses the learned groups for (RQ1) high performance. Next, we quantitatively measure the interpretability of SOP with (RQ2) the performance at different inference-time spar- sity levels, (RQ3) various faithfulness metries, and (RQ4) whether the group masks are leaking predictive signals.
Backbone Predictor h : Rd + R then makes a predic- tion with the input x masked by each binary group mask g
Then, we measure the interpretability of SOP semantically using (RQ5) semantic coherence of the groups Finally, we validate SOP's utility on (RQ6) model debugging for unwanted behaviors in correct and incorrect predictions and (RQ7) scientific discovery in cosmology.
Vi hgi Or), == m
where y/ E IR is the output logit and @ is Hadamard product. The backbone predictor can be arbitrary pre-trained model.
Experiment Setups. We evaluate on two vision and one language datasets: ImageNet-S (Gao et al., 2022) image classification using Vision Transformer (Dosovitskiy et al., 2021) backbone, CosmoGrid (Kacprzak et al., 2023) cos- mology image regression using CNN (Matilla et al.. 2020), and MultiRC (Khashabi et al., 2018) text classification us- ing BERT (Devlin et al., 2019). We use patches as im- age features and tokens as text features. For baselines, we compare with other SANNS including XDNN (Hesse et al., 2021), BagNet (Brendel & Bethge, 2019), and FRESH (Jain et al., 2020). Additionally, we convert post-hoc feature at- tribution methods to self-attributing models by using top 20% scored features single groups and denote with "-F" (e.g. "LIME-F"), including LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), IG (Sundararajan et al., 2017), GC (Selvaraju et al., 2016), FG (Srinivas & Fleuret,
Group Selector 0 {0, l]mxdd x Rd + [0, 1jm then takes in the encoding of each group and uses a sparse cross- attention module to assign each group a score.
((T()), = (cI, (c1,...,,om = SparseCrossAtrr (Ci, 2)
where the query C%. E I is initialized using the tar- get class's weights with & hidden dimensions and the key z = (()(g))))) .... hn (9m Ox)) E Rm** are last hidden states of all groups. As using sparse number of groups avoids overloading human users, we replace the softmax in the cross attention with a sparse variant, the sparsemax op- erator (Martins& & Astudillo, 2016). Dynamicully assigning scores allows the model to choose the most helpful groups for prediction, while the sparse number of groups ensures easy human interpretability.
2019), RISE (Petsiuk et al., 2018), Archipelago (Tsang et al.,
2020), MFABA (Zhu et al., 2024), AGI (Pan et al., 2021),
The final prediction is made by aggregating predictions from each group ys with its assigned score c,.
AMPE (Zhu al., 2023) and BCos (Böhle et al., 2022). We use a maximum of 20 groups for SOP and the same 20 for- ward passes for all controllable baselines (Table 2). Details for datasets, models and baselines are in Appendix C.
f(x) y c191 + .....t Cm3ym
To address the gradient flow issue caused by binarized groups, we incorporate scaling factor based on the at- tention score in the final loss, as detailed in Appendix B.3. Additional details self-attention, cross-attention, and em- bedding models are provided in Appendix B.
4.1 Performance
(RQ1) How Well Does SOP Perform? As the theory suggests, SANNs can only perform well with groups of features that align with underlying correlations. We evalu- ate whether the learned groups allow SOP to achieve lower errors comparing to previous SANNs which rely on fixed or limited feature groups. Table 1 shows that SOP achieves
We use binary groups to avoid leaking information resulting unfaithful explanations.
5

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-Enn Learning of Feature Groups
Model- Agnostic
ImageNet-S-.
VIT 1OU1
CosmoGrid CNN
MulGRC
BERT
Category
Method
Err.
MSE.
Pur.
Em.
1OU1
Backbone
0.097 +: 0.011
0.009 0.001
0.318 + 0.021
LIME-F
Yes
0.327 +: 0.014
0.360 + 0.012
0.049 0.003
0.375
0.018
0.475 + 0.031
0.177 = 0.012
SHAP-F
Yes
0.306 + 0.027
0.391 1:39=:000 0.011
0.028 + 0.002
0.397
0.016
0,455 = 0.032
0.135 = 0.020
IG-F
Yes
0.581 = 0.013
0.324 + 0.003
0.042 + 0.003
0.391 0.011
0.485 = 0.027
0.119 = 0.008
GC-F
Yes
0.455 + 0.016
0.398 +: 0.015
0.036 +: 0.002
0.438 + 0.019
0.485 + 0.015
0.099 + 0.001
FG-F
Yes
+ 0.024
0.511
0.016
0.396 0.011
0.107 0.005
Post-hoc-
RISE-F
Yes
0.732 + 0.009
0.131 0.009
0.036 0.003
0.342 0.006
0.366 = 0.025
0.150 = 0.018
Converted
Archi-F
Yes
0.526 +: 0.016
0.290 + 0.010
0.069 0.002
0.487 + 0.004
0.515 1 0.011
0.098 + 0.002
MFABA-F
Yes
0.493 +: 0.016
0.383 +: 0.010
0.035
+
0.498 +: 0.014
0.466 2 0.023
0.113 + 0.006
AGL-F
Yes
0.407 +: 0.011
0.499 1: 0.012
0.040
0.002
0.522 0.010
0,446 0.019
0.147 + 0.012
AMPE-F
Yes
0.484 + 0.016
0.417 0.012
0.002 0.037
0.366 0.037
0.475 + 0.028
0.116 0.011
BCos-F
No
0.954 + 0.006
0.234 + 0.003
XDNN
No
0.871 + 0.007
0.332 + 0.004
Self-Atributing
BagNer
No
0.501 + 0.011
0.314 0.016
FRESH
No
0.537 + 0.020
0.464 + 0.015
0.386 + 0.039
0.176 + 0.016
SOP (ours)
Yes
0.267 +: 0.017
0.630 0.006
0.025
0.002 0.647
0.011
0.366 0.021
0.176 0.008
Table (Main Results: Error vs. Purity) This table presents error rate/MSE and IOU/purity metrics results comparing eepla-xpl models on ImageNet, CosmoGrid and MultiRC. We find that SOP achieves state-of of art performance comparing with all 14 baselines. The best result for each metric is bolded, and the second-best is underlined. For non-model-agnostic baselines, we only include for ImageNet-S where specialized pretrained models readily exist. Details of the metrics are explained in Appendix C.
Computation Cost
Methods
1x forward pass
IG-F, GC-F, FG-F, BCos-F, XDNN, BagNet, FRESH LIME-F, SHAP-F, RISE-F, MFABA-F, AGI-F. AMPE-F, SOP
20x forward passes
(ld)) forward passes
Archipelago-F (due to pairwise interaction testing)
Table 2. Computation cost of different attribution methods in terms of number of forward passes. SOP uses 20x forward passes, and we the number of forward passes for perturbation-bast based baselines that compare with.
the lowest errors and MSE for all tasks. SHAP-F is the second best on vision tasks but lags on MultiRC. No other SANN consistently performs well on all tasks, demonstrat- ing that SOP's learnable groups do enable state-of-the-art performance across diverse settings.
ImageNet VisionTransformer
0.8
Error
0.6
0.4
4.2
Quantitative Measures for Interpretability
02
(RQ2) Can SOP Perform Well at Different Sparsity Lev- els? Sparser explanations are easier to understand for hu- mans (Lombrozo, 2007; Poursabzi-Sangdeh et al., 2021), but the best sparsity level is often unknown at training time. SOP's group generator learns to generate groups at a spe- eific sparsity, and we test whether it performs well across other sparsity levels without retraining. Figure 3 shows that as sparsity increases (2 80%, keeping- < 20% fea- tures per group), SOP's error grows more slowly than other SANNs, maintaining much lower errors at extreme sparsity. Similar trends are observed in CosmoGrid and MultiRC (Appendix C.4.2). For MultiRC, FRESH slightly outper- forms SOP for untrained sparsity levels, as it is optimized for language tasks. Overall, SOP trained on one sparsity also performs well on other sparsity levels at inference time.
0.0
Q2
0.4
0,6
0.8
Sparsity
LIME-F - GC-F -.. Archi-F
AMPE-F -.. BagNet
SHAP-F - FG-F
MFABA-F
BCos-F
FRESH
IG-F
RISE-F
AGL-F
XDNN
SOP
Figure 3. (ImageNet Sparsity vs. Error L) We report how error increases when sparsity increases (fewer (npwerinput features included in each group). where SOP's slowest increase is the most desired.
Beyond group size, we examine how the total number of groups affects performance. Ablation studies on ImageNet- S using 1, 2, 5, 10, 20 groups show that performance im- proves with more groups but saturates at 5 groups. Us- ing only groups increases the error rate by percentage points compared groups, suggesting SOP can achieve
'Requires specialized architectures and thus only included for ImageNet-S where pre-trained models exist.
*Requires Transformer backbone and not applicable to CNN.
6

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
approximately 4x greater computational efficiency while maintaining comparable performance.
ImageNet
Category
Method
Ins.1
Del.j
LIME-F
0.859 = 0.005
0.476 + 0.004
SHAP-F
0.878 = 0.007
0.421 + 0.008
(RQ3) How Faithful Is SOP with Respect to Classic Met- rics? While self-attributing neural networks ensure faith- fulness by construction through linear aggregation, we check how well SOP performs on classic faithfulness metries: fi- delity (Yu& & Varshney, 2017; Chen et al., 2019), insertion, and deletion (Petsiuk et al., 2018; Samek et al., 2017). Fi- delity measures how well summed explanations match the model's prediction using KL-divergence, while insertion and deletion evaluate the impact of high-scoring features via perturbations and area-under-curve (AUC) computation. Definitions and results are in Appendix C.4.3, C.4.4.
IG-F
0.661 + 0.006
0.664 + 0.008
GC-F
0.817 + 0.007
0.416 +: 0.007
FG-F
0.805 + 0.006
0.430 1: 0.004
Dost-Hoc-Converte
RISE-F
0.635 + 0.007
0.708 1 0.003
Archi.FF
0.719 = 0.004
0.548 + 0.004
MFABA-F
0.720 + 0.005
0.547 + 0.010
AGI-F
0.781 + 0.007
0.509 + 0.007
AMPE-F
0.723 + 0.006
0.581 + 0.005
BCos-I
0.308 1 0.005
0.339 1 0.009
XDNN
0.251 + 0.007
0,210 + 0.003
BagNet
0.626 = 0.014
0.595 + 0.009
Self-Attributing
FRESH
0.759 + 0.003
0.417 + 0.004
SOP
0.930 + 0.003
0.109 + 0.000
Table 3. (ImageNet Insertion/Delction)) We evaluate inser- tion/delction metries on ImageNet, and find that SOP achieves best insertion and deletion scores. This table reports percent insertion and deletion scores for ImageNet with interval 10%. The best result for each metric is bolded, and the second-bess is italicized.
All SANNs, including SOP, achieve a perfect fidelity score of O. while no post-hoc method does (Table A5), indicating only SANNs faithfully match predictions. SOP outperforms all baselines on insertion across all tasks, and on deletion for ImageNet, while methods like Archipelago, LIME, and FRESH perform better on deletion in some cases (Tables 3 and A6). In fact, as deletion score measures how fast the predicted probability drops when the most scored features are deleted, it biases towards models that use a small number of features, regardless of how faithful the model is. SOP which relies on signals from multiple groups then naturally performs better on insertion than deletion. Ablations using smaller step sizes and occlusion values (Table A7) show consistent and thus robust results. In summary, SOP per- forms strong on most classic faithfulness metrics in addition to its built-in faithfulness.
ImageNet- Vision mee
0.15
Group Pred Ace
0.10
0.05
0.00
LIME-F SHAP.F
IG-F
GCFF
FG-F
MFABAF
AGI-F
AMPE-F
BCos-F
XDNN
BagNet
FRESH
SOP
Figure 4. (ImageNet Group Probing Accuracy) The powerful group generator in SOP not doing all the work and not com- promising SOP's interpretability. A CNN model trained on group masks from SOP is unable to obtain accuracies more than random (0. (0.1% accuracy), while MFABA-F, AMPE-F, IG-F etc. do. RISE and Archipelago is omitted for the significant computational cost. Results for linear and VIT probing models are in Appendix C.4.5.
(RQ4) Do SOP's Attributions Contain Predictive Sig- nals? The SANN's claim of linear interpretability relies on the assumption that feature groups (which form the in- terpretable atoms) do not inherently encode label informa- tion. If these groups already contain predictive signals, the backbone model-not the coefficients-would drive predic- tions, rendering the coefficients uninformative. To validate this, we train probing models to predict labels solely from group masks: high accuracy indicates that label information is pre-encoded in the groups. Figure shows that group masks from SOP achieves random accuracy (0.10%) on ImageNet-S using a CNN probing model. In comparison, probing models for groups from other SANNs achieve sig- nificantly higher accuracies, such as FG-F (13.40%) and AGI-F (1)..66).... Thus SOP's generated groups do not leak information about the label compared to other SANNs, and the model's interpretablity is not weakened.
cepts. We thus ask how semantically coherent are the group explanations generated by SOP compared to other mod- els? To evaluate this, we compute interseettion-over--iiin (IOU) for ImageNet-S and MultiRC where ground-truth annotations exist, and threshold based purity for Cosmo- Grid using expert-informed metrics for cosmological struc- tures (Matilla et al., 2020). Table shows that SOP has the best semantic coherency on two vision tasks and a close second on the language task. Figure illustrates an example of an image displaying a beagle, where SOP learns more semantically coherent groups than other methods without di- rect supervision. The exact formulations for IOU and purity, threshold ablations for CosmoGrid, and additional examples can be found in Appendix C.4.1. Additional experiments for human evaluation can be found in Appendix C.4.6.
4.3 Semantic Measures for Interpretability
(RQ5) How Semantically Coherent are SOP Groups? Explanations need to be semantically coherent such as re- lating to object segments or human- unnertaaaddet con-
7

---

Sum-of-Parts, Parts: Self-Attributing Neural Networks with End-to- Learning of Feature Groups
Image: Beugle
(b) LIME-F (Ribeim al. 2016)
(c) SHAPF (Lundherg & Lee, 2017)
(d) IG-F (Sumdararajan et 2017)
(e) GC+F (Selvaraju et al., 2016)
(l) FG-F (Srinivas Fleuret. 2019)
(g) RISE-F (Petsiuk et al., 2018)
(h) Archi
(Tsing al., 2020)
ii MFABA-F (7hu et al. 2024)
ii AGL-F (Pan et al., 2021)
(k) AMPE-F (Zhu et al., 2023)
() BCosFF (Bohle et al., 2022)
(m) XDNN (Hesse et al., 2021)
(n) BagNet
(o) FRESH (Jain et al., 2020)
(p) SOP (Ours)
(Brendel & Bethge, 2019)
Figure 5. We show example groups from different SANNs for Beagle in ImageNet, and find that SOP learns to generate groups more semantically coherent than other SANNs. "-F" indicates sellfattributing models converted from post-hoc methods. The highlights show the groups selected by each method for ImageNet, with unused patches hatched-out. Each group has 20% features.
4.4 Utility of SOP Explanations
ImageNet-S - Vision Transformer
(RQ6) Can We Use SOP Explanations to Debug Models?
0.7
Correct
Incorrect
% Object Pixels
As SOP explanations directly compose the final prediction, we want to see if we can use the group explanations to identify undesirable model behaviors. One such behavior is overly relying on background features for correct predic- tions, indicating potential spurious correlations. We analyze SOP's explanations check which feature groups the model make correct and incorrect predictions. We find that SOP uses more objects in correct (64.9%) than incorrect (57.3%) examples. Figure 6 shows similar behavior for other SANN baselines. We conjecture that using objects, instead of relying on spurious background correlations, helps the model make correct predictions. Thus, explanations from SOP and other SANNs can help illuminate the reasoning behind model behaviors.
0.5
0.4
0.3
0.2
0.0
LIME-F SHAP-F
IG-F
CCCF
FG-F
RISE-F
Archi- FF
MPABA. F
AGI-F
AMPE-F
Cra-F
NDNN
BugNet
FRESH
SOP
Figure 6. (ImageNet-S Percent Groups that are Objects) We analyze the proportion of groups that are objects on ImageNet-S. and find that groups consistently consist more ohjects when predicting correctly. This trend is consistently across methods.
galaxies (Gatti et al., 2021). While the direct relation from weak lensing maps to Ωun and 0g is unknown, cosmologists create simulated weak lensing maps from different Ω, and og values and train CNNs (Ribli et al., 2019; Matilla et al.,
(RQ7) Can SOP Explanations Assist Scientific Discov- ery? The ultimate goal of interpretability methods is for domain experts to use these tools and explanations in real settings. To validate the usability of our approach, we col- laborate with cosmologists and use SOP discover new knowledge about the expansion of the universe and the growth of the cosmic structure.
2020; Fluri et al., 2022) to reversely predict Qy and ag from the mass maps. An open question in cosmology remains:
What structures from weak lensing maps drive the inference of the cosmological parameters Qm and og ?
Problem Formulation. Cosmologists hope to understand the relations of cosmological structures with two cosmolog- ical parameters related to the initial state of the universe: Om and ds. The parameter 22n, captures the average energy density of all matter in the universe, while dg describes the fluctuation of matter distribution (Abbott et al., 2022). How- ever, these parameters are not directly measurable. What we can obtain are weak lensing mass maps, which are spa- tial distribution of matter density in the universe calculated using precise measurements of the shapes of 100 million
As validated in Section 4.3 (RQ5), the groups from SOP cor- respond to more cosmological structures- voids and clusters- than other self-attributing models. Voids are large regions under-dense relative to the mean density (pixel intensity < Oar) and appear as dark regions in the mass maps, whereas clusters are areas of concentrated high density (pixel inten- sity > 3a) and appear as bright dots, as shown in Figure 7.
8

---

Sum-of-Parts: Self-Attriuttng Neural Networks with End-to-Fan Learning of Feature Groups
Cosmological Findings. We then use the groups automati- cally generated by SOP and find the following new findings related to voids/clusters and lm//
D
1. Figure 8a shows that for both voids and clusters, when they are used in the prediction, they weigh higher when predicting fmm than dg..
(a) Weak
(b) Void
(c) Weak
(d) Cluster (> 3o: 6.4%)
Lensing Map << 0o: 76%)
Lensing Map 2
Figure 7. (a) and (b) show one weak lensing map with void (under- dense region) found by SOP. (c) and (d) show another map with cluster (areas of concentrated high density) found.
2. Figure 8b shows that a lot of voids contribute 100% to the prediction, while most clusters contribute partially. This finding is consistent with previous work (Matilla et al., 2020) that found that voids are the most impor- tant feature in prediction for low noise maps, as when the model does use voids, it relies on them more.
Void
Cluster
Void
Cluster
Therefore, we are able to obtain explanations meaningful to expert cosmologists from SOP, which is a promising step towards meaningful scientific discovery. We include additional cosmology background in Appendix D.
1.0
Attribution
Density
0.5
0.0
Related Works
0.00
0.25
0.50
0.75
1.00
Cosmological Parameter
Contribution
Self-attributing neural networks uses feature-based in- terpretable atoms and include per-feature models such as NAM (Agarwal et al., 2021), and group-based models such as BagNet (Brendel & Bethge, 2019) and FRESH (Jain et al., 2020). While previous works did use per-feature SANNs mainly in tabular data domains (Agarwal et al., 2021) and group-based SANNS in image and text domains (Brendel & Bethge, 2019; Jain et al., 2020), they did not formalize the problem of growing error in per-feature SANNs. Also, pre- vious models rely on specific architectures while our frame- work allows the use of arbitrary pre- -reitre models (Li, 2023; Jo & Kim, 2023). Other self-explaining neural net- works include prototype-bas basedd (Ma et al., 2024; Wen al., 2024) and concept-based (Koh et al., 2020; Yang et al., 2023; Lai et al., 2024; Yang et al., 2024) models, which explain with prototypical examples or concepts instead of input fea- tures. Self-attributing models that we use have attributions to input features, it not directly comparable to prototypes and concepts. Program-execution models (Lyu et al., 2023) provide faithful reasoning but do not attribute to features.
(a) Om vs. Ts Attributions
(b) Voids vs. Clusters Weights
Figure 8. (a) Qm relics more on cosmological structures than da. as shown by box plots of group weights from SOP for predicting Om and 0s. Voids on average contribute 50.7% to 2. and 41.5% to ds. and clusters contribute 34.0% to Ω.,. and 28.5% to as- (b) Many voids contribute 100% to the predictions, while clusters are weighted less, as shown by the density plot.
tions are used to validate utilities (Kim al., 2022; Akula et al., 2020; Hase & Bansal, 2020). We are using standard human distinction task from the HIVE protocol (Kim et al., 2022). Matilla et al. (2020) attempted to use post-hoc ex- planations in scientific discovery for cosmology. Our work uses a self- atriibutii model validate the same results.
6 Conclusion
We propose Sum-of-Parts (SOP), new framework for con- verting an arbitrary pre-trained model to self-attribting neural network. Our framework is model-agnostic and uses end-to-end learned groups without supervision, overcoming theoretical performance limitations of per-feature or fixed- group SANNs. SOP empirically achieves state-of-the-an performance among SANNs across multiple datasets, and is shown to be useful for model debugging and uncovering ground truth (Gao et al., 2022; Young et al., 2020) and an inpert-informed threshold-base purity when the ground truth not available (Matilla al., 2020). Human evalua-
For evaluating SANNs, Nauta et al. (2023) advocates prior- itizing performance over faithfulness as the primary metric: we adopt this approach in our evaluation. Coherence (Nauta et al., 2023) is also proposed to evaluate how well the expla- nation aligns with domain knowledge ground truth, using metrics such as Intersection over Union (Bau et al., 2017: Wang & Vasconcelos, 2020), outside-inside insiide relevanace ra- tio (Nam et al., 2020), and pointing game accuracy (Du et al., 2018; Huang & Li, 2020). We thus use IOU when there is
insights in cosmological discovery. We hope this general framework allows people to build their own self- self-attributing models more easily and use the resulting explanations to extract meaningful insights from complex patterns.

---

Sum-of-Parts:
Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Impact Statement
Kuropatkin, N., Lahav, O., Leget, P.-F,, Lemos, P., Liddle,
A., Lidman, C., Lima, M., Lin, H., MacCrann, N., Maia, M., Marshall, J., Martini, P., McCullough, J., Melchior, P., Mena-Fernándee J., Menanteau, F., Miquel, R., Mohr,
This paper presents work aimed at providing interpretable explanations for machine learning models that are impor- tantly, true the model's reasoning process, We show that the groups of features learned in our model-agnostic framework for group-based self-attributing models provide insights in scientific discovery such as cosmology. The groups generated by our model can also be trustworthy ref- erences for decisions in other high-stakes domains such as medicine, law, and automation. We hope our work can help improve the state of explainable and trustworthy machine learning models.
J., Morgan, R., Muir J., Myles, J., Nadathur, S., Navarro- Alsina, A., Nichol, R., Ogando, R., Omori, Y., Palmese, A., Pandey, S., Park, Y., Paz-Chinchón, F,, Petravick, D., Pieres, A., Malagón, A. P., Porredon, A., Prat, J., Raveri, M., Rodriguez--onroy, M., Rollins, R., Romer, A., Roodman, A., Rosenfeld, R., Ross, A., Rykoff, E., Samuroff, S., Sánchez, C., Sanchez, E., Sanchez, J., Cid, D. S., Scarpine, V., Schubnell, M., Scolnic, D., Secco, L., Serrano, S., Sevilla-Noarbe, I., Sheldon, E., Shin, T., Smith, M., Soares-Santos, M., Suchyta, E., Swanson, M., Tabbutt, M., Tarle, G., Thomas, D., To, C., Troja, A.. Troxel, M., Tucker, D., Tutusaus, I., Varga, T., Walker, A.. Weaverdyck, N., Wechsler, R., Weller, J., Yanny, B., Yin, B., Zhang, Y., and and, J. Z. Dark energy survey year 3 results: Cosmological constraints from galaxy clustering and weak lensing. Physical Review D, 105(2), January 2022. doi: 00,1103/phyxrev..105.22233 105.023520.
Acknowledgment
This research was partially supported by a gift from AWS AI to Penn Engineering's s ASSET Center for Trustworthy AI, by ASSET Center Seed Grant, ARPA-H program on Safe and Explainable AI under the award D24AC00253- 00, by NSF award CCF 2442421, and by funding from the Defense Advanced Research Projects Agency's ' (DARPA) SciFy program (Agreement No. HR001 1 12520300). The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government.
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Ad- vances in neural information processing systems, 31, 2018.
We thank Anton Xue, Shreya Havaldar, and everyone else at BrachioLab, and additionally Simeng Sun, Chaitanya Malaviya, Yue Yang for valuable feedback on writing. We thank Jiayi Xin for valuable discussion about this work.
A Agarwal, R., Melnick, L.., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., and Hinton, G. Neural additive mod- els: Interpretable machine learning with neural nets. In Beygelzimer, A., Dauphin, Y.. Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
References
Abbott, T., Aguena, M., Alarcon, A., Allam, S., Alves, O.,
Akula, A. R., Wang. S.,, and Zhu, S.-C. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In AAAI Conference Artificial Intelligence, 2020.
Amon, A., Seddrade-Oliviine F., Annis, J., Avila, S., Ba-
con, D., Baxter, F., Bechtol, K., Becker, M., Bernstein,
G., Bhargava, S., Birrer, S., Blazek, J., Brandao-Souza,
A., Bridle, S., Brooks, D., Buckley-Geer, E., Burke, D.,
Alvarez Melis, D. and Jaakkola, T. Towards robust in- terpretability with self-explaining neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018.
Camacho, H., Campos, A., Rosell, A. C., Kind, M. C., Carretero, J., Castander, F., Cawthon, R., Chang, C., Chen, A., Chen, R., Choi, A., Conselice, C., Cordero, J., Costanzi, M., Crocce, M., da Costa, I.,, da Silva Pereira, M., Davis C., Davis T., Vicente, D., DeRose, J., Desai, S., Valentino, E. D., Diehl, H., Dietrich, J., Dodelson, S., Doel, P., Doux, C., Wagner, A., Eckert, K., Eifler, T., Elsner, F., Elvin-Poole, J., Everett, S., Evrard, A., Fang. X., Farahi, A., Fernandez, E., Ferrero, L., Ferté, A., Fosalba, P. Friedrich, O., Frieman, J.,
Anders, J., Pasliev, P., Dombrowski. A.-K., Müller, K.- R., and Kessel, P. Fairwashing explanations with off- manifold detergent. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICML'20. JMLR. 2020.
J. Gatti, M., Gaztanaga, E., Gerdes, D., Giannantonio, T., Giannini, G., Gruen, D., Gruendl, R., Gschwend, J., Gutierrez, G. Harrison, I., Hartley, W., Herner, K., Hin- ton, S., Hollowood, D., Honscheid, K., Hoyle, B., Huff, E., Huterer, D., Jain, B., James, D., Jarvis, M., Jeffrey, N.,
Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3319-3327, 2017.
Jeltema, T., Kovacs, A., Krause, E., Kron, R., Kuehn, K.,
10

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-Enn Learning of Feature Groups
Bilodeau, B., Jaques, N., Koh, P. W., and Kim, B. Impos- sibility theorems for feature attribution. Proceedings of the National Academy of Sciences, 121(2), January 2024. ISSN 1091-6490. doi: 10. 10.10773pmas..3344/44 23044061204
image recognition scale. In International Conference on Learning Representations, 2021.
Du, M.. Liu, N., Song, Q. and Hu, X. Towards explanation of dnn-based prediction with guided feature inversion. In Proceedings of the 24th ACM SIGKDD International Con- ference on Knowledge Discovery & Data Mining, KDD '18. pp. 1358-1366 1367,1e6e New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 9781 450355520. doi: 10.1145/3219819.322000
Bluecher, S., Vielhaben, J., and Strodthoff, pling pixel flipping and occlusion strategy for consistent XAI benchmarks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Decou-
Böhle, M., Fritz, M., and Schiele, B. B-cos networks: Align- ment is all we need for interpretability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 10329-10338, June 2022.
Fluri, J., Kacprzak, T., Lucchi, A., Schneider, A., Refregier, A., and Hofmann, T. Full wCDM analysis of KiDS- 1000 weak lensing maps using deep learning. Physical Review D. 105(8), April 2022. doi: 10.1 10.pph//pys. 105. 105.083518.
Brendel, W. and Bethge, M. Approximating cnns with bag-of-local-feature models works surprisingly well on imagenet. International Conference on Learning Repre- sentations, 2019.
Gao, S., Li, Z.-Y., Yang. M.-H., Cheng, M.-M., Han, J., and Torr, P. Large-scale unsupervised semantic segmentation. TPAMI, 2022.
Chen, C., Liu, Y., Zhang, X., and Xie, S. Scalable explana- tion of inferences on large graphs. In 2019 IEEE Interna- fional Conference on Data Mining (ICDM), pp. 982-987, Los Alamitos, CA, USA, November 2019. IEEE Com- puter Society, doi: 10. (0.1109//DDM2201900000
Gatti, M., Sheldon, E., Amon, A., Becker, M., Troxel, M., Choi, A., Doux, C., MacCrann, N., Navarro-Alsina, A.. Harrison, I., Gruen, D., Bernstein, G., Jarvis, M., Secco, L. Ferté, A., Shin, T., McCullough. J., Rollins, R. P., Chen, R., Chang, C., Pandey, S., Tutusaus, I., Prat, J., Elvin-Poole, J., Sanchez, C., Plazas, A. A., Roodman, A., Zuntz, J., Abbott, T. M. C., Aguena, M., Allam, S., Annis, J., Avila, S., Bacon, D., Bertin, E., Bhargava, S., Brooks, D.,, Burke, D. L., Carneoo Rosell, A., Carrasco Kind, M., Carretero, J., Castander, F. J., Conselice, C.,, Costanzi, M., Crocce, M., da Costa, L. N., Davis, T. M., De Vicente, J., Desai, S., Diehl, H. T., Dietrich, J. P., Doel, P., Drlica- Wagner, A., Eckert, K., Everett, S., Ferrero, I., Frieman, J., García-Bellido, J., Gerdes, D. W., Giannantonio, T. Gruendl, R. A., Gschwend, J., Gutierrez, G., Hartley W. G., Hinton, S. R., Hollowood, D. L.. Honscheid, K., Hoyle, B., Huff, E. M., Huterer, D., Jain, B., James, D. J., Jeltema, T., Krause, E., Kron, R., Kuropatkin, N., Lima, M. Maia, M. A. G., Marshall, J. L., Miquel, R., Morgan, R., Myles, J., Palmese, A., Paz-Chinchón F., Rykoff, E. S., Samuroff, S., Sanchez, E., Scarpine, V., Schubnell, M., Serrano, S., Sevilla-Noarbe, I., Smith, M., Santos, M., Suchyta, E.,, Swanson, M. F. C., Tarle, G., Thomas, D., To, C., Tucker, D. L., Varga, N.N. Wechsler, R. H., Weller, J., Wester, W., and Wilkinson, R. D. Dark energy survey year 3 results: weak lensing shape catalogue. MNRAS, 504(3):4312 4336, July 2021. doi: 10. 00.10993/nrras/taab91...
Covert, I. C., Kim, C., and Lee, S.-I. Learning to estimate shapley values with vision transformers. In The Eleventh International Conference on Learning Representations, 2023.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp- 4171-41888 Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguisties. doi: (0.18653/v1/NI/ 1/N19/111191111131
Young, J., Jain, S., Rajani, N. F. Lehman, E., Xiong, C., Socher, R., and Wallace, B. C. ERASER: bench- mark to evaluate rationalized NLP models. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4443-4458 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 http://22222.cmaiii400000000 main.408.
Domahidi, A., Chu, E., and Boyd, S. ECOS: An SOCP solver for embedded systems. In European Control Con- ference (ECC), pp. 3071-3076, 2013.
Hase, P. and Bansal, M. Evaluating explainable AI: Which algorithmic explanations help users predict model behav- ior? In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pp.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M.. Heigold, G., Gelly, S., Uszkorcit, J., and Houlsby, N. An image worth 16x16 words: Transformers for

---

Sum-of-Parts: Self-Attrbuting Neural Networks with End-to-End Learning of Feature Groups
Online, July 2020. Association for Computational Lin- guistics. doi: 10. 186653//1/0220.ac-maa....
lensing mass map reconstruction. MARAS, 505(3):4626- 4645, August 2021. doi: 10. 10.00//maa////abllllll 1455.
Hesse, R., Schaub-Meyer, S., and Roth, S. Fast axiomatic attribution for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Jethani, N., Sudarshan, M., Covert, I. C., Lee, S.-I., and Ran- ganath, R. Fastshap: Real-time shapley value estimation. International conference on learning representations, 2021.
Huang, Z. and Li, Y. Interpretable and accurate fine-graie greegga recognition via region grouping. In The IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jo, W. and Kim, D. Neural additive time-series models: Ex- plainable deep learning for multivariate time-series pre- diction. Expert Systems with Applications, 228:120307,
2023. ISSN 0957-4174. doi: https://doi.org/ 10.1. eswa. eswa.2023.122 2023.1207....
Jain, S., Wiegreffe, S., Pinter, Y., and Wallace, B. C. Learn- ing to faithfully rationalize by construction. In Anmual Meeting of the Association for Computational Linguistics, 2020.
Kacprzak, T., Fluri, J., Schneider, A., Refregier, A., and Stadel, J. CosmoGridVI: a simulated LambdaCDM theory prediction for map-level cosmological inference. JCAP, 2023(2)0050, February 2023. doi: 10.1088/ 1475-7516/202230//0555
Jain, S., Salman, H., Wong, E., Zhang, P., Vineet, V., Vem- prala, S., and Madry, A. Missingness bias in model debugging. In International Conference on Learning Representations, 2022.
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., and Roth, D. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Walker, M., Ji, H., and Stent, A. (eds.). Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 252- 262, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/NI 1/1.//////11-//1 1023,
Jeffrey, N., Gatti, M., Chang, C., Whiteway, L., Demir- bozan, U., Kovacs, A., Pollina, G., Bacon, D., Hamaus, N., Kacprzak, T., Lahav, O., Lanusse, F., Mawdsley, B., Nadathur, S., Starek, J. L.,, Vielzeuf, P., Zeurcher, D., Alarcon, A., Amon, A., Bechtol, K., Bernstein, G. M., Campos, A., Rosell, A. C., Kind, M. C., Cawthon, R., Chen, R., Choi, A., Cordero, J., Davis, C., DeRose, J., Doux, C., Drliia- Wagner, A., Eckert, K., Elsner, F., Elvin- Poole, J., Everett, S., Ferté, A., Giannini, G., Gruen, D., Gruendl, R. A., Harrison, I., Hartley, W. G., Herner, K., Huff, E. M., Huterer, D., Kuropatkin, N., Jarvis, M., Leget, P. F., MacCrann, N., McCullough, J., Muir, J., Myles, J., Navarro- AAsina, A., Pandey, S., Prat, J., Raveri, M.. Rollins, R. P., Ross, A. J., Rykoff, E. S., Sánchez, C., Secco, L. F., Noarbe, I., Sheldon, E., Shin, T., Troxel, M. A., Tutusaus, I., Varga, T. N., Yanny, B., Yin, B., Zhang, Y., Zuntz, J., Abbott, T. M. C., Aguena, M., Allam, S., Madrade-Oliveing. F., Becker, M. R., Bertin, E., Bhargava, S., Brooks, D., Burke, D. L., Carretero, J., Ca- stander, F. J., Conselice, C., Costanzi, M., Crocce, M., da Costa, L. N., Pereira, M. E. S., De Vicente, J., Desai, S., Diehl, H. T., Dietrich, J. P., Doel, P., Ferrero, I., Flaugher, B., Fosalba, P., García-Bellido, J., Gaztanaga, E.,, Gerdes, D. W., Giannantonio, T., Gschwend, J., Gutierrez, G., Hinton, S. R., Hollowood, D. L., Hoyle, B., Jain, B., James, D. J., Lima. M., Maia, M. A. G.. March, M., Marshall, J. L., Melchior, P., Menanteau, F., Miquel, R.,
Kim, S. S. Y., Meister, N., Ramaswamy, V. V., Fong, R., and Russakovsky, O. HIVE: Evaluating the human inter- pretability of visual explanations. In European Confer- ence Computer Vision (ECCV), 2022.
Klauschen, F., Dippel, J., Keyl, P., Jurmeister, P., Bockmayr, M., Mock, A., Buchstab, O., Alber, M., Ruff, L., Mon- tavon, G., et al. Toward explainable artificial intelligence for precision pathology. Annual Review of Pathology: Mechanisms of Disease, 19(1):541-570, 2024.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, July 2020.
Lai, S., Hu, L.,, Wang, J., Berti-Equille L., and Wang, D. Faithful vision-language interpretation via concept bot- tleneck models. In The Twelfth International Conference on Learning Representations, 2024.
Mohr, J. J., Morgan, R., Ogando, R. L. C., Palmese, A.,
Paz-Chinchón, F., Plazas, A. A., Rodriguez-Monroy, M.,
Roodman, A., Sanchez, E., Scarpine, V., Serrano, S.,
Li, Q. A novel approach for balancing accuracy and inter- pretability in neural networks. 2023 3rd International Conference on Electronic Information Engineering and Computer (EIECT), pp. 464 470, 2023.
Smith, M., Soares-Santos, M., Suchyta, E., Tarle, G.,
Thomas, D., To, C., Weller, J., and DES Collaboration. Dark Energy Survey Year 3 results: Curved-sky weak
12

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Li, Z., Ji, J., and Zhang, Y. From kepler newton: Explain- able ai for science, 2021.
Artificial Intelligence, 34:2501-2508, 04 2020. doi: 10. 111609/aaii.334403.3....
Lombrozo, T. Simplicity and probability in causal explana- tion. Cognitive Psychology, 55(3):232-257, 2007. ISSN 0010-0285. doi: https://doi..or/110..000 10t.://0000..00o.ooosccc...... 09.006.
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., and Seifert, C. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. ACM Computing Surveys, 55(13s):1-43 July 2023. ISSN 1557-7341. doi: 10. 10.1145/3583558.
I Lundberg, S. M. and Lee, S.-I. A unified approach to inter- preting model predictions. In Proceedings of the 31st In- termational Conference on Neural Information Processing Systems, NIPS' 17, pp. 4768 1768-477, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Pan, D., Li, X., and Zhu, D. Explaining deep neural net- work models with adversarial gradient integration. In International Joint Conference on Artificial Intelligence, 2021.
Lyu, Q. Havaldar, S., Stein, A., Zhang. I.., Rao, D., Wong, E., Apidianaki, M., and Callison-Burch, C. Faithful chain-of-thought reasoning. In Park, J. C., Arase, Y., Hu, B., Lu, W., Wijaya, D., Purwarianti, A., and Kris- nadhi, A. A. (eds.), Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 305-329, Nusa Dua, Bali, November Nusa 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.ppp
Petsiuk, V., Das, A., and Saenko, K. Rise: Randomized input sampling for explanation of black-box models. In British Machine Vision Conference, 2018.
Poursabzi-Sangdeeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W. and Wallach, H. Manipulating and measuring model interpretability. In Proceedings of the 2021 CHI Conference on Human Factors in Comput- ing Systems, CHI '21, New York, NY, USA, 2021. Asso- ciation for Computing Machinery. ISBN 9781 9781450380966. doi: 10. 10.1145/34111664444533
Lyu, Q., Apidianaki, M., and Callison-Burch, C. Towards faithful model explanation in NLP: A survey, Compu- tational Linguistics, 50(2):657-723, June 2024. doi: 10.1162/coli 105111.
Reyes, M., Meier, R., Pereira, S., Silva, C. A., Dahlweid, F.-.,, von Tengg-Kobligk, H., Summers, R. M., and Wiest, R. On the interpretability of artificial intelligence in radiology: Challenges and opportunities. Radiology: Artificial Intelligence, 1900439 2020. doi: 10.1 10.1148/ 2yai.202019000
Ma. C., Donnelly, J., Liu, W., Vosoughi, S., Rudin, C., and Chen, C. Interpretable image classification with adaptive prototype-basee baaed vision transformers. In The Thirty- Annual Conference on Neural Information Processing Systems, 2024.
Ribeiro, M. T., Singh, S., and Guestrin, C. "why should i trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
Martins, A. and Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference Machine Learn- ing, volume 48 of Proceedings of Machine Learning Re- search, pp. 1614-1623, New York, New York, USA, June 2016. PMLR.
Ribli, D., Pataki, B. A., Zorrilla Matilla, J. M., Hsu, D., Haiman, Z.,, and Csabai, I. Weak lensing cosmology with convolutional neural networks on noisy data. Monthly Notices of the Royal Astronomical Society, 490(2): 1860, 09 2019. ISSN 0035-8711. doi: 10.1093/mmras. st72610.
Matilla, J. M. 7., Sharma, M., Hsu, D., and Haiman, Z. Inter- preting deep learning models for weak lensing. Physical Review D, 102(12), December 2020. ISSN 2470-0029.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Feii L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):111-52, 2015. doi:
doi: 10. 10.1103/phyxrvv..00.222255
MTurk. Amazon mechanical turk. https://wwxy mturk. com/ Accessed: 2024-01-28.
10. 10.10077/112633-15.oo
Nam, W., Gur, S., Choi, J., Wolf, L., and Lee, S.-W. Relative attributing propagation: Interpreting the com- parative contributions of individual units in deep neu- ral networks. Proceedings of the AAAI Conference on
Samek, W., Binder, A., Montavon, G., Lapuschkin, S., and Müller, K.-R. Evaluating the visualization of what a deep neural network has learned. IEEE Transactions on Neural
13

---

Sum-of-Parte Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Networks and Learning Systems, 28:2660-2673, 11 2017. doi: 10.1 S..110/NNNSS
Yang. Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-
Burch, C., and Yatskar, M. Language in bottle: Lan- guage model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In- ternational Journal of Computer Vision, 128:336-359, 2016.
Yang. Y., Gandhi, M.. Wang, Y., Wu, Y., Yao, M. S., Callison-Burch, C., Gee, J. C., and Yatskar, M. A text- book remedy for domain shifts: Knowledge priors for medical image analysis. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, Advances in Neural Information Processing Sys- tems, volume 37, pp. 90683-90713. Curran Associates, Inc., 2024.
Srinivas, S. and Fleuret, Full-gradient representation for neural network visualization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Sundararajan, M., Taly, A., and Yan, Q. Axiomatic attribu- tion for deep networks. In International Conference Machine Learning, 2017.
Tjoa, E. and Guan, C. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE transac- tions on neural networks and learning systems, 32(11): 4793 4813, November 2021. ISSN 2162-237X. doi: 10.1 2.1109/mnss.222030777..
Yu, H. and Varshney, L. R. Towards deep interpretability IMUS-ROVER II): Learning hierarchical representations of tonal music. In International Conference on Learning Representations, 2017.
Zednik, C. and Boelsen, H. Scientific exploration and ex- plainable artificial intelligence. Minds and Machines, 32 (1):219-239) 2022.
Tsang, M., Rambhatla, S., and Liu, Y. How does this in- teraction affect me? interpretable attribution for feature interactions. In Advances in Neural Information Process- ing Systems, 2020.
Zhu, Z., Chen, H., Zhang, J., Wang, X., Jin, Z., Xue, J., and Salim, F. D. Attexplore: Attribution for explana- tion with model parameters exploration. In The Twelfth International Conference on Learning Representations, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, At- tention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Zhu, Z., Chen, H., Zhang, J., Wang, X., Jin, Z., Xue, M., Zhu, D., and Choo, K.-K. R. Mfaba: more faithful and accelerated boundary-based attribution method for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 38(15):17228 17236, March 2024. doi: 10. sitiaaaaiii8i11..26666
Wang, P. and Vasconcelos, N. Scout: Self-aware discrim- inant counterfactual explanations. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp.
2020. doi: 10.1109/
Zytek, A.. Arnaldo, I., Liu, D., Berti- Berti-Equille, L., and Veera- machaneni, K. The need for interpretable features: Mo- tivation and taxonomy. ACM SIGKDD Explorations Newsletter, 24(1):1-13, 2022.
CVPP42600.2020.0000
Wang, S., Tang, H., Wang, M., Zhang, H., Liu, X., Li, W., Hu, X., and Zhang, L. Gnothi seauton: Empow- ering faithful self- inffrpetttret in black-box models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //wpnrevee....eece/e/e net/ forum? dVUMSKK
Wen, X., Tan, W., and Weber, R. O. Gaprotonet: multi- head graph attention-based prototypical network for in- terpretable text classification, 2024.
Wu, J., Kang, W., Tang, H., Hong, Y., and Yan, Y. On the faithfulness of vision transformer explanations. In 2024 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10936-10945, Los Alamitos, CA. USA, June 2024. IEEE Computer Society. doi: 10. 1109/CVPR5273..22440000
14

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
A Theory Details and Proofs
In addition to the insertion error, we can define the deletion error for a SANN and a target true function when excluding a subset.
Definition A.I. (Deletion Error) Let a; (x)),((re)) be the total contribution of the ith feature group the prediction of SANN. Then the deletion error of self self-attributing neural network f(x) Ei= ((z).h(za.)
Σ, ag for target
function f _ R when removing subset of features S from an input is
DelErr(G, a, 8) f(
Σ 03
SnG.z0
xy
j $ S
where (x-s)}
0
otherwise
Let [d] [1,...,d]
d]. Then the total deletion error over all possible deletions is Essia a,
We can solve the exact total deletion error for multilinear monomials with linear programs. These lowerbounds are then the minimum error in performance of SANNs attempting to learn the simple monomials.
Theorem A.2 (Lower Bound on Deletion Error for Monomials). Let p {0.1}* + (0,1} be a multilinear monomial
function of d variables, p(x) = I[ Then, Escial DelErr(G, a, S) 2 (A), where (del(A) = (A1 -x2) is the lower bound, X is a dual feasible point, and e is constant as defined in (9).
i=1 0g with a E Id be any per-feature self-attributing neural network. Consider the set of all possible perturbations to the input, or the power set of all features P((d), We can write the error of the self-attributing neural network under given perturbation
Proof. Let x 1a, and let f(x)= >i
P (or C [d] as
error(a, 5) = 1[$ + 01
Σ a iE
= cs - M al
(2)
where (Ms, are defined as (Ms); [1 if iES and es contains the remaining constant terms.
otherwise,
This captures the faithfulness notion that o, is faithful if it reflects a contribution of z; to the prediction of the target function. Then, the self-attris attributti neural network f(x) Σi= as with a" that achieves the lowest possible faithfulness error over all possible subsets is
arg min error(a, S)
(3)
SEP
This can be more compactly written as
a" = arg min Jc MMal
(4)
The minimum total deletion error can then be solved by the following linear program
P (a, 3) = min M TB
a
B 2 c - Ma
(5)
8 2 Mo c
To obtain the lower bound for total deletion error, we can solve the dual of this linear program. Given the above primal linear program, we can find the Lagrangian
L(a,3, =
B)
113- Mo) No (Ma- - c)
(6)
=1 1 + + - XJ
(1 t - X2) 32) Ma (A] - -o -\J)..
-2-
15

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
For the dual function be bounded below, the coefficients of and must be
0L M (A2 A1)
da
0L
\1 - A2 = 0= 0 +
03
Since we are minimizing over a, 9, the dual objective is to maximize
(Λ - X cc
(8)
subject to: \ + x2 1, M. -
In summary, the dual problem is
Daet (λ) max (A1 22)
21:d2
(9)
subject to: \ + 1 = 1, (A2 ( -1)= 0, 2.
Let À be feasible, then Dins (Aı - s InsErr(a, S). We can then maximize the lower bound to the primal program (5) in linear programming solvers such as CVXPY which maximizes the dual program (9).
EsEp
Conjecture A.1 (Deletion Error for Monomials Grows Exponentially with Dimension). Let {0,1}² {0, 1} be multilinear monomial function, p(x) H T4. Then, the lower bound of total deletion error for follows an exponential trend as dimension d grows, where the lower bound is approximately no + enn+rad, where (Hu:).:72)
(-1.030, -1.171, 0.665).
We solve for a" (5) using ECOS in the cvxpy library for d {2...., 203. To fit the exponential function, we fit a linear model to the log transform of the output which has high degree of fit (with relative absolute error of 0.012), with the resulting exponential function shown in Figure 2b.
In other words, Theorem A.2 states that we can find the exact lower bound of the total deletion error of a monomial, and Conjecture A. posits that lower bound of the total deletion error of any feature attribution of monomial grows exponentially with respect to the dimension, as visualized in Figure 2b
For digh-diimesiooall problems, this suggests that there does not exist feature attribution that satisfies all possible deletion tests. On the other hand, monomials can easily achieve low insertion error, as formalized in Theorem A.3.
Theorem A.3 (Insertion Error for Monomials). Let p: 1) - {0, 1} be multilinear monomial function of d variables, p(x)= _ ass Then, for all x, there exists a self-attributing neural network f(xx) = Ei=1 az for p at x that incurs at most total insertion error.
Proof. Consider a 0g. If I 1g then this achieves 0 insertion error. Otherwise, suppose x 1g. Then, for all subsets S * [4]. p(æs) = Eies O; incurs insertion error for all but one subset. For the last subset S [d]. the insertion enror is 1. Therefore, the total insertion error is at most for = 0g.
However, once we slightly increase the function complexity to binomials, we find that the total insertion error of any feature attribution likely follows an exponential trend with respect to d, as shown in Figure 2a.
Theorem 2.3 (Lower Bound on Insertion Error for Binomials). Let f0, > {0, be a multilinear binomial polynomial function. Furthermore suppose that the features can be partitioned into (S1, S2, Sz) of equal sizes where p(x) = llessus,, I4 + +lJessuss Zj. Then, Esç( InsErr(G,as a, a, S) 2 Dins (A), where Dine = (Å1 >>2)T c is the lower bound, Đ is a dual feasible point, and e is constant as defined in (16).
Proof. Consider x = 1a. The addition error for a binomial function can be written as
error(a,,
->e a; - US S S] - 1[S US S S]
= M a csl
(10)
iES
16

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
where (Ms, cs) are defined as (Ms), if S otherwise, possible insertion error that any attribution achieve is
and cs contains the remaining constant terms. Then, the least
arg min SEP
error(a, ara arg 1 Moa
(11)
The minimum total insertion error then be solved by the following linear program
(a, 8) = min M
a
82 c Ma
(12)
$ Ma -
To obtain the lower bound for total insertion emrr, we can solve the dual of this linear program. Given the above primal linear program, we can find the Lagrangian
L(a,3,A1 A2) = + 3) c-
- Ma) + No (Ma c) -
(13)
21 Mo N3 Ma Xo = -1 1 X2) 3 (A1 - x²) Ma + (A -2
For the dual function be bounded below, the coefficients of and 3 must be zero:
0L M (A2 \1)= =0
da
(14)
0L
\1 A2 0 0 +2=1
03
Since we are minimizing over a, 3, the dual objective to maximize
(A - >2 )c
(15)
subject to: ê + A2 = 1, -
In summary, the dual problem is
Dins (\) max(A1 - -22)
21.d2
(16)
subject to: \ + = 12 M (A2 - 0, de 0.
Let Å be feasible, then (A) (A1 To) ESEp InsErr(a, S). We can then maximize the lower bound the primal program (12) in linear programming solvers such CVXPY which maximizes the dual program (16).
Conjecture A.2 (Insetiion Error for Binomials Grows Exponentially with Dimension). Let be multilinear binomial function of d variables as defined in Theorem 2.3. Then, the lower bound of total insertion error for follows an exponential trend as dimension grows, where the lower bound is approximately Ag + edi where (A Sh,,)1,>u) (4.778, 1.332,0.198).
We maximize the lower bound by solving the dual linear program in (16) using ECOS in the cvxpy library for d e (2.... [.]....0....11 To get the exponential function, we fit a linear model to the log transform the output, doing grid search over the auxiliary bias term. The resulting function has a high degree of fit (with a relative absolute error of 0. 08188), with the resulting exponential function shown in Figure 2a.
17

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
A.1 Groups
Theorem A.4 (Insertion and Deletion Error for Groups). Consider pt and p2, the polynomials from Conjecture A.I and Conjecture A.2. Then, there exists group-based telf-attributing neural network with deletion and insertion error for both polynomials.
Proof. Let [d] denote (I....,d).
.d.. First let pi (x) II. æ; and consider self-attributing neural network with one group, f(x) = E ((z)dh(z(a)) E has one group = {[d] with contribution = {1). If 0,
S) = -2) - (*
Σ
= = 1 01 =0
Otherwise, matter what subset S is being tested, S [d] is always true, thus:
S) - (x-s)
Σ
11-0 -1-0-11=0
Oki
E87G,70
Therefore the total grouped deletion error for p1 0. Next let p2(2)
I, us, 2; IsEs,us, 4y and consider
self-attributing neural network with two groups, G {SI S2, S2 U S3} with contributions {1, 13. If S [d]. then
InsErr(G, - ((ss)
Σ ag -
GGCS
empty, then the insertion error is trivially Otherwise suppose S' is missing an element from one of S₁ or S3. WLOG suppose is from Si but S2 or S3. Then,
InsErr(G,,a, 8)
((as)-
Σ ai
=1- (1)
Otherwise, suppose are missing elements from both S₁ and S3. Then,
InsErr(G, ((xx)-
Σ ai
0
(0)
#Ggs
Lastly, suppose we missing elements from S2. Then,
InsExr(G, 8) ((xs)-
Σ az
=0-0=0
#G,5S
Thus by exhaustively checking all cases, p2 has zero grouped insertion error.
Therefore the total grouped insertion error for
P2 is O.
Theorem A.S (Detailed Statement: Insertion and Deletion Error for Groups for General m-nomial Polynomials). Let hisk
p IRd + R. be any general m-nomial polynomial function of order with m terms, p(x) = Em Σrek, aik II " C [d], G & Gevi, i' E [f,...,mb) K₃ E z+ is the set of indices for terms associated with group G. and bijk E z the exponent for feature j in subset in the term indexed by E K,. Then, a self aelf-attributing neural network needs at most m groups to achieve zero deletion and insertion error for polynomial p.
where (Gi,...,
Proof. Let [d] denote (1....,d) d). Let gi()) = ELEK, ask IyEG, 2. hisk be the ith polynomial with K; terms, and then we can rewrite p(x) q1 Q1(x) ... + (z). We prove by induction that we can have a self-attributing neural network of m groups = [G....., Gm) with contributions a {qi (z), ... 4m (x)) to achieve zero deletion and insertion error for polynomial p. For groups and contribution scores for up to m groups, we denote with ((m) and (m)) and omit the superscripts when the context clear.
18

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Base Case: Suppose m then pi (z) ELeK, aık IJEG, =y qi (a: and consider self-attributing neural network with one group, ( {Gi} with contribtuions a = {ar (x)). As there are no other input features, G1 = =c]. If S is empty, then the insertion error is trivially 0. If S = [d]. then G1 S S,
Insertion.
InsErr(G() (2) ((xs)- ((()
Σ a; (g)
Q1 (x)
Otherwise, S C [d] G1, then Gi S.
InsErr(G()) () (( (-(xs) - - Σ
04 = 00 0 - 0
HGGCS
We proved that we can have a self- self-attributinng neural network with groups G(!) = {G1 and group contributions a
{qi { for polynomial Pı (x) = 91 (z). which only has one group. Therefore, we can need at most one group to achieve zero grouped insertion error for monomial p1.
Inductive Step: Assume that it holds for (m 1)-nomial polynomial (m-1( q1(x)) + ..... qm 1 (2) that need at most (m 1) groups to achieve contributions a (m-1) = (x) ................................ (mm....... (x)) which means that
insertion emror, where the groups are ((m-1) {GI, Gm-1) with group
InsErr(G(m-ain (m-1))) .) ()) (1(xs) -
Σ ag iG
Pm (Is)
Σ
((xx) =0
(17)
iGGESA
This holds for all S C [d].
Now, we prove it for m-nomial polynomial Pm (x) = qı (x) + +... 9m (I). There are two cases. First, if G. & S, meaning that not all Gm features are in S, but there are some parts of Gm S. Then f (xs) does not contain the polynomial term Im (x) that uses Gm. Thus,
InsErr ( (m()) ()) ff
Σ a;
Pm (Is) -0 -
Σ
((x)
&G.CS iim
IIsEEr(G((--)) a (m-
S)
Otherwise, if G. C S, meaning that all Gm features are in S, and no features that are in Gm are contained in S, then (xx)) contains the polynomial term gm (x) that uses Gm. Thus,
InsErr InsErr(G(m), (m) (*(xs) (*(
Σ Oa
&G,5s
(am(Is) + Pmmm((5s) -0 - Σ (((x)
EGGES
(gm(xs) + (mmm((5s)) -0 (ml(xs) +
Σ
(x(x)
iG. mm
- - Pm-1 -
Σ
(x(
&G,CSA isfm
InsErr(G(mm1)) (m-1) (m- =)
The last steps of the above derivations use the induction from (17). Thus by exhaustively checking all cases, Pm. has zero
group insertion error with self-attributing neural networks with groups ((m) and group contributions (m)
19

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Deletion. Base Case: Suppose m 1, then pi (z) EKEK, aik JLES, 2 qi (z) and consider self-attributing
neural network with one group G {G1} and contribution a {x)(.)). As there are other input features, G1 = =d] If S is empty, then
DelErr(G (a. = f((x) -f
Σ
= llq((z) - 91 (z) 0 = 0
Otherwise, no matter what subset is being tested, S C [d] = S1 is always true, thus S n S; + 0:
(a = ff
Σ
= lqi((z) - (x)) = =0
kSnG:
We proved that can have self-attributing neural network f(x) = (()),((214) qı (xx) with groups G {G1} and group contributions (1)) {4 { for polynomial (zx) = (x), which only has one group. Therefore, we can need at most one group achieve zero grouped deletion error for monomial p1.
Inductive Step: Assume that it holds for (m n)-miil polynomial Pm-1 (a) Q1 (x) + + qm (I) that need most (m 1) groups to achieve grouped deletion error, which means that
DelErr(G(m-1), ( (m--)),)) f'(x) ff((zs))-
Σ ai
k:
(18)
(m=)(x)= (x
(((x) =0 0
u.80G / im
This holds for all S S [d].
Now, we prove for m-nomial polynomial Pm (x) = qi (x) + ... + 4m (2). There are two cases. First, if Gm & SS meaning that not all Gm features are -S, but there are some parts of Gm in S. Then (*-s)) does not contain the polynomial term Im (r) that uses Sm, and S n G, + 0. Thus,
DelErr(G(m)) (m))),
(x) (x)
Σ Oi
isSnG
m Σ (z()) Pm- Pm-1((--))
Im (m((z)
Σ
((xx)
ss5 A ttm
(zm( + Σ (i(x)) - Pm - 120-))
gm +
Σ
(i(x))
iSNG: A itm
(g)(() + Pmm)(x)) Pm- (2-s)) - 9m (m(()) +
Σ
(x)
isSn. A im
(x) -(m(x)) + Pm-1(1x) - Pm (m-1((x-))
Σ
(x(
&SnG: A if
DelErr(G(mn (mm)) 1) =
(m--1)
20

---

Sum-of-Partee Self-Attrbbttng Neural Networks with End-to-End Learning of Feature Groups
Minimum Addition Error
Minimum Deletion Error
2000
60
1500
1000
20
500
5
10
15
10
15
Size of Polynomial d
Size of Monomial d
(a) Minimum total square insertion for binomials.
(b) Minimum total square deletion for monomials.
Figure A9. Errors for per-feature SANNs grow fast unavoidably. The minimum (a) total mean square insertion error of monomials of size d and (b) total mean square deletion emrrrs of binomials of size d are the minima over all possible per-feature self-explaining models. The dots are the exact minima computed analytically, while the line is hest-fit expomential function.
Otherwise, if .. -, meaning that all S. features are in -S, and no features that are in Gm contained in S, then (*(-s) contains the polynomial term gm (2:) that uses Gm. and S n G; = 0. Thus,
DelErr(G(m) (m))
(x) - (xr-)) - Σ Oi
&SnG:
Σ (((x) - (gm + Pm (x)
Σ
((xx)
is80, A ifm
(4m (x)) + Pm- pm-)(s)) - (gm(x)) + Pm Pm-1((1-5)
Σ
(x(x)
&:8n A km
(4m - 4m (x)) Pm-1(1) Pm (x
(x(x))
&SnG; A iefmm
DelErr(G((mil a (m-1), 1) =S5==0 =0
The last steps of the above derivations use the induction from (18). Thus by exhaustively checking all cases, Pm has zero grouped deletion error with self-attributing neural network with groups ((m) and group contributions (m)
A.2 Alternative Mean Square Error Insertion/Deletion Errors
In the main paper, we use mean absolute error for Insertion Error and Deletion Error. Here we also compute the exact Deletion Errors for monomials and Insertion Errors for binomials up d 20. We conjecture that the two enrors will also follow an exponential growth.
Definition A.6. (Mean Square Deletion Error) Let diction a SANN. Then the mean square deletion error of a self-attributing neural network f(x) = Em ((x),)(xa.) =
be the total contribution of the ith feature group to the pre-
Σ, O4 for target function f Rd R when removing subset of features from an input
DelErrMse(G.aapp
(()) (((ss)) - Σ
SnG
[2 if S
where (x-s);
otherwise
21

---

Sum-of-Partee Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Let [d] (l,...,d .d. Then the total mean square deletion error over all possible deletions is Esc(a] DelErr(G,a, a,
Theorem A.7 (Lower Bound on Mean Square Deletion Error for Monomials). Let p : [0,11² + {0,1] be a multilinear monomial function of d variables, p(r) I Zs. Then, Esgla DelEiTMSE 2 a", where d'de = (M M) c the lower bound, and M and e are constants as defined (28).
Proof. Let 1a, and let E Rd be any feature attribution. Consider the set of all possible perturbations to the input, the power set of all features P, We can write the error of the attribution under given perturbation S E P as
error(a, S) = 115 + 01
aj IES
= |lcs - Mgall all
(19)
where cs) are defined (Ms): 1 if i EES otherwise,
and cs contains the remaining constant terms.
This captures the faithfulness notion that a; is faithful if reflects a contribution a; to the prediction. Then, the feature attribution a* that achieves the lowest possible faithfulness error over possible subsets
a² arg min error(a, S) arg min 1 c Mall SEP
(20)
1
if j j
where My-
for an enumeration of all elements S P.
0 otherwise
This is quadratic function without constraint, and thus we can analytically solve the exact minimum by finding where the gradient is zero.
To solve
min 1 ||c Mall
(21)
We first expand the squared norm
c 2
Ma)
(22)
Substituting into the objective function:
min 1 1 Ma) Ma)
(23)
Since is a summation operator over all elements, this simplifies to:
min Σ ||c (Ma)
(24)
Then, we compute the gradient. We define
f(a) Σ |c (I
(25)
Taking the derivative with respect to
VS(a) -2 (c (c- Mo)
(26)
Setting the gradient to zero:
Ma = =T
(27)
Finally, we solve for If M M is invertible, obtain the optimal solution:
Dn = (MTM) M
(28)
22

---

Sum-of-Parts: Self-Attrbuting Neural Networks with End-to-End Learning of Feature Groups
As M is the enumeration of all elements S₃ P, the columns in M are linearly independent each other, and thus M E {0, [0,1) 122dxd is invertible with column rank d.
rank(M) = d.
(29)
By the fundamental rank theorem,
rank(M T M) = (M)
(30)
Since M T M& E Zdxd it is then invertible.
Thus we prove that we can solve the optimal solution with (28).
Conjecture A.3 (Mean Square Deletion Error for Monomials Grows Exponentially with Dimension). Let 1] + {0, 1} be multilinear monomial function of d S 20 variables. p(z) II J₁. Then, the lower bound of the total mean square deletion error for follows an exponential trend as dimension d grows, where the lower bound of total mean square deletion error is approximately "Yo + tn where (70, 71, 72) = (-1.104, 1.7200, 0.626).
We solve for a" in (20) using (28) for d E f2..... 20). To fit the exponential function, we fit linear model to the log transform of the output which has high degree of fit (with relative square error of -0.293), with the resulting exponential function shown in Figure A9b.
Definition A.8. (Mean Square Insertion Error) Let = ((x),((xa.) be the total contribution of the ith feature group to the prediction of a SANN. Then, the mean square insertion error of a self attributing neural network f(z) = Σ ((r),((a.)) = E, as for a target function f Rd + when inserting a subset of features S from an input
MnEErrms(GGaa, a, = M(as) -(as)
Σ aj G.CS
where (Iss) = Ij if j E otherwise
The total mean square insertion error over all possible insertions is Esgia InsErr(G,,,5)
Theorem A.9 (Lower Bound on Mean Square Insertion Error for Binomials). Letp 10, + {0, 1,2) be multilinear binomial polynomial function of variables. Furthermore suppose that the features can partitioned into (S1, S2, Sa) of equal sizes where p(z) Ilee, 24 + -IJssus, Iy. Then, Esçiaj InsErrMse(G, a, S) 2 a", where aaa (MTM) MT c is the lower bound, and M and c are constants defined in (40).
Proof. Consider 1a. The addition emror for binomial function be written as
error(a: (a, S) =
Σ
a. - 1[S, U - U. S si
= =IM a- - cs cs
(31)
iES
if iES
where (Ms,es) es) are defined (Ms):
and es contains the remaining constant terms. Then, the least
0 otherwise,
possible insertion error that any attribution can achieve is
arg min error(a, S) arg
arg min 1 T ||c- Mall²
(32)
SEP
where (M, c) are constructed by stacking (Ms, cs) for some enumeration of S E P. This is quadratic function without constraint, and thus we can analytically solve the exact minimum by finding where the gradient is zero.
23

---

Sum-of-Parts: Self-Attribttng Neural Networks with End-to-End Learning of Feature Groups
To solve
min 1 ||c -
(33)
We first expand the squared norm as
||c -alM (c-Ma) Ma)
(34)
Substituting into the objective function:
min 1 Ma) Ma)
(35)
Since 1 is summation operator over all elements, this simplifies to:
min ||0 (Ma):ll
(36)
Then, compute the gradient. We define
f(a) E lca (M
(37)
Taking the derivative with respect to a:
Vf(a) = -2M (M)
(38)
Setting the gradient zero:
Ma M
(39)
Finally, we can solve for a. If M M is invertible, we obtain the optimal solution:
a (M M)_ T
(40)
As M is the enumeration of all elements S, P, the columns in M are linearly independent to each other, and thus M E 10, 1]2"xd is invertible with column rank d.
rank(M) = d.
(41)
By the fundamental rank theorem,
rank(MT = rank(M
(42)
Since M T M gdxd it is then invertible.
Thus we prove that we can solve the optimal solution with (40).
Conjecture A.4 (Mean Square Insertion Error for Binomials Grows Exponentially with Dimension). Let p: {0, + {0, 1,2) be multilinear binomial polynomial function of d < 20 variables. Furthermore suppose that the features partitioned into (S1,S2, Sz) of equal sizes where p(x:) = IIess,s,, Fs + IIJJusssss Iz. Then, the lower bound of total mean square insertion error for follows an exponential trend dimension d grows, where the lower bound is approximately Ao + exx((1 + X2d) total insertion error, where (Ao, \1, A2) (6.451, 1.457, 0.192).
be
We solve for a* in (32) using (40) for d E (,...., 20). To fit the exponential function, we fit linear model to the log transform of the output which has high degree of fit (with relative square error of -0.110), with the resulting exponential function shown in Figure A9a.
Additionally, the coefficients for the most critical exponential terms ?22 and À2 are very close between mean absolute error and mean square error. This shows that with both definitions of insertion/deletion errors we have consistent results.
24

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
A.3 Discussion on Theoretical Proofs
While our theorems and that of Bilodeau al. (2024) both present impossibility results for feature attributions, the assumptions and resulting theorem are different.
Bilodeau et al. (2024) put forth result that says (put simply) that linear models cannot accurately capture complex models, where complexity is measured by having a large number of piece-wise linear components. Indeed, if we had shown that a linear model is not a good approximation of a highly non-linear model, then this would not be a novel contribution. This is also an unsurprising result (it is not surprising that a linear model cannot approximate a highly non-linear model).
However, our result paints a significantly bleaker picture: we show that linear feature attribution is unable model the extremely simple functions in our theorems. Our examples distill the problem the fundamental issue in its purest form: correlated features. Specifically, is the polar opposite assumption than that of Bilodeau et al. (2024), and we argue that it is more surprising for feature attribution to be impossible for simpler functions than for complex functions.
show feature attribution is impossible with only one group of correlated features. This
Second, we provide not only a negative impossibility result for standard feature attributions, but also a positive result for grouped feature attributions that provides a path forward and motivates the approach in our submission. This is in contrast to Bilodeau al. (2024), which only presents negative impossibility results in standard feature attributions without clear suggestions on where to go.
In summary, our theoretical results differ in Assumption (we assume simple functions with single correlation whereas Bilodeau et al. (2024) assume complex functions with many piece-wise lincarities) Theoretical results (we show both positive and negative results, whereas Bilodeau et al. (2024) only show negative results).
B Method Details
B.1
Group Generator Details
The group generator I : Fd - {0, 1)mxdd takes in imput x E Id and outputs groups (91..... .m), Each 9s E {0, is a binary mask such that if the feature is included in group It. then Dij = 1, otherwise 0.
[(x)= = (0). 9m
(43)
To generate these groups, we first project the input æ into embedding dimension with embedding function he : Rd - dx In our experiments, we typically up to the second to last layer in the backbone model the embedding function to obtain the contextualized embedding of each feature, and we finetune this copy of projection layer while keeping the original backbone model unchanged.
Soft Multiheaded Attentottnnion Then, we use self attentttnnnoon mechanism a Ra + 10, 1|axd (Vaswani al., 2017) to parameterize a probability distributions over features.
a(x) softmax (a (a- (<))))(((()))))))
(44)
where Wo, Wx E & are learned parameters, and a is temperature scaling hyperparameter. In practice, one might not need that many groups and can use m < d drroups, resulting in a(x) E [o, 1]mmdd However, the outputs of self-attention are continuous and dense. To make groups sparse, we binarize the attention by taking top r = 0.2 features for each group. Fach group gi is then the binarized a(x) where gij if jth feature within top 7 according a(x), and otherwise gij = 0.
Input Embedding Encoder. In practice the embedding function he is initialized with parts of the backbone predictor h. The embedding function he can be finetuned as part of the group generator while not changing the backbone predictor.
Binary Groups and the Risk of subsets) are binary. However, the scores for each group are real valued. Since we are using discrete groups of features for making each prediction, the group masks need to be binarized. Nonbinary groups (i.e. close to zero but non-zero) will have information leakage from the nonzero input features.
Data. The groups (i.e. the image patch subsets or language token
A potential risk of blacking out pixels is that it will create are not significantly affected by this bias due to the significant data augmentations used during pretraining. For example,
(OOD) data. However, modern Transformers
25

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Jain al. (2022) show that ResNet suffers more from masking out tokens while ViT mostly is able maintain its original prediction even when some parts blacked out. In our ImageNet and MultiRC experiments, we these transformer based models, and all baselines use the same transformer in fairess, except XDNN, BCos and BagNet which do not have Transformer counterparts. For CosmoGrid, though we are using CNN-based model, the data form is very different from natural images, and zeroing parts of the map is not as OOD as it would be for natural images.
B.2 Group Selector Details
Sparse Multiheaded Cross-Attention. The group selector 0: {0, lmmxd Rd + [0, 11 then assigns weight each group such that sparse subset of the groups get nonzero weights. The scores (c1,.... Cm) are produced by cross attention using the target class's s weights Cj E IR* as the query and all groups' hidden states 2 = (hn (0) @ ....... ha (Im O z)) E Rm*k as the key, where hn Rd - R$ is usually the same backbone as h but outputs the hidden states with hidden dimension k:
((T()), = (c1...... Gm) = sparsemax (3 C2 (oWWx))
(45)
where Wor, WK E Rexk and C E Rt are learned parameters, and 3 is a temperature scaling factor.
Having scores on a sparse subset of groups helps avoid overloading human users. As the outputs of softmax are continuous and dense, we use a sparse variant, the sparsemax operator (Martins & Astudillo, 2016), to assign scores to the groups. The sparsemax operator uses a simplex projection to make the attention weights sparse and only assigns nonzero scores few groups.
In practice, we can initialize the query C₁ to row of the weight matrix in the linear classifier of the pretrained model h, and Wor and Wx' to the identity matrix, since the pretrained model already learns relation between the hidden states and class weights Cn.
Hidden State Encoder. In practice, the embedding function her and encoder for hidden states hsp are both initialized as parts of the backbone predictor h. The embedding function he can be finetuned as part of the group generator, while the hidden state encoder h4 is always kept frozen and computed at the same time of making the prediction with h. Therefore, h4, does not incur additional forward passes.
B.3 Scaling the Loss
Since the gradient cannot pass through the thresholded binary mask, we use a scale À E [-1,1] that is the difference between sums of attention weights of selected and unselected features and multiply that to the logits for optimization.
d
A
a(x)
top
a(x)) topr
ttop
1....,m
(46)
The gradient-passing scales (Ai...., ) the group scores (cI...., Cm). and predictions (J1...., are then combined with a weighted average to make the final prediction
f(x) = y- A1c1311 + + Cmmmm
(47)
Classification. For classification, we multiply the scaler À on the logits. If the prediction from group i is prefered, then A will also increase and thus up-weigh the selected features in the group. Conversely, if the prediction from group i is not prefered, then \ will decrease and thus dewn-weigh the selected features and up-weigh other features in the group.
Scaled CrossEntropyLoss((.. y", τ) CrossEntropyLoss
Xrc.yi v*)
(48)
where y is the predicted logit from each group and y" is the ground truth label.
Regression. For regression, multiply the negated scaler the loss instead of the logits because need to scale the loss instead of the absolute value.
Scaled MSELoss "J.""., xx = min( ΣΣ calus v-((~a))
(49)
26

---

Sum-of-Partss Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Experiments Details
C.1
Datasets
We evaluate SOP two vision tasks and one language task. For vision, ImageNet for image classification and CosmoGrid for image regression. For language, we use MultiRC for reading comprehension framed as binary classification.
C.1.1 IMAGENET AND IMAGENET-S
ImageNet (Russakovsky et al., 2015) is a standard image classification benchmark with 1000 classes.
Task: Image classification.
We use a Vision Transformer (ViT) (Dosovitskiy et al., 2021) ("googlelwit-hase-paachh 16-244") pretrained on ImageNet 2 21k and finetuned on ImageNet Ik. For the projection layer in the group generator, we use the second-to-last layer this ViT model.
Backbone Model:
Input Features: We use patches of size 3 x 16 x 16.
Ground-The Truth Segmentation for Purity: We use ImageNet-S (Gao et al., 2022), which contains ground-trute object segment annotations for 919 classes of ImageNet. Purity scores are computed using subset of the ImageNet validation set, with experiments on ImageNet-S done with one example for each class.
License: ImageNet permits non-commercial use.
C.1.2 COSMOGRID
CosmoGridV1 (Kaeprzak et al., 2023) is an image regression dataset for predicting two cosmological parameters related to the initial state of the universe: Qm (related to energy density) and dg (related to matter fluctuation) (Abbott et al., 2022). The task uses weak lensing maps, which are projected distributions of galaxy masses onto 2D image (Gatti al., 2021; Jeffrey et al., 2021).
The underlying data comes from the CosmoGridV suite, a set of cosmological N-body simulations produced with PKDGRAV3, a high-performance N-body treecode for self- saff-gravitating astrophysical simulations. These simulations span different cosmological parameters, including 12m and Øg. The output of the simulations are snapshots representing the distribution of matter particles as a function of position on the sky at different cosmic times (representing different distances from the observer). These simulation outputs are then post-processed weighted, projected maps of the mass distribution that can be estimated from current weak lensing observations (e.g., (Jeffrey et al., 2021)). The data we use follows the post-processing from Jeffrey et al. (2021), and we refer to it as CosmoGrid.
produce weak lensing mass maps, which
Task: Image regression to predict cosmological parameters 2n and Øg. Input size: (1 x 66 x 66), output size: 2.
We use a CNN (Matilla al., 2020) trained on the CosmoGrid training set. We use the second to last layer's output as hidden states, which has the dimension 14 14 32, meaning that the image of size 66 66 divided into 14 x 14 grids, each grid with approximately 5 5 4 x 4 pixels.
Backbone Model:
License: CosmoGrid (Kacprzak et al., 2023)6 has CC BY 4.0 DEED license.
1.3 MULTIRC
MultiRC (Khashabi et al., 2018), sourced from the ERASER benchmark (DeYoung et al., 2020), is reading comprehension dataset adapted for binary classification. Each example consists of passage, a question, and an answer; the task to predict the answer is correct (True or False) based the passage.
Task: Binary classification for reading comprehension.
'https://hughe http://ccccc.c.o..oggggoooo e/vi t- settt-baeeppaaeeee 2-224
27

---

Sum-of-Partes
Self-Attributing Neural Networks with End-to-Enu Learning of Feature Groups
Method
Abhr
Model-Agnostic
Self-Attrbbting
Learnable Groups
Self-Attributng Ver.
LIME (Ribeiro et al., 2016)
LIME
X
LIME-F
SHAP (Lundberg Lee, 2017)
SHAP
X
SHAP-F
IntGrad (Sundararajan et al., 2017)
IG
X
X
IG-F
GradCAM (Selvaraju al., 2016)
GC
V
X
X
GC-F
FullGrad (Srinivas & Fleuret, 2019)
FG
FG-F
RISE (Petsiuk et al., 2018)
RISE
X
RISE-F
Archipelago (Tsang et al., 2020)
Archi.
X
X
Archi.F
MFABA (Zhu et al., 2024)
MFABA
X
X
MFABA-F
AGI (Pan et al., 2021)
AGI
AGL-F
AMPE (Zhu et al., 2023)
AMPE
X
AMPE-F
BCos (Böhle al., 2022)
BCos
X
BCos-F
XDNN (Hesse et al., 2021)
XDNN
X
X
XDNN
BagNet (Brendel & Bethge, 2019)
BagNet
BagNet
FRESH (Jain al., 2020)
FRESH
FRESH
SOP (ours)
SOP
SOP
Table A4. Properties of all post-hoc feature attributions and self-attributing neural networks we use. SOP is the only attribution method that both model agnostic and self-attributing and has learnable groups.
Input Features: We use tokens as individual features.
Ground-Trund Annotations for Purity: For text experiments, we use annotations from the ERASER benchmark (DeY- oung et al., 2020) for purity computation.
C.2
Model Training
C.2.1 BACKBONE MODELS
We Vision Transformer (Dosovitskiy et al., 2021) (goggle/vi-base-patch1-- CosmoGrid experiments, we postprocessed data from Jeffrey et al. (2021) and CNN (Matilla et al., 2020) trained on the training set of the CosmoGrid backbone. We use BERT (Devlin et al., 2019) bbertbbas--uncase) finetuned on MultiRC for the text backhone.
for the ImageNet backbone. For
C.2.2 SOP MODELS
For SOP model training, we use learning rate of 5e-6 for all experiments. For ImageNet-S and MultiRC, we use one head in the attention in the group generator. For CosmoGrid, diverse groups, while the groups generated based queries from the same head are similar. For all experiments, only 20 groups from all the possible groups, by choosing attentions created by queries spaced out evenly. We find that we do not need more than 20 groups for good performance. For ImageNet, we train for epoch, but taking the checkpoint at 0.5 epoch as already converges, while we train for 3 epochs for CosmoGrid and 20 epochs for MultiRC. For ImageNet, we use Gaussian blurring of kernel size 5 on the patches, and kernel size 15 on MultiRC, so that we obtain more contiguous groups. For CosmoGrid though, we do use Gaussian blur because we do not assume that connected groups are better for weak lensing maps.
four heads. We observe that using more heads result in more
select
C.2.3 COMPUTE RESOURCES
We use NVIDIA A100 with 80G memory and NVIDIA A6000 with 48G memory on our internal cluster for experiments. Typically, one training for ImageNet finish in a day with one epoch. The training for CosmoGrid finish in 2 hours and converge in one epoch. The training for MultiRC also converges in one epoch. The evaluation takes around 5 minutes for each method. There were preliminary experiments that take more time to debug and modify the architecture.
C.3 Baselines
As we are building a new type of self-attie aelf-attributing neural networks, we compare with self-attributing neural networks that attribute to input features for all main experiments. The baselines we compare with are either already self-attributing neural networks, or converted from post-hoc attributions. The models that already fall under the class of self-attributing neural
28

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
networks are XDNN (Hesse et al., 2021), BagNet (Brendel & Bethge, 2019), FRESH (Jain et al., 2020) and SOP (ours). For post-hoe baselines, we construct a self-explaining version by passing thresholded post-hoe attributions into the backbone model to make predictions, following the framework of FRESH (Jain et al., 2020). We show all the methods we use in Table A4. We can see that SOP is the only method that is both model-agnostic and a self-attributing neural network.
Self-Attributing Neural Networks. For faithful baseline, the closest previous work we can compare with is FRESH (Jain et al., 2020). FRESH builds a two-stage system for faithful explanations for text. It takes the attention output from pretrained transformer, and binarize it by taking top r = 0.2, select the input tokens based on the binary mask, and then train another transformer to take the selected group of input tokens and make prediction only based on the group.
In FRESH, they train a separate model to predict with the selected group, while we freeze the pretrained backbone and train the group generator and group selector end-to-end for prediction. Since we build attentions outside the backbone model, we are able extract groups with any pretrained backbone model (such both transformer and CNN), instead of only transformer-basee based models. FRESH (Jain al., 2020) trains two separate components for group generation and the backbone. We assume that we don't change the backbone model, thus we only compare with similar version to FRESH where we take the attention from the Vision Transformer and pass directly into the backbone model.
Although NAM (Agarwal et al., 2021) and other faithful models also have faithful attributions to individual features, they require a separately trained submodule for each feature. For example, BagNet (Brendel & Bethge, 2019) uses group attribution on fixed patches, while XDNN (Hesse et al., 2021) attributes to pixels. When there are existing trained specialized models for a dataset (e.g. ImageNet), then we compare with them. However, when there is no such trained model for a new dataset (e.g. CosmoGrid), only compare with methods that utilize already trained models. This is because it is infeasible to train separate model from scratch for all different tasks.
Post-hoc-converted Faithful Model. We then compile a series of FRESH-like baselines, which take the attribution scores from different post-hoc methods, including attention, and take top r = 0.2 following Jain et al. (2020), and convert them into faithful explanations by making the prediction only based on these groups. The post-hoc attributions are from LIME (Ribeiro et al., 2016), SHAP (Lundberg & Lee, 2017), IntGrad (Sundararajan et al., 2017), GradCAM (Selvaraju et al., 2016), FullGrad (Srinivas & Fleuret, 2019), RISE (Petsiuk et al., 2018), Archipelago (Tsang et al., 2020), MFABA (Zhu et al., 2024), AGI (Pan et al., 2021), AMPE (Zhu et al., 2023), among which Archipelago already produces group attribution and we take the top predicted groups. We add "-F" after the name of each baseline to indicate that this is the faithful version using only the attributions for prediction, to differentiate with the original post-hoc methods. For SOP, we also take top 0.2 for each group
Other Baseline Details. The basclines BCos, XDNN, and BagNet all depend on specific models and cannot be applicd to any model backbone directly. For BCos, we use the simple btt.eeiittbbbbbbaeeee patch1 224 model that is a Vision Transformer model with the linear transformations replaced by their B-cos transformation. For XDNN, they remove the bias turn in AlexNet, VGG16 and ResNet50. As they do not have available version for Vision Transformers, we use their trained xfixup_resnet50 model for ResNet50 for our experiments. FRESH also depends on the attention mechanism inside the Transformer architecture, requiring the model backbone to be a Transformer model, and thus we use the attention from Vision Transformer for ImageNet experiments. BagNet also depends on its specific CNN and cannot be applied to existing trained models. Therefore, we only compare with BCos, XDNN, BagNet and FRESH for ImageNet and only compare with other agnos-cg baselines for CosmoGrid experiment.
C.4 Evaluation
C.4.1 SEMANTIC COHERENCE
Intersection-oove--nnioo (IOU) for ImageNet-S. and MultiRC For ImageNet-S (Gao et al., 2022), there are ground truth segmentations for subset of images in ImageNet. For MultiRC (Khashabi et al., 2018), there are ground truth human annotated explanations. We measure intersection-over-uiion (IOU) of the group with the ground truth annotations (object for I and explanation for MultiRC). Purity of group g: E {0, 1 with respect to the ground truth group o E {0, is then
1g: 0
Purity MulaRC (0i, (p) = -1-a
(50)
1g;
29

---

Sum-of-Partee Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Threshold-based Purity for CosmoGrid. In our collaboration with cosmologists, we identified two cosmological structures learned in our group attributions: voids and clusters. Voids are large regions that under-dense relative the mean density and appear as dark regions in the weak lensing mass maps, whereas clusters are areas of concentrated high density and appear as bright dots. As there are no ground-truth segments for voids and clusters, we use proxy function from our collaborator cosmologists to compute how much of a void or cluster a certain group is. In this section, we describe how we extracted void and cluster labels from the group attributions.
Let be a group from SOP when making predictions for an input z. Previous work (Matilla et al., 2020) defined a cluster as region with a mean intensity of greater than +3a, where a is the standard deviation of the intensity for each weak lensing identify groups containing clusters those whose features
map. This provides a natural threshold for our groups: have a mean intensity of +3a. Specifically, we calculate
Intensity(z,,
1
Σ E:S,00
S
Then, a group is labeled as a void if Intensity(a;, S) 2 30. Similarly, Matilla et al. (2020) define a void as a region with mean intensity less than 0. Then, group is labeled cluster Intensity(1,
O.
In consultation with cosmologists collaborators, refine the criteria in the main paper not just the mean intensity.
The alignment of group Ji to void is the percentage of pixels in the group that are below 0, given the mass y, is below 0. The aligmment of group 9i to cluster is the percentage of pixels in the group that are above 3 standard deviation of r's pixel values.
19t 3 < 0
Alig (O: x)
1[g < <0],
99:1
(51)
[oc C > (x)x
Alignaluuer (gt, x)
g
The purity is computed the mean of void alignment and cluster alignment scores that are above thresholds Tu and T. and averaged for the two predictions 2m and 0g.
CosmoGria (gt, x) 11 [AAigg (gis 1)) > Tel
(52)
[Align (gi, x) Tel
CosmoGrid Purity Threshold Ablations. In Table 1, we evaluate CosmoGrid purity with T, 0.6 and To 0.015, so that say group is void if more than 60% of its pixels are below O and the total mass is below 0, and group is cluster if more than 1.5% of its pixels are above (x))..
We show more ablation of how changing the hyperparameters in CosmoGrid purity affects the results in Figure A10. The result is that SOP consistently has better purity and performance trade-off than other baselines across all varying levels of Ta and T
Additional Examples We show additional examples in Figure A11, A12, A13, A14, A15. The groups obtained by SOP are the most semantically localized and coherent, and thus easy to interpret.
C.4.2 SPARSITY
Additional plots comparing how different sparsities in groups affect errors for CosmoGrid and MultiRC are shown in Figure A19. We can see that for CosmoGrid, SOP has the lowest MSE on average, and the best at sparsity 0.8 (keeping top 20% features) where it is trained. For MultiRC, SOP is competitive with FRESH while being better at the sparsity it is trained (sparsity 0.8).
C.4.3 FIDELITY
Fidelity assesses if attributions sum up to be the same as the model's prediction (Nauta et al., 2023). Faithful attributions should have low fidelity for the attributions of each feature to accurately represent the contribution from the feature.
30

---

Num-of-Parts: Self-Attribttngg Neural Networks with End-to-End Learning of Feature Groups
CosmoGrid CNN, 7. 0.2, 0.015
CosmoChid- - CNN, 0.4, T. 0.015
CosmoCrid - CNN, To 0.6, 0.015
CosmoGrid CNN, 7, 0.8, T.
Bx
*u cours)
*0l (ORES)
Backbok
SD2 [ouns
Backbopa
Puiny
KG
AMPE- AMPE-PLAME
MFABA-F AAII-R
Andi.
PG MIAADA
GC.PAGE M
+AMPE GIME
Archi.
Pusty
0.5
PG
GC
MIFAAA- Archia
0.25
0.20
IG-F
PIME-P
044
SHAP-R
IG-F
04
SHAP-F IG
RISE-
0.4
RISE-F MMPEE
SHA
RISE
AMPE-F
0.94
0.34 MSE Loss
0.66
MSE Loss
MSR Loss
MSE Los
CosmoCirid CNN, T.
CosmeGrid CNN, 066, 0.01
CosmoCrrd
T, 0.15
CosmoGrid CNN, 0.6, 0.02
Buokbol
Backleds
SDP (oins)
Backbo
Backhong
OP (OIES)
0.8
PG-F
FISF
0.6
RGGNGEE
0.5
MFABA
A
Paany
MFABA-F GC- AMPE
Purky
MPARA-F
Parity
NEABAAF
Andi. J
Porkky
0.4
AGLE
0,6
SHAP-RiSE histiie JLIME-F
0.5
SHAP
RISE
"GCC
Archi.
STLAP AMPEIF
XI-LIME F
0.3
SHAPP gDc-
RİSE-F AMPE-F
IGF
0.02
0.34
0.06
0.02
0.04
0.06
0.02
0.04
0.06
0.02
0.0C
0.06
MSE
MSE
Figure A10. (CosmoGrid Purity Threshold Ablation) Ablation of void/cluster thresholds Tμ and Te for MSF Loss Purity. The first four plots show that when we fix the To 0.015, SOP has better purity until To = 0.8, and the later four plots show that when we fix the To 0.6, SOP consistently have the best pareto frontier than all baselines. SOP the best on average.
H
Image:
(b) LIME.F (Ribeiro al., 2016)
SHAP-F (Lundberg & Loc, 2017)
(d) IG.F (Sundararajan et al., 2017)
FG-F
(g) RISE.F (Petsiuk al., 2018)
(h) Archi. (Tsang al,, 2020)
(Selvanju et al., 2016)
(Srinivas & Fleuret, 2019)
(i) MEARA-F
(k) (Zhu al., 2023)
(I) BCos-F (Bohle et al., 2022) (Hesse al., 2021)
(m) XDNN
(n) RagNet
(0) (Jain et al., 2020)
(p) SOP (Ours)
(Zhu et al,, 2024)
(Pan al., 2021)
(Brendel & Bethge. 2019)
Figure AIL. Example groups from different feature attribution methods for an image Tench. "-F" indicates self-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches), with unused patches hatched-out.
Fidelity can be measured by the KL-Divergence between the model predicted probabilities p(x) [0. 1] for all & classes and the probability using summed attributions p(x) [0, 1]th for all l classes (Yu & Varshney, 2017: Chen al., 2019; Anders al., 2020).
Fidelity En-D (KL(((z)) p(x))]
(53)
where أ = softmax(f(z)) is the predicted probability of the model, and = softmax EE a.) is the probability of summed attribution scores for data x in distribution DD As self-attributing neural networks all follow the design f(x) ( and the attribution scores a; h(r) self-attributing neural networks by design achieve fidelity. We report fidelity of post-hoe models in Table A5 and show that self-attributing neural networks are the only ones that achieve perfect fidelity.
For SOP, the sum of attribution scores is the sum of group predictions weighted by the group selector scores Σ, aj Σ, Cill. For FRESH (Jain et al., 2020), they only have one group, so the score for each class is the same as the prediction from that one group. All the self-attributing neural networks achieves fidelity of 0 by construction.
For most post-hoc baselines, the sum of attribution scores simply the sum of all the scores for each feature. We report the fidelity scores for post-hoe methods in Table A5. We can see that no post-hoe method achieves perfect fidelity of
31

---

Hum-of-Partss Self-Attrbbting Neural Networks with End-to-Enu Learning of Feature Groups
(a) Image:
(b) LIME-F (Ribeiro al., 2016)
(c) SHAP-F (Lundberg & Lee, 2017)
(d) IG-F (Sundararajan al., 2017)
(e) GC-F (Selvaniuu et al., 2016)
(l) FG-F (Srinivas & Fleuret, 2019)
(g) RISE-F (Pelsiuk al., 2018)
(h) Archi-F
Grocery store
(Tang et al., 2020)
ii MFABA-F (7hu et al,, 2024)
i AGLFF (Pan al., 2021)
(k) AMPE4F (7hu et al., 2023)
() BCo-FF (Bohle et al., 2022)
(m) XDNN (Hese al., 2021)
(n) BagNet
(0) FRESH (Jain et al., 2020)
(p) SOP (Ours)
(Brendel & Bethge, 2019)
Figure A12. Example groups from different feature attribution methods for image of Grocery store. "-F" indicates leff-atrrbutinn models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches). with unused patches hatched-out.
(a) Image: Cassette tape
(b) LIME-F (Rtheiro al., 2016)
SHAPPF (Lundherg Lee, 2017)
(d) IG-F (Sundararajan 2017)
(e) GC-F (Sevanjju 2016)
FG-F (Srinivas & Fleuret, 2019)
(g) RISE-F (Petsink al., 2018)
(h) Archi- (Tsang 2020)
ii) MEABA-F (7h al, 2024)
i AGLE (Pan al., 2021)
(k) AMPE-F (7hu al., 2023)
(I) BCos (Bohle al,, 2022) (Hesse 2021)
(m) XDNN
BagNet (Brendel & Bethge,
(0) FRESH (Jain 2020)
(p) SOP (Ours)
2019)
Figure A13. Example groups from different feature attribution methods for an image of Cassette tape. "-F" indicates self aoffattrrbuting models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches). with unused patches hatched-out.
C.4.4 INSERTION AND DELETION
Intuitively, if we add features from the most important to the least important one-by-one starting with a blank image, the probability for the predicted class should up quickly. Conversely, if we delete them from the most to least important, the probability should ideally drop instantly. The mainstream insertion and deletion metrics (Samek et al., 2017; Petsiuk et al., 2018) intend to evaluate the features are actually important by inserting or deleting them one-by-one and computing the trea-under-the-curve (AUC).
Original Insertion and Deletion.
As insertion and deletion criteria are designed for pixel-level instead groups features, we consider modified version of insertion and deletion for groups, which grow shrink the groups instead of adding pixels to the model. We report the percentage probability AUC instead of the absolute, to accommodate the difference in accuracy when the models are different. We use step size of 10% following previous work (Wu et al., 2024). Table shows that SOP performs the best for both insertion and deletion on ImageNet. Among other methods, the perturbation-be based methods such as SHAP and LIME perform better than gradient- based methods. Table A6 shows that SOP performs the best for insertion on Cosmogrid and MultiRC.
32

---

Sum-of-Parts Parts: Self-Attrbuting Neural Networks with End-to-Ein Learning of Feature Groups
(a) Image:
(b) LIME-F (Ribeiro al., 2016)
(c) SHAP-F (d) IG-F (Sundararajan et al., 2017)
(e) GC+F (Selvaraju al., 2016)
(f) FG+F (Srinivas Fleuret. 2019)
(g) RISE-F (Petsiuk et al., 2018)
(h) Archi-F
Casselie player
(Lundberg & Lee, 2017)
(Thang et al., 2020)
(i) MFABA-F (7hu et al. 2024)
①i) AGL-F (Pan al., 2021)
(k) AMPEsF (7hu et al., 2023)
() BCos-F (Bihle et al., 2022)
(m) XDNN (Hesse al., 2021)
(n) BagNet
(0) FRESH (Jain et al., 2020)
(p) SOP (Ours)
(Brendel & Bethge, 2019)
Figure A14. Example groups from different feature attribution methods for image of Cassette player. "-F" indicates ceff-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches). with unused patches hatched-out.
Image:
(b) LIME.F (Ribeiro al.. 2016)
SHAP-F (Lundberg & Loc, 2017)
(d)
FG-F (Srinivas & Fleuret. 2019)
(g) RISE-F (Petsiuk al.. 2018)
(h) Archi.
(Sundararajan al. 2017)
(Selvaraju al.. 2016)
(Tsang al., 2020)
MFABA-F Zhu et al., 2024)
(k) AMPE-F (Zhu al., 2023)
(1) BCos-F (Bohlc et al., 2022) (Hesse al., 2021)
(m)
BagNet (Brendel & Bethge. 2019)
(0) (Jain et al., 2020)
(p) (Ours)
(Pan al., 2021)
Figure A15. Example groups from different feature attribution methods for an image Castle. "-F" indicates self-attributing models converted from post-hoc methods. The highlights show the groups selected by each method (top 20% attributed patches), with unused patches batched-out.
The mainstream insertion and deletion metrics (Samek et al., 2017: Petsiuk et al., 2018) are approximations intended to evaluate the faithfulness of post-hoc methods (Nauta et al., 2023). Nevertheless, we include results for insertion and deletion for completeness.
Adapting Insertion and Deletion for Groups. The mainstream insertion and deletion criteria has problem when directly used to evaluate group attribution. When inserting single group of features, it is unclear what order one should use insert individual features. In images, inserting pixels by row versus randomly can result in very different results. As result, it is not immediately obvious how to directly apply the standard insertion and deletion criteria to the group setting. Nevertheless, we can consider a modified version of insertion criteria for groups, where instead of adding a single pixel at time, we instead gradually grow the size of all groups simultaneously. This procedure is equivalent to the classic insertion criteria when there is single group. Specifically, we grow the size of the groups gradually from with an interval of 10% of all features (approximately around 5017 pixels), and compute the AUCs of the highest predieted probability of the faithful models.
For deletion, we perform similar procedure by shrinking the groups gradually with the same interval of 10% pixels and computing the AUCs on the highest predicted probability the faithful models. Because each faithful model's prediction start with a different value, it is hard to compare the deletion score. Therefore, we make small modification to the original deletion metric and penalize the cases where the prediction wrong by setting the score of that point 1.
33

---

Sum-of-Parts Self-Attributnng Neural Networks with End-to-End Learning of Feature Groups
ImageNet Fidelity
CosmoGrid Fidelity
MultiRC Fidelity.
Method
LIME
3.866 + 0.244
0.100 + 0.000
1.550 0.271
SHAP
0,015 + 0.006
0.100 = 0.000
1.628 + 0.283
IG
7.161 + 0.212
0.100 = 0.000
1.921 = 0.276
GC
10,406 + 1.098
1.697 1 0.277
1.697 1 0.277
FG
13,567 +: 0.158
1.921 + 0.277
1.921 0.277
RISE
0.884 +: 0.533
0.100 + 0.000
2.771 + 0.231
Archi.
10.850 0.354
1.684 + 0.277
MFABA
6.674 0.166
1.690 0.267
1.690 0.267
AGI
5.416 - 0.549
1.665 = 0.254
1.663 0.254
AMPE
13.671 + 0.326
2.532 + 0.493
2.532 0.493
BCos
13.372 + 0.373
Table A5. Fidelity scores for post-hoc methods ImageNet, CosmoGrid, and MultiRC. Lower values indicate better fidelity. We omit numbers for self--xplaining models as they by design achieve fidelity of O. Post-hoc attribution methods are not able to achieve fidelity of which empirically shows that they not faithful the model prediction. For CosmoGrid, compute KL-divergence for fidelity, an additional softmax operation is applied on top of the CNN predicted logits as KL--divereence only works on probability distributions.
MultiRC
CosmoGrid
Category
Method
Ins.
Del.
Ins.
Del.f
LIME
0.869 + 0.020
0.778 + 0.005
0.028 + 0.001
0.028 + 0.001
SHAP
0.840 + 0.016
0.839 0.017
0.023 0.001
0.023 1 0.001
IG
0.878 = 0.009
0
0.027 = 0.001
0.027 = 0.001
GC
0.882 + 0.011
0.876
0,026 0006 0.001
1002=00000
FG
0.928 = 0.009
0.834
0,025 = 0.001
0.025 + 0.001
Host-Hac-Converite
RISE
0.96/ + 0.017
0.832 +
0.027 + 0.001
0.027 + 0.001
Archi.
0.669 + 0.021
0.920 0.015
0.036 0.002
0.036 + 0.002
MFABA
0.863 + 0.011
0.873 + 0.007
0.023 + 0.004
0.023 + 0.004
AGI
0,929 + 0.013
0,901 0.007
0.024 1 0.004
0.024 1 0.004
AMPE
0.868 + 0.007
0.892 1 0.018
0.027 1 0.004
0.027 1 0.004
FRESH
0.937 + 0.013
0.710 + 0.028
Self-Attributing
SOP
1.018 1 0.022
0.949 1 0.007
0.020 1 0.001 0.027 1 0.002
Table A6. Insertion/Deletion metrics for MultiRC and CosmoGrid. For MultiRC (left columns), higher insertion (Ins. and lower deletion (Del.j) are better. Best result is bolded, second-best is italicized. As FRESH is designed for text, it is unsurprising that it achieves better results on some metrics text. We compute percent insertion/deletion scores as discussed in Appendix C.4.4, which is why it possible for SOP to obtain insertion score larger than 1. For CosmoGrid (right columns), lower insertion (Ins..) and higher deletion (Del.1) are better due to MSE loss. Best result is bolded, second-best is underlined. All tables report percent insertion and deletion scores with an interval of 10%. FRESH is not applicable to CosmoGrid.
Also, as we are using different atlfibbtri neural networks, the base prediction can be different, making it unfair when comparing the insertion and deletion scores. We thus use the percent probability comparing with using all the input instead of the raw probability.
For self-attributing neural networks that seleet groups by thresholding attention, such SOP and FRESH, we grow the size of the group according to the attention scores. For models that have fixed group sizes, such as XDNN which uses groups of single pixels, and BagNet which uses groups of fixed patches, we perform insertion/deletion by adding/removing patches from the aggregation, following the same percentage for step sizes. Similar things are performed when evaluating accuracy for different sparsity levels.
Different Step Sizes. Here in the Appendix, we also report more finegrained insertion and deletion scores using step sizes of 16 x 16 patches (which equals 256 pixels) in Table A7. The result is very similar and SOP still achieves the best insertion and deletion scores.
Different Occlusion Strategies. There can be different occlusion values to use when masking out the groups. To test the choice of feature deletion values affects the results, we ran additional insertion/deletion experiments using an alternative deletion value as used in Bluecher et al. (2024) (specifically, randomly sampled color from the image). The results in
34

---

Sum-of-Partee Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
ImageNet- VIT
Category
Method
Step size 256 pix. Ins.t
Occlusion w, Colors Ins.t
Del.
Del.
LIME-F
0,815 = 0.005
0.428 + 0.004
0.797
0.007
0.555 + 0.023
S
0.831 = 0.006
0.373 + 0.008
0.909
0.018
0.524 = 0.023
IG-F
0.611 + 0.006
0.617 + 0.008
0.679
0.030
0.786 + 0.059
GC-F
0.772 +: 0.007
0.366 + 0.007
0.789
0.021
0.540 0.031
FG-F
0.759 + 0.005
0.383 + 0,004
0.830 0.029
0.447 + 0.026
Post-Hoc Hoc-Convverted
RISE-F
0,590 = 0.008
0.661 = 0.004
0.652 0.043
0.776 + 0.019
F
0.676 0.003
0.501 = 0.004
0.824 0.058
0.608 = 0.025
MFABA-F
0.674 = 0.006
0,499 = 0.010
0.855 0.048
0.610 0.085
AGI-F
0.735 + 0.006
0.462 + 0.008
0.875 0.048
0.585 + 0.050
AMPE-F
0.675 + 0.007
0.534 + 0.005
0.738 = 0.078
0.645 + 0.041
BCos-F
0.257 + 0.005
0.288 + 0.008
0.574 0.175
0.380 + 0.032
XDNN
0.199 + 0.007
0.156 + 0.003
0.245 0.038
0.254 + 0.044
BagNet
0.560 = 0.006
0.417 = 0.007
0.878
0.023
0.228 + 0.022
Self-Explaining
FRESH
0.713 = 0.002
0.369 0.004
0.746 0.033
0.512 + 0.046
SOP
0.890 1 0.004
0.014 0.000
0.910 0.010
0.106 + 0.000
Table A7. ImageNet Insertion (Ins.t) and Deletion (Del.J) metrics. Higher Ins. and lower Del. are better. Best results are bolded, second-bes, are italicized. Step size results use x 16 patches (256 pixels. 5% of image). Occlusion uses a random color. All tables report percent scores with an interval of 10%.
Table A7 aligns with our previous results of replacing the deletion value with 0 and SOP still performs the best for both insertion and deletion on ImageNet. This shows that the evaluation is consistent with other feature deletion values.
Discussion: Biases in Insertion and Deletion Tests. SOP performs well on the insertion test for all tasks, while being only the best on deletion task for one task. We notice a bias in the deletion test. As the deletion test favors SANNS such that deleting the most important feature results in sharp performance drop, biases towards models that rely primarily on a small number of features. On the other hand, if a model distributes its dependence to multiple different groups of features, removing the new most important features will not lead large performance drop. While deletion test is initially designed to evaluate post-hoe feature attributions that attempt to explain the same backbone model, it is not well-suited to evaluate self-attributing models, as it will score models that depend on few features more, regardless of the underlying faithfulness of the explanation.
For example, if there are multiple flowers in an image, and the model only looks at the upper left corner, while ignoring all other parts of the image, then removing the flower the upper left corner will reduce the model predicted probability for the flower O. However, if the model averages the prediction from different parts of the image, its predicted probability for the flower will only drop little bit if its most used flower is removed. This doesn't mean that the second model's explanation is any less faithful than the first one. On the other hand, the model's score for the one flower could be smaller number that faithfully reflects the small amount of confidence reduced.
Figure A16 shows average deletion curves for SOP. We see that even as we delete features from groups, SOP is able to maintain relatively high performance, resulting in a worse deletion score. Such behavior is natural because the training objective in SOP encourages the group selector to select highly predictive groups, and multiple groups can compensate for the information missing in another group.
Insertion/Deletion Results for CosmoGrid and MultiRC. Table A6 shows insertion and deletion results for MultiRC and CosmoGrid. We see that SOP is consistently good on the insertion metric while LIME is better at deletion. We conjecture that this is due different groups in SOP compensating for each other.
C.4.5 INFORMATION LEAK FROM THE GROUP GENERATOR
We show all the results for probing if the groups contain the labels in Table A8. We can see that models trained on the SOP groups are not able to predict the labels well. While results from linear and ViT models are not differentiating different methods, CNN probing models show that the groups from other methods like FG-F and AGI-F are leaking a lot of information. Archipelago is omitted because of the significant computational cost. to generate explanations for training
35

---

Sum-of-Partss Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Deletion Curve ImageNet
Percent Probability
Deletion Curve MulúRC
Deletion Curve CosmoGrid
Percent Probability
1.0
0.5
0.10
0.9
Loss
0.8
0.05
0.7
0.0
0.6
0.9
Fraction of Patches Removed
0.00
0.0
0.3
0.6
0.9
Fraction of Patches Removed
Fraction of Patches Removed
Method
LIME-F
RISE-F
BCos-F
Method
Method
SHAP-F
Archi.-F
XDNN
LIMB-F
GC-F
AGI-F
LIME-F
Archi.-F
AMPE-F
IG-F
MFABA-F
BagNet
SHAP-F
RISE-F
AMPE-F
SHAP-F
MFABA-F
FRESH
FG-F
AGI-F
FRESH
IG-F
Archi.FF
SOP
IG-F
AGI-F
SOP
GC-F
AMPE-F
SOP
RG-F
MFABA-F
RISE-F
Figure A16. Deletion curves across datascts. SOP's multi- group approach allows it to maintain low loss when only a few input features are preserved. Other methods show unstable behavior, especially CosmoGrid.
LIME-F SHAP-F IG-F
GC-F
FG-F MFABA-F AGI-F AMPE-F BCos-F XDNN BagNet FRESH SOP
Model
CNN
0.10
0.17
9.38
2.96
13.40
8.27
10.66
6.67
0.12
0.14
0.06
2.55
0.10
Linear
2.58
0.15
3.41
3.46
2.95
2.95
3.06
3.28
2.32
3.14
0.15
3.61
2.62
VIT
0.13
0.11
0.10
0.07
0.16
0.11
0.08
0.12
0.13
0.09
0.28
0.08
0.07
Table A8. (ImageNet Group Prohing Model Accuracy) A model trained on group masks from SOP is unable obtain accuracies much more than random. This indicates that the powerful group generator in SOP is not doing all the work and not compromising SOP's interpretability. RISE and Archipelago are omitted because the significant computational cost to generate explanations for training examples.
examples, while the experiments with all other baselines already demonstrate the interpretability of linear combination in SOP.
C.4.6 HUMAN DISTICTION TEST
SOP learns to use predictions from feature groups for the final prediction. However, if the group explanations are the same for different classes, they are not semantically relevant to the classes. We follow the HIVE protocol (Kim et al., 2022) and conduct a human distinction task on ImageNet-S to assess if people can predict which class the model predicts based solely on the group explanations for the classes. In the study, human evaluators are shown an image with group explanations for four classes and asked to guess the model prediction. Figure A18 shows bar plot of human distinction task accuracies, where explanations from SOP is among the ones having most distinctive explanation for the predicted class. The large eлror bars do not declare a single winner, while being consistent with original experiments in HIVE (Kim et al., 2022). Nevertheless, SOP groups for different classes are distinctive enough to provide meaningful explanations.
We follow HIVE (Kim et al., 2022) for constructing our distinction task for a human simulation test. Figure A A18 shows the accuracy of the human distinction task and its standard deviation bootstrapped 1000 times. This human evaluation conducted on Amazon Mechanical Turk (MTurk) on 100 examples each evaluated by 3 mturk users.
The large standard deviation is consistent with the literature and thus there is absolute winner method for human simulation (Kim al., 2022). Figure A17 show the interface display to the human evaluators.
HHere show the attributions for different classes from the original post-hoc methods, since the converted SANNs only uses group explanation-one the highest predicted classes.
36

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
Instruction
In this study, you will be asked to evaluate segments or areas in an image. An area in the image will be highlighted by black outline, and rest of the image will be hatched out. You will be asked to select the class you think correct based on the explanation the model gives for that class.
Photo
Class 1
Class 2
Class
Class
Which class do you think is correct?
O 1 O
2
O 3
O 4
Submit
Figure A17. Human Distinction Task MTurk Interface. Each worker shown the original image with four copies that have explanations for four classes highlighted. The predicted class is one the four classes. The worker asked to select the class that they think is correct based the explanations for each class.
Method
Error
IoU
Insertion
Deletion
FastShap-F
0.8270
0.0705
0,4556
0.7974
ViT-Shappey-F
0.7323
0.2447
0.5329
0.5715
AutoGnothi-F
0.7095
0.2229
0.5758
0.5337
Table A9. (ImageNet-S) Comparison of methods by error, IoU, insertion, and deletion metrics for Additional Baselines.
We pay each worker 0.05 per task, totaling 7.5 dollars per hour. For each task, we have three workers to evaluate. We show 100 images from 100 different classes (one image per class) for each explanation.
C.5 Computation Cost
Table 2 shows the computation cost of SOP and other models. All the baseline attribution methods whose extra number of forward passes can be controlled use at most 20 forward passes-the same number SOP We can see that other perturbation-basee baselines also require multiple forward passes. Gradient-based methods use one forward pass and we keep them that way. For Archipelago-F. it requires pairwise comparison and thus incurs a higher cost of (
C.6 Additional Baselines
Table A9 shows results of three additional basclines on ImageNet-S: FastShap-F (Jethani et al., 2021), ViT-Shapley-F (Covert et al., 2023), AutoGnothi-F (Wang et al., 2025). They are all post-hoc methods that attempt to approximate Shapley values faster, and we convert them as other post-hoc methods to self-attributing version. However, as they are approximations of Shapley values, they still fall short against SOP
37

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
ImageNet Vision'Trastormee
Human Simul. Acc.
0.6
Method
0.4
LIME
RISE
BCos
SHAP
Archi.
XDNN
0.2
IG
MFABA
BagNet
GC
AGI
FRESH
0.0
FG
AMPE
SOP
Figure A18. (ImageNet Human Distinction Task) We report humans distinction task accuracy for predicting which of four classes is the model prediction by looking at the attribution for each class (Kim et al., 2022). Most methods are similar the accuracy human predicting the correct class based solely the explanation, and the error hars are high, which is consistent with prior work HIVE while using a larger sample size than the original HIVE.
CosmoGrid - CNN
MultiRC - BERT
-
0.50
0.06
0.45
MSE
Error
0.04
0.40
0.35
0.02
0.2
0.4
0.6
0.8
0.2
0.6
0.8
Sparsity
Sparsity
LIME-F
GC-F
Archi-F
AMPE-F
LIME-F
GC-F
Archi-F
AMPE-F
SHAP-F
FG-F MFABA-F
SOP
SHAP-F
FG-F
MFABA-F
FRESH
IG-F
RISE-F AGL-F
IG-F
RISE-F AGL-F
SOP
(a) (CosmoGrid Sparsity vs. MSE) We report how mean square error increases when sparsity increases (fewer input feature are included in each group), where SOP's slowest increase is the most desired.
(b) (MultiRC Sparsity vs. Error) We report how enror increases
when sparsity increases (fewer input feature are included in each group), where SOP and FRESH have the desired slowest increases.
Figure A19. (Sparsity vs Error) for CosmoGrid and MultiRC
D Additional Cosmology Background
While outperforming other methods on standard metrics shows the advantage of our group attributions, the ultimate goal of interpretability methods is for domain experts to use these tools and be able to use the explanations in real settings. To validate the usability of our approach, we collaborated with domain experts and used SOP to discover new cosmological knowledge about the expansion of the universe and the growth of cosmic structure. We find that the groups generated with SOP contain semantically meaningful structures to cosmologists. The resulting scores of these groups led findings linking certain cosmological structures to the initial state of the universe, some of which were surprising and previously not known.
Weak lensing maps in cosmology calculate the spatial distribution of matter density in the universe using precise mea- surements of the shapes of 100 million galaxies (Gatti et al., 2021). The shape of each galaxy is distorted (sheared and magnified) due to the curvature of spacetime induced by mass inhomogenities as light travels towards us. Cosmologists have techniques that can infer the distribution of mass the universe from these distortions, resulting weak lensing map (Jeffrey et al., 2021).
Problem Formulation. Cosmologists hope to use weak lensing maps to predict two key parameters related to the initial state the universe: Pm and dg. 92m captures the average energy density of all matter in the universe (relative to the total energy density which includes radiation and dark energy), while dg describes the fluctuation of matter distribution (see e.g. (Abbott al., 2022)). From these parameters, a cosmologist can simulate how cosmological structures, such as galaxies,
38

---

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups
superclusters and voids, develop throughout cosmic history. However, 9.m and øg are not directly measurable, and the inverse relation from cosmological structures in the weak lensing map Ωn and Øg is unknown.
One approach to inferring 92m and dg from weak lensing maps, demonstrated for example by Ribli et al. (2019); Matilla et al. (2020); Fluri et al. (2022), is to apply deep learning models that can compare measurements to simulated weak lensing maps. Even though these models have high performance, we do not fully understand how they predict 2mm and Øg. As a result, the following remains an open question in cosmology:
What structures from weak lensing maps drive the inference of the cosmological parameters Dun and ag?
In collaboration with expert cosmologists, use convolutional networks trained to predict Ωm and dg as the backbone of SOP model get accurate predictions with faithful group attributions. Crucially, the guarantee of faithfulness in SOP provides confidence that the attributions reflect how the model makes its prediction, opposed to possibly being red herring. We then interpret and analyze these attributions and understand how structures in weak lensing maps of CosmoGridV1 (Kacprzak et al., 2023) influence Sm and os.
will be interesting to explore how these results change as we mimic realistic data by adding noise and measurement artifacts. Other aspects worth exploring are the role of "suuer-clusters" that contain multiple clusters, and how to account for the fact that voids occupy much larger areas the sky than clusters (i.e., should be surprised that they perform better?).
39


Reviews:
[{"review_id": "Jv2IO7vdvM", "reviewer": "Reviewer_GA2V", "summary": "The paper addresses the limitations of existing self-attributing neural networks (SANNs) in high-dimensional data. The authors theoretically prove a lower bound on the error of per-feature SANNs and demonstrate that group-based SANNs can overcome this limitation. The main algorithmic contribution is the Sum-of-Parts (SOP) framework, which transforms any differentiable model into a group-based SANN. SOP achieves state-of-the-art performance for SANNs on vision (ImageNet-S, CosmoGrid) and language (MultiRC) tasks. The learned feature groups are shown to be interpretable through quantitative metrics (performance at different sparsity levels, faithfulness) and semantic metrics (semantic coherence, human distinction). Furthermore, the paper demonstrates the utility of SOP explanations in model debugging and cosmological scientific discovery.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the valuable and encouraging review! We provide an additional PDF at https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf for additional figures.\n\n## Explaining the Deletion Performance of SOP.\nTo explain how SOPs usage of multiple groups can affect the deletion, we show the average deletion curves for SOP in Figure 2 in the linked PDF. We see that even as we delete features from groups, SOP is able to maintain relatively high performance in some cases, resulting in a worse deletion score. **Such behavior is natural because the training objective in SOP encourages the group selector to select highly predictive groups, and multiple groups can compensate for the information missing in another group.**\n\n## What is the computational complexity of SOP?\nThe cost of SOP (running time and memory) is dominated by the forward passes through the backbone predictor $h$, which can be seen in the definition of $f$ in Section 3. **Therefore, when using $m$ groups, the cost of an SOP forward pass is equivalent to $m$ forward passes through the backbone.** The costs of the group generator and the group selector are negligible in comparison, since each one is a much smaller attention module on sequences of $m$ inputs. We will make this cost explicit at the end of Section 3.\n\nIn our experiments, we control computation fairly across all baselines, which is discussed at length in Appendix C.2 & C.3. All methods can use at most 20 forward passes per inference with the exception of Archipelago which requires quadratic forward passes (Appendix C.3 line 1531-1533). For SOP, this amounts to **m=20 groups and thus 20 forward passes** per inference (Appendix C.2.2 line 1462-1463). SOP can match or beat all other baselines with equal compute.\n\nA detailed summary of computation is in Table 2 in the linked PDF, which we will add to Table 2 of the main paper to make this information more visible.\n\nLastly, the training needed to fine-tune SOP modules (Appendix C.2.2 line 1464-1465) are minimal compared to training the backbone (e.g. one epoch for ImageNet).\n\n## Questions about Group Diversity.\nMultihead attention and explicit diversity regularization could potentially change group diversity. We found that **simply using more attention heads in the group generator actually helped create more diverse groups,** an observation that we briefly mentioned in Appendix C.2.2 line 1462-1462. Concretely, we observed that using four heads in the group generator results in 32% fewer overlapping patches in groups than one head on ImageNet.\n\n## Comparing SOP to Baselines with the Same Backbones.\nWe confirm that SOP is already compared to baselines with the same backbones whenever possible. Specifically, **we compared 10 model-agnostic baselines that use the exact same backbone model as SOP**, and 4 baselines that are architecture specific. This is summarized in the main paper in Table 1 (see “Model-Agnostic” column) and discussed at length in Appendix C.3. We included architecture-specific baselines as readers wanted to see such comparisons. \n\n## Generalizability of Polynomials in the Theory.\nWe note that our theory tackles polynomials in the most general setting, with no restrictions on the type or degree of polynomials, and therefore has broad implications. Thanks to the Stone-Weierstrass theorem, every continuous function can be uniformly approximated by a polynomial function. **Since our result applies to any polynomial, it also applies to those that uniformly approximate real-world data patterns.**\n\n## Sensitivity to Sparsity / Number of Groups.\n- *Sparsity Analysis*. **We have already done this sparsity analysis in Section 4.2 RQ2 (Figure 3)**, with further analysis in Appendix C.4.2 (Figure 18).\n\n- *Number of Groups Analysis*. In Figure 1 in the linked PDF, we measure how the number of groups affects accuracy for ImageNet. Naturally, **having more groups improves accuracy**, and it saturates at 5 groups, suggesting that SOP can actually be 4x more efficient than originally reported.\n\n## Further Visualization & Analysis of Groups.\n**We have already included visualizations** in Figures 5+7 in the main paper as well as Figure 11~15 in the appendix for more visual examples, and **RQ5 and RQ6 do a semantic analysis of said groups**. Furthermore, **RQ7 and RQ8 provide insights on how different features groups learned with SOP affect predictions**.\n\n## More Context for Cosmology Study.\nWe referenced additional detailed and accessible cosmology background in Appendix D, where we introduce the problem and discuss the kinds of findings and groups cosmologists find interesting. \n\n## Biases in Pretrained Models.\nPretrained models are known to encode a range of social biases including racism and sexism. One could utilize algorithms that edit models for fairness or debiasing models. Such biases equally affect all baselines in our work using the same pretrained model.", "claims_and_evidence": "Overall, the claims are well-supported by evidence; however, some minor points may require further clarification or additional evidence.\n\n1. Performance : While SOP achieves strong results on insertion metrics, its performance on deletion metrics is somewhat weaker compared to certain baselines like Archipelago or FRESH in specific cases. Although the authors explain this behavior as natural due to SOP's reliance on multiple feature groups rather than single groups, this aspect could benefit from further empirical exploration to fully justify this claim.\n\n2. Overhead : The authors propose SOP as a flexible and model-agnostic framework; however, they do not extensively discuss computational overhead or scalability implications clearly. Given that SOP involves dynamic attention-based group generation and multiple forward passes through pre-trained models per input example, computational efficiency could be a concern in practical applications.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem at hand. However I have some doubts and would like the authors comment on them.\n\n1. Group Diversity: Do you think multi-head attention may produce redundant groups? If yes, methods like diversity regularization could help?\n\n2. Deletion Bias: Deletion tests generally favor models using fewer features, slightly disadvantaging SOP’s multi-group approach.", "theoretical_claims": "The theoretical claims are well-supported by rigorous proofs and empirical validation, particularly for Theorems 2.3 and 2.4. The authors also posit conjectures (e.g., Conjecture A.1 and A.2) suggesting exponential growth of insertion/deletion errors for monomials and binomials as feature dimensions increase. These conjectures are supported by empirical fits to numerical results but lack formal proofs.", "experimental_designs_or_analyses": "The experimental designs and analyses are well-executed and provide convincing evidence for most claims made in the paper. However there are some minor issue where where further validation could strengthen the work.\n\n1. Pre-Trained Backbones: SOP uses pre-trained models (e.g., ViT, BERT), while some baselines (e.g., BagNet) do not. Hence it is ideal to compare SOP to baselines using the same pre-trained backbone for fairness.\n\n2. Computational Efficiency : SOP’s dynamic group generation and sparse attention may increase computational cost compared to baselines. Hence the authors should include a detailed analysis of training/inference times and memory usage.\n\n3. Generalizability: The proofs focus on polynomial functions, which may not capture all real-world data patterns (e.g., non-polynomial interactions). I would like the authors to comment here.", "relation_to_broader_scientific_literature": "SOP's key contributions lie in its theoretical insights into the limitations of per-feature SANNs, its novel framework for end-to-end learning of semantically coherent feature groups without supervision, its state-of-the-art performance among SANNs, and its rigorous validation of interpretability and utility in diverse applications. These advancements significantly contribute to the ongoing research efforts in the field of interpretable machine learning.", "essential_references_not_discussed": "The paper does a thorough job of discussing related works in the context of self-attributing neural networks and interpretability.", "other_strengths_and_weaknesses": "The paper makes a compelling contribution to interpretable ML by addressing the trade-off between performance and faithfulness in SANNs. But some minor weaknesses exist according to me.\n\n1. Computational Cost: While the paper demonstrates strong performance, it does not extensively discuss the computational cost associated with the SOP framework. The use of attention mechanisms, especially multi-headed self-attention, can be computationally intensive. Providing an analysis of the training and inference time complexity and potentially comparing it to other SANN methods would be valuable for understanding the practical feasibility of SOP.\n\n2. Hyperparameter Sensitivity: The paper mentions using a sparsity level of τ=20% for the group generator but does not delve deeply into the sensitivity of the results to different hyperparameter settings, such as the number of groups (m) or the sparsity level. An analysis of how these hyperparameters affect the performance and interpretability of SOP would strengthen the paper.\n\n3. Interpretability of Learned Groups: While the paper presents quantitative and qualitative evidence for the interpretability of the learned groups, further visualization and analysis of the actual feature groups discovered by SOP could provide more intuitive insights. Understanding what kind of feature groupings the model learns for different tasks and classes could further enhance the understanding and trust in the explanations provided by SOP.\n\n4. Cosmological Evaluation Metrics: The threshold-based purity metrics for CosmoGrid rely heavily on domain expertise. While justified by collaboration with cosmologists, these metrics may be less accessible or transparent to non-expert readers without additional context or sensitivity analyses.", "other_comments_or_suggestions": "Typos:\n\n1. L89: \"anddifferent\" -> \"and different\"\n2.L211: \"in and Figure\" -> \"in Figure\"", "questions_for_authors": "It would be great if the authors consider below recommendations to improve the paper:\n1. Provide a computational complexity analysis and discuss optimizations for scalability.\n2. Include a discussion of potential biases inherited from pre-trained backbones and mitigation strategies.\n\nOverall, the paper presents significant original contributions through rigorous theory, innovative methodology, strong empirical validation across multiple domains, and practical utility in scientific discovery. Minor weaknesses include limited exploration of computational overhead, and brief explanations regarding certain evaluation metrics, which I mentioned before. These weaknesses do not substantially undermine the paper's core contributions but indicate areas where additional clarity or evidence could further strengthen the work.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x1pKyInmLB", "reviewer": "Reviewer_rZvr", "summary": "The paper introduces Sum-of-Parts (SOP), a framework for transforming any differentiable model into a group-based Self-Attributing Neural Network (SANN). The key innovation is the end-to-end learning of feature groups without requiring explicit group supervision. SOP addresses the limitations of per-feature SANNs, which struggle with high-dimensional, correlated data, by proving that group-based SANNs can achieve zero error if the groups align with the underlying correlations in the data. The framework consists of three components: a group generator, a backbone predictor, and a group selector. SOP achieves state-of-the-art performance on vision (ImageNet, CosmoGrid) and language (MultiRC) tasks while providing interpretable and faithful explanations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We provide an additional PDF at https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf for additional figures and will respond to your questions below.\n\n## Comparison with Additional Shapley-Based Baselines\nWe first would like to clarify a key distinction between SANNs and the referenced line of Shapley-based methods.  Specifically, **these Shapley-based methods (SBMs) are not Self-Attributing Neural Networks (SANNs)**. Instead, these methods are faster approximations of the SHAP baseline that we already compare to. The goal of these SBMs is to efficiently approximate post-hoc explanation Shapley values, with later works (FastSHAP, ViT-Shapley, Auto-Gnothi) training additional surrogate models to estimate Shapley values. AutoGnothi, for example, trains multiple side networks to generate approximations of Shapley explanations. **These SBM explanations are crucially never passed into the backbone to make a prediction.** In contrast, in a SANN, the features in an explanation are directly passed into the backbone to guarantee that predictions rely solely on the explanation. Therefore, the explanations of AutoGnothi and other Shapley-based methods are more similar to post-hoc explanations and do not fall into the category of SANNs. This fact is reflected in SBM experiments: all the papers of FastShap, ViT-Shapley, and Auto-Gnothi only compare to post-hoc explanation methods and do not compare to any SANNs, as a direct comparison is not possible. \n\nIn our paper, we provide comparisons to SANN variants of post-hoc methods. Since we already compared to a SANN variant of SHAP (in Table 1, denoted SHAP-F), for completeness, we ran analogous comparisons with SBMs using their released codebases. Since the suggested methods involve training additional side networks, for fair comparison, we ensured that all baselines had equal or greater training resources than SOP. **We show the results in Table 1 in the additional PDF,  where we find that their performance as a SANN is actually even worse than SHAP, and consequently worse than SOP.** Intuitively, this result is not too surprising since these methods are faster but more inaccurate approximations of SHAP, trading off accuracy of the Shapley value for speed. KernelSHAP was too expensive to be fairly run as a baseline, as it requires a minimum number of forward passes at least equal to the number of features to find the solution to their loss function in Theorem 2 in their paper and code [2], making it an order of magnitude more expensive than all other baselines.  \n\n## On Computational Cost of SOP\nSince the reviewer brings up computational cost as a potential weakness, we point out that **our SANN experiments are orders of magnitude larger scale than those of the referenced SBMs**. The AutoGnothi and ViT-Shapley papers conduct experiments on small scale datasets such as ImageNette, which only has 10 easily distinguishable classes where it is easy to get 100% accuracy. In contrast, **we train SANNs on full ImageNet (1000 classes!), where SOP achieves SOTA performance with only one epoch of training.** Since we are evaluating SBMs in a significantly larger setting than originally proposed,  we checked the accuracy of surrogate models to ensure these baselines were trained properly. We confirmed that the resulting surrogate models trained on masked inputs have comparable accuracy on ImageNet to other work in the literature [1, Figure 10] that also trains models on similarly masked inputs, so we are certain these baselines were sufficiently trained. \n\nIf the reviewer is still concerned about cost, we have also included an ablation study (Figure 1 in the additional PDF) showing that SOP performance on ImageNet saturates at just 5 groups, meaning we can further reduce inference costs by an additional factor of 4 beyond what we originally reported.\n\n[1] Jain et al. Missingness Bias in Model Debugging. ICLR 2022.\n\n## Human Distinction Task With More Samples\nFirst, we point out that **our human distinction task follows the exact same protocol as the original HIVE paper published at ECCV** [2a], which uses 10 examples for evaluating human distinction tasks [2b]. We believe this to be partly because evaluating the task on a single example requires significant human effort, with the original study consisting of hundreds of evaluator tasks. **We have increased the study to 50 examples,** which amounts to thousands of evaluator tasks. In summary, we found results consistent with what we originally reported previously (that SOP is among the best, but that the distinction test cannot distinguish between the best), but with slightly smaller error bars and smaller differences between methods. The expanded study is shown in Figure 3 in the additional PDF.\n\n[2a] Kim et al. HIVE: Evaluating the Human Interpretability of Visual Explanations. ECCV 2022.\n\n[2b] https://github.com/princetonvisualai/HIVE/blob/main/materials/HIVE_suppmat.pdf", "claims_and_evidence": "1. Claim 1: Per-feature SANNs have a lower bound on error that grows with the number of features, especially when features are highly correlated.\n\nEvidence: The paper provides theoretical proofs (Theorems 2.3 and A.2) showing that per-feature SANNs cannot model even simple polynomial functions with correlated features. Empirical results (Figure 2) show that the error grows exponentially with the number of features.\n\n2. Claim 2: Group-based SANNs can achieve zero error if the groups align with the underlying correlations in the data.\n\nEvidence: The paper proves (Theorems 2.4 and A.5) that group-based SANNs can perfectly model complex polynomials with zero insertion and deletion errors. Empirical results show that SOP outperforms other SANNs and post-hoc methods on vision and language tasks.\n\n3. SOP achieves state-of-the-art performance while maintaining interpretability.\n\nEvidence: SOP achieves the lowest errors on ImageNet (0.267), CosmoGrid (0.025 MSE), and MultiRC (0.366 error), outperforming baselines like SHAP-F, FRESH, and BagNet (Table 1). The learned groups are validated using quantitative metrics (e.g., intersection-over-union, purity) and human evaluations.\n\nHowever, the authors do not compare a recent self-interpretable works, i.e., AutoGnothi[1], which limit its trustworthiness. I suggest the authors compare SOP with AutoGnothi.\n\n[1] Wang S, Tang H, Wang M, et al. Gnothi Seauton: Empowering Faithful Self-Interpretability in Black-Box Models[J]. ICLR, 2025.", "methods_and_evaluation_criteria": "1. Methods:\n- Group Generator: Uses a multi-headed self-attention mechanism to dynamically generate feature groups for each input.\n- Backbone Predictor: Makes predictions for each group using a pre-trained model (e.g., Vision Transformer, CNN, BERT).\n- Group Selector: Assigns scores to each group using a sparse cross-attention mechanism, ensuring that only a few groups contribute to the final prediction.\n\n2. Evaluation Criteria:\n- Performance: Error rates (ImageNet, MultiRC) and mean squared error (CosmoGrid).\n- Interpretability: Intersection-over-union (IOU) for ImageNet-S, threshold-based purity for CosmoGrid, and human distinction tasks.\n- Faithfulness: Fidelity (KL-divergence between model predictions and summed attributions), insertion and deletion tests.\n- Utility: Model debugging (correct vs. incorrect predictions) and scientific discovery (cosmology insights).", "theoretical_claims": "1. Theoretical Claim 1: Per-feature SANNs have a lower bound on error that grows with the number of features, especially for correlated data.\n\nSupport: Theorems 2.3 and A.2 prove that per-feature SANNs cannot model simple polynomial functions with correlated features. The error grows exponentially with the number of features (Figure 2).\n\n2. Theoretical Claim 2: Group-based SANNs can achieve zero error if the groups align with the underlying correlations in the data.\n\nSupport: Theorems 2.4 and A.5 show that group-based SANNs can perfectly model complex polynomials with zero insertion and deletion errors.", "experimental_designs_or_analyses": "1. Datasets: ImageNet, CosmoGrid, and MultiRC.\n2. Baselines: SANNs: XDNN, BagNet, FRESH.\n3. Post-hoc Methods: LIME, SHAP, IG, GradCAM, RISE, Archipelago, MFABA, AGI, AMPE, BCos.\n\nPlease compare Shapley Value based methods like KernelSHAP[2], FasSHAP[3], ViT-Shapley[4] and AutoGnothi[1].\n\n\n[2] Lundberg S M, Lee S I. A unified approach to interpreting model predictions[J]. Advances in neural information processing systems, 2017, 30.  \n[3] Jethani N, Sudarshan M, Covert I C, et al. Fastshap: Real-time shapley value estimation[C]//International conference on learning representations. 2021.  \n[4]. Covert I C, Kim C, Lee S I. Learning to Estimate Shapley Values with Vision Transformers[C]//The Eleventh International Conference on Learning Representations.", "relation_to_broader_scientific_literature": "I believe it is crucial to investigate self-attributing in terms of groups of features rather than individual tokens or features. However, AutoGnothi [1] already achieves faithful interpretability without compromising performance. It is therefore important for the authors to provide a deeper discussion of related work and offer a more thorough comparison of the results.", "essential_references_not_discussed": "[1] Wang S, Tang H, Wang M, et al. Gnothi Seauton: Empowering Faithful Self-Interpretability in Black-Box Models[J]. ICLR, 2025.", "other_strengths_and_weaknesses": "1, Computational Cost: Generating and evaluating groups for large datasets (e.g., ImageNet) can be computationally expensive.\n2, Out-of-Distribution Data: The binarization of groups may create out-of-distribution data, although the paper argues that modern Transformers are robust to this.\n3. Limited Human Evaluation: The human distinction task is conducted on a small subset of images (10 examples), which may not be representative.", "other_comments_or_suggestions": "See above comments.", "questions_for_authors": "See above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zys1WQRzpC", "reviewer": "Reviewer_cf6T", "summary": "This paper proposes a method for extending models to serve as self attributing neural networks (SANNs). The novelty is that it uses a group-based SANN where the grouping itself is learned end-to-end. SANNs are useful in that they provide an added explainable layer on top of black-box models. The paper's theoretical section shows that per-features SANN have a lower bound on the error that group-based SANN do not have. The architecture of the model is made of 3 parts: 1) a group predictor (selecting subsets of the input into groups and masking inputs), 2) a backbone model (pre-trained and frozen) applied to each masked subsets, 3) a scoring module that combines the backbone output for each group into a prediction via sparse cross-attention. The empirical section aims to show both that the method keeps the predictive power and that it provides semantically meaningful groups. On the performance, it is compared to other SANN SOTA approaches on 3 different datasets and shows it is superior to them. On the semantics of the grouping, three assessments are performed: 1) coherence (IoU and threshold-based purity are used to measure coherence and show superior behavior); 2) class-distinctiveness (this is evaluated by humans by checking if the class can be predicted by looking at the grouping); 3) semantic usefulness (do the groupings help in 2 tasks: model debugging - where model's behaviours in grouping more or less objects can be predictive of an over-reliance on background objects; cosmology-discovery assistance - where the method is shown to help understand how cosmic density is correlated to space voids).\n\n## update after rebuttal:\nI have considered the author's rebuttals of all reviews. I believe that most issue have been addressed. Therefore, I raise my rating one notch.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We respond to your questions as follows:\n\n## Backing up the Claim that SOP’s Groups are Superior\nWe would like to clarify a misunderstanding of the experiments: In referring to semantic utility, we believe the reviewer is confusing the claims on group quality with the wrong set of experiments.\n\nSpecifically, **experiments showing that SOP’s groups are semantically superior are in Section 4.3,** which validate the semantic coherence of the groups. These experiments measure the semantic meaning of the groups, and evaluate all baselines on all datasets, including cosmology.\n\nWe believe the reviewer is misinterpreting Section 4.4 as an evaluation of the groups. Instead, **Section 4.4 highlights how these semantic groups can be utilized in downstream tasks, not to measure groups**. In practice, practitioners will want to use and interpret the best performing model and have an easier time interpreting results that are more semantically aligned, which is SOP as validated respectively in Sections 4.2 and 4.3. For example, our cosmologist collaborators are only interested in interpreting models that can predict the ground truth parameters with low error to obtain trustworthy insights for scientific discovery. \n\n\n## Clarification for Experiments Metrics\n- *Error Metrics.* To our knowledge, **our error metrics are within standard practice in the literature for measuring the performance in these tasks.** ImageNet is almost universally evaluated with top-1 error/accuracy. Cosmogrid uses mean-squared error as it is a regression problem, and is also consistent with prior work. Reporting F1/AUC is not common in these settings. MultiRC can be evaluated with either accuracy or F1 score, and many before us have used accuracy without a significant difference. We reported accuracy to be consistent with ImageNet, but can easily change this to F1 (the results and findings are the same).\n\n- *Exact formulation of IoU-based Semantic Utility.* **At the end of RQ5, we direct the reader to Appendix C.4.1 (line 1550) for detailed descriptions including the exact formulation for IOU, how it is calculated, and the annotations used,** which are object segmentations and human-annotated explanations. Semantic groups that align with objects in images or structures in mass maps are easier for humans to interpret in downstream analysis, as in RQ7/RQ8.\n\n\n## Significance of Theoretical Claims\nWhile the idea that grouping is more expressive than per-feature methods may seem intuitive, this has remained a heuristic with no formal proof.  **Our work provides the first formal mathematical proof of this key property**. Furthermore, our theorem provides two key insights that go beyond this intuition. **First, our theorem proves that SANNs require only a finite number of groups**, contrasting other theoretical DL results that have classically assumed infinite width layers to achieve good performance. Second, our theory proves that it is **impossible for per-feature SANNs to achieve good performance.** Until now, it was not known whether per-feature SANNs could be improved with more data, larger models, or better training. \n\n## Why do some baselines have high error? \nIn our experimental setup, **we carefully controlled for all baselines to have equal compute costs and equal sparsity when measuring accuracy**. This is critical in order to attribute better performance to the SANN approach rather than more computation or more features. In Appendix C.3 we describe the common setup: all methods use at most 20 forward passes with 80% sparsity (keeping 20% features). The high error rates arise from baselines failing to be efficient with respect to compute or sparsity. RISE-F typically requires thousands of random masks to be effective, which is impractical for ImageNet inference. The performance of XDNN and BCos-F is due to their design depending on having all the features (0% sparsity) to make accurate predictions.\n\n## How is the number of groups controlled?\n**The maximum number of groups is a hyperparameter that can be selected based on the user’s available computational resources.** We analyzed how different maximum numbers of groups affect performance on ImageNet in Figure 1 in https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf . We see that **performance increases with more groups until it plateaus at 5 groups**, indicating that our model can potentially be 4x more efficient!\n\n## Request for pseudo-code and code repository\n**The requested pseudo-code and code repository are already in the submission**. The pseudo-code is in Algorithm 1 at the top of Appendix B (page 24, line 1265-1279). The code repository is included in the supplementary material.\n\n\n## Other comments: \n- We will add the backbone performance to Table 1, which is 0.097 error for ImageNet, 0.00869 MSE for Cosmogrid, and 0.318 error for MultiRC. \n- We will add a table of contents to add structure and clarity to the Appendix.", "claims_and_evidence": "The claim that the method achieves better predictive performance compared to other SANN method is well supported empirically. However, the claim that the grouping provided by the method are superior to other methods is not entirely supported, as the semantic utility is not compared to other method for the cosmology experiment and has similar behaviour than other methods for the model debugging experiment.", "methods_and_evaluation_criteria": "The evaluation criteria for accuracy are 'error' and it is not specified how it is calculated. More standard measures such as AUC and F1-score would be more meaningful. For semantic utility, IoU is used, but it is not well explained how it is calculated (using what type of annotations) and intuitively why the grouping would be more useful when intersecting with those annotations.", "theoretical_claims": "The theoretical claims mostly formally proves that grouping allows for more expressibility than per-feature methods, which is not really new in my opinion and not quite center with respect to the novelty of the paper, which is more the end-to-end learnable grouping with model agnostic backbone.", "experimental_designs_or_analyses": "The experimental design is fine and covers both accuracy preservation and semantic utility. I wish the the accuracy of the backbone itself would have been provided in the main table (1) in order to evaluate the accuracy degradation for prediction. Also some of the reported error for the baseline seem suspiciously high (> 0.7 !) for some baselines on the ImageNet task (those numbers are obtained by the authors, not lifted from the published reference).", "relation_to_broader_scientific_literature": "It seems the paper does refer to the related literature adequately, although i'm not super familiar with this field.", "essential_references_not_discussed": "It appears that the related works are properly referenced, see box above.", "other_strengths_and_weaknesses": "pros:\n- a novel end-to-end method for SANNs that learns the grouping itself via masking.\n- an interesting application to cosmological discovery where the method confirms that voids are more relevant to estimation of space density that denser clusters.\n\ncons:\n- it is unclear how the number of groups is controlled. Is it a fixed number? If yes, it should be studied how its value affects performance.\n- the model and overall system is relatively complex and would benefit from pseudo-code algorithms and especially a code repository, none are given.\n- given that the appendix itself has 27 (!) pages, a table of content would have been nice. Furthermore, 27 pages is too many. Please only keep what is absolutely necessary to understand the paper (first 8 pages).\n- There are several typos and failed references left, please proof-read again.", "other_comments_or_suggestions": "typos:\n\n089-right: anddifferent\n\n179-left: double 'in contrast'\n\n201-right: unfinished sentence after interpretability.\n\n393-left col: unclear explanation for (1) - missing a 'respectively'\n\n394-right: (c) and (c)  -> (c) and (d)\n\n1337: however + on-the-other-hand\n\n1365: missing words after incur\n\n1104: bad reference", "questions_for_authors": "see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rHJjF7zKp", "reviewer": "Reviewer_9FSB", "summary": "This paper proposes a new approach to producing self-explainable (attributing) NNs. It does so after identifying limitations of previous approaches. The authors thereupon introduce a group-based SANN approach. They evaluate on several datasets and investigate multiple aspects of the approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! In the following response, we clarify the posed questions and commit to adjustments as needed.\n\n## Experimental Setup Clarifications\n- *What features are groups based on?*  Groups are simply subsets of the input features (the $x_{G_i}$ from Definition 1). A SANN takes such groups and embeds them in a latent space (the $h(x_{G_i})$ from Definition 1) with a neural network to make predictions. In our experiments, $x_{G_i}$ manifests as subsets of **image patches and text tokens (line 261-262)**. We will make this more explicit by adding the following statement to Definition 1: “We refer to $x_{G_i}$, the subset of input features of $x$ corresponding to the subset $G_i$ as a group of features of $x$.”\n- *Datasets and Metrics.*  **On line 256-262, we gave the general categorizations of the tasks (image classification for ImageNet, image regression for CosmoGrid, and text classification for MultiRC), with further details in Appendix C**. To summarize: The classification tasks of ImageNet/MultiRC are to predict the right object in the image or answer the multiple choice question (Appendix C.1.1 and C.1.3). The regression task for CosmoGrid is to predict cosmological parameters of the universe from telescope images (Appendix C.1.2), where the MSE (Mean Square Error) measures how much the predicted cosmological parameters differ from true values (lower is closer and thus better).\n\n## SOP for Model Debugging (RQ7)\nThe goal of debugging is to identify a “bug”, or the reason behind errors. Here, we find that when SOP is correct, it tends to be using more of the object, whereas when SOP is incorrect, it tends to use more of the background. This analysis shows that one potential bug behind the errors of SOP is when the model looks at the background as an inaccurate proxy for the object. **This finding is only reliable because of SOP’s (and other SANNs) faithfulness guarantee–since predictions depend solely on the selected feature groups**. Without the faithfulness guarantee, one then cannot be confident if the prediction depends on the explanation. Since SOP has more semantic objects and backgrounds in its groups (from RQ5), the analysis is applicable to more SOP predictions than other SANNs.\n\n## Clarifying the Cosmology Study (RQ8)\n**The mapping between Section 4.3 and RQ8 is correct.** Specifically, RQ8 asks a scientific question on the dependency of network predictions on cosmological structures. However, **analyzing these cosmological structures with SOP only makes sense if the SOP groups contain such cosmological structures to begin with. This fact was established previously in Section 4.3 (RQ5)**, which measured the semantic coherency of SOP groups in various settings including cosmological structures in CosmoGrid, the setting of RQ8. That is why RQ8 references Section 4.3. We will update the reference to refer specifically to RQ5.\n\nIn the final conclusion, we called the result “meaningful” as the conclusion was understandable to experts, and “scientific discovery” as these are relations that experts want to understand but currently do not. **We did not intend for this to be a final irrefutable claim: rather, it is an initial step into the unknown that allows follow-up work to independently confirm or investigate more deeply.** We will adjust the wording to reflect this. These claims were made jointly with expert cosmologist collaborators, who helped us write background, justification, and conclusions of RQ8.\n\n## Relation to Object Factorization\nAt a high level, both SOP and methods such as slot attention use attention mechanisms to group input features without direct group supervision. **However, the goals (and corresponding design decisions) are quite different.**\n\nSOP forms feature groups that are beneficial for the end-to-end prediction task, without trying to reconstruct the original input. In contrast, techniques like slot attention try to break an input into its parts by learning groups that can reconstruct the input. Slot attention forces competition between groups to contain disjoint components and get coverage over the entire input, whereas SOP allows for groups to both overlap or not cover irrelevant features. Overlapping information (e.g. in Figure 1) can benefit prediction, i.e. when objects have multiple relevant contexts/groups in the image. We will add this discussion to the related work.\n\n## Other Comments\n- We will correct the grammatical error in Line 201, and enlarge Figure 3 (a larger version can be found in Figure 4 in https://github.com/icml2025-3311/icml2025-3311/blob/main/ICML2025_3311_rebuttal.pdf ).\n- For line 326, we meant for this to refer to human understandability being improved if the explanations are semantic, and will adjust accordingly.\n- Lastly, we can certainly swap RQs to be fully in the appendix and enrich the results for other RQs in the main paper. If you can let us know which RQs should be moved, we can adjust accordingly.", "claims_and_evidence": "I think the claims are quite well supported by the experimental results, except for some which I have commented on in the other boxes.", "methods_and_evaluation_criteria": "I have also read through the relevant parts in the supplements, but I still don't understand what the groups are based on. The authors always mention ..., but I don't understand whether these are latent features or simply partitions of the input features (e.g., image patches). Could the authors clarify this here, but also make this more explicit in the main text?", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "There are things I can't seem to find in the experimental section. How exactly or for which task do the authors evaluate the models e.g. for RQ1? I.e. what is the task for these datasets and what does the metric MSE tell us for this task? Could the authors give more details in terms of metrics for the evaluation section?", "relation_to_broader_scientific_literature": "I think the work relates well to prior work both on the topics of explainability and SENNs in particular.", "essential_references_not_discussed": "I was wondering how this work relates to topics such as object factorization via approaches such as slot attention as the general idea seems quite similar and I think it would be valuable to discuss this in the context of the related works. Somewhat older, but Important references (there are most likely more novel approaches to search for):\n\nFrancesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, Thomas Kipf: Object-Centric Learning with Slot Attention. NeurIPS 2020\n\nGautam Singh, Yeongbin Kim, Sungjin Ahn: Neural Systematic Binder. ICLR 2023\n\nThis is a little ol, but might have additional references: Klaus Greff, Sjoerd van Steenkiste, Jürgen Schmidhuber: On the Binding Problem in Artificial Neural Networks. CoRR abs/2012.05208 (2020)", "other_strengths_and_weaknesses": "I find the paper is well written and structured. The general idea seems interesting and novel. \n\nI don't know where to write this in ICML's new template, so I'll write it here. I think the method is very cool and I actually lean towards accept, however I then noticed several things that are not optimal yet in the paper (see all of my questions and comments). And I think these need to be tackled for readers to properly follow the authors argumentations and claims. If the authors can comment and apply the suggestions throughout all of the boxes I would be very happy to raise my score to an accept.\n\nSome weaknesses:\n\nI find it bad to have several important, main results in the appendix. E.g., for RQ3, RQ7. In this case I would rather reduce the number of experiments in the main paper (the authors have done quite a lot, maybe move these completely to the appendix) and fill the free space with missing important results.\n\nI am not sure about the correctness of RQ7's evaluations and what they tell us. Could the authors elaborate on this. How does it help to identify that 5% more often for correct samples the model is focussing on object partitions for stating \"Thus, explanations from SOP and other SANNs can help illuminate the reasoning behind model behaviors.\"\n\nThere seems to be a wrong mapping to sections in line 381: \"As validated in Section 4.3,\" in the context of RQ8, what does section 4.3 have to do with RQ8?\n\nI also find the conclusion in line 403 a little strong given the experimental evidence. Perhaps the authors could tone this down a little or argue for why they can make this statement from the one experiment.", "other_comments_or_suggestions": "In line 201 the phrase \"while the sparse number of groups ensures that the human interpretability.\" doesn't make sense grammatically.\n\nFigure 3 is quite small to identify individual differences. Could the authors try enlargening this?\n\nIn line 326 the authors state \"Explanations need to be semantically coherent such as relating to object segments or human-understandable concepts\". I don't think this is correct. Explanations per see don't need to be this. We would hope that they do so so they are aligned with our own human knowledge, but in unsupervised learning in terms of explanations there is not guarantee that this is the case.", "questions_for_authors": "What is the intuition why in RQ5 SOP does learn explanations that are more coherent with object segmentations? It is not trained so, so where does the influence come from?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose Sum-of-Parts (SOP)  framework for SANN. The reviewers find the work solid in both theoretical and empirical results.  The concerns from the reviewers are properly addressed.
Decision: Accept (poster)