Improving Explanation Reliability through Group Attribution
Authors: Sangmin Lee
Jongseong Jang
Woohyung Lim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainable AI
Attribution methods
Group attribution
Attribution reliability

Abstract: Although input attribution methods are mainstream in understanding predictions of DNNs for straightforward interpretations, the non-linearity of DNNs often makes the attributed scores unreliable in explaining a given prediction, deteriorating the faithfulness of the explanation. 
However, the challenge could be mitigated by attributing scores to groups of explanatory components instead of the individuals, termed group attribution. While a group attribution would explain the group-wise contribution more reliably, it does not explain the component-wise contributions so that estimating component-wise scores yields less reliable explanation, indicating the trade-off of group attributions.
In this work, we introduce the generalized definition of reliability loss and group attribution, and formulate the optimization problem of the trade-off with these terms. We apply our formalization to Shapley value attribution to propose the optimization method G-SHAP. We show the effectiveness and explanatory benefits of our method through empirical results on image classification tasks.

Under review as a conference paper at ICLR 2023
IMPROVING EXPLANATION RELIABILITY THROUGH GROUP ATTRIBUTION
Anonymous authors Paper under double-blind review
ABSTRACT
Although input attribution methods are mainstream in understanding predictions of DNNs for straightforward interpretations, the non-linearity of DNNs often makes the attributed scores unreliable in explaining a given prediction, deterio- rating the faithfulness of the explanation. However, the challenge could be miti- gated by explaining groups of explanatory components rather than the individuals, interaction among the components be reduced through appropriate group- ing. Nevertheless, a group attribution does not explain the component-wise con- attribution becomes less reliable than
tributions that its component-interpree the original component attribution, indicating the trade-off of dual reliabilities. In this work, we first introduce the generalized definition of reliability loss and group attribution to formulate the optimization problem of the reliability trade- off. Then we specify our formalization to Shapley value attribution and propose the optimization method G-SHAP. Finally, we show the explanatory benefits of our method through experiments on image classífication tasks.
INTRODUCTION
The advance in deep neural networks facilitates a training model to learn high-level semantic features in variety of fields, but intrinsic difficulties explaining predictions of DNNs become primary barrier to real-world applications, especially for domains requiring trustful reasoning for model predictions.
While various approaches have been proposed to tackle the challenge, which includes deriving global behavior or knowledge of a trained model (Kim et al.) 2018). explaining the semantics of target neuron in model, (Ghorbani et al., 2019] [Simonyan et al. 2013 Szegedy et al., 2015). introducing self-interrretaable models (Zhang et al.) 2018] Dosovitskiy et al. 2020 Touvron et al.)
2020 Arik Prister 2019), input-attribution methods became the mainstream of post-hoc explana- tion methods since they explain a model prediction by assigning scalar score to each explanatory component (feature) of its input data, yielding the straightforward explanation for end-users through data-corresponded visualization such as a heatmap.
However, since each explanatory component explained with single scalar score, the nonlinearity in DNNs makes their scores ancy between the explained and actual model behavior for a prediction, deteriorating the faithfulness of the explanation.
less reliable in explaining a model's prediction. It results in the discrep-
As it is the inherent challenge of input attribution methods, the problem has been studied and tackled with various approaches and perspectives: (Grabisch & Roubens 1999) formalizes the axiomatic interactions for cooperative games, (Tsang et al.] [01 8) explains the statistical interaction between input features from learned weights in DNN, (Kumar et al. (2021) introduces Shapley Residuals to quantify the uncxplained contribution of Shapley values, (Janizek et al.) 2021) extends Integrated Gradients (Sundararajan al. 2017) to Integrated Hessians to explain the interaction between input
features.
While these approaches have improved the explainability to the DNN's nonlinearity, their explain- ing scores are not corresponded to each explanatory components in many cases, reducing the inter- pretability of explanations.

---

Under review as a conference paper at ICLR 2023
Explaming (gg - o(Aeg 4kg) (0.0.0)
Componett atrittioon
Group atributoon
Xa
Exy)
2,00
1.00
2,00
ná-izeo
1.66
C.66
0.00
0.915
nel-koos
ounn
0.00
0,00
Comporentineepeeee bullen
X2
score
2,00
12
rel-
0.00
0.00
0.00
1.24
Figure I: Trade-off of the dual reliability loss of a group attribution for a simple non-linear function. Grouping 21, 22 resolves their interaction that it reduces the reliability loss of the group {x1, 42 but increases those component-interpreeed scores. Here attribution score and its reliability loss are defined as the input gradient and expected L2 error of its tangent approximation, which are 0 and Ex--N(,.))(((xx + te,) - f(x) - =1. respectively.
du: fx)
Instead, it can be alleviated by explaining model's prediction in terms of groups explanatory com- ponents rather than the individuals, termed group attribution. Appropriate grouping can weaken the interaction among the components, yielding more reliable explanation.
However, a group attribution does not attribute scores to the individual components so that interpret- ing a group attribution in terms of the individual components results in less reliable explanation than the original component attribution.
Therefore, both group-wise and component-interpretd attribution reliability should be considered for deriving group attribution, implying trade-off optimization problem. Figure illustrates this problem with simple a non- function.
In this paper, present work as follows: In Section 2, we introduce the generalized definition of reliability loss and group attribution to formulate the optimization problem of the reliability trade- off. In section we integrate our formalization with Shapley value attribution (Lundberg & Lee) 2017) and propose the grouping algorithm G-SHAP. We choose the Shapley value as our scoring policy for two reasons: 1) it has been utilized popular attribution method for its model-agnostic characteristic and well-founded axiomatic properties. 2) it becomes less reliable when there are strong interactions among the explanatory component's 's contribution, as it take the aggregation of the contributions over all coalition states. In section 4, we show the explanatory benefits of our method through experiments on image classification tasks as follows: 1) we verify the grouping effect of G-SHAP through quantitative and visual analysis. 2) we validate our grouping approach by comparing it with several baseline grouping method which would yield the similar grouping result to ours. 3) we show the improvement in local explainability of prediction through the estimation
game, which utilizes the deletion game (Petsiuk et al. 2018 Wagner et al. 2019) to measure the error of model output changes.
Our contributions are summarized as follows:
We introduce two novel concepts to improve the limited reliability of input attribution meth- ods: reliability loss that quantifies the discrepancy between the explained and the actual model behavior for prediction, group attribution that explains prediction in terms of groups of explanatory components. Since a group attribution becomes less reliable in ex- plaining component-wise contributions, we formulate the optimization problem to resolve the reliability trade-off. While we choose the Shapley value formulation consists of generalized terms, applicable for other input attribution methods.
scoring policy,
2. We propose G-SHAP, a grouping algorithm for Shapley value attribution. We empirically show that G-SHAP has better local explainability of a model prediction than SHAP. We also validate the effectiveness of our grouping approach by comparing with several baseline grouping methods, which would yield the similar grouping results to ours.
2

---

Under review as a conference paper at ICLR 2023
2 GENERAL FORMALIZATION FOR RELIABILITY LOSS AND GROUP ATTRIBUTION
2.1 RELIABILITY LOSS OF AN ATTRIBUTION
Let y² = f(x") be a model prediction to explain, where x" (.f:-.,,I5) xx) E RR and f - R the input data and the model function, respectively. Let ④b be attributing (scoring) function that takes a model function / and a target input x" and returns the attribution scores Φb E IR As consider f and x* is fixed, we introduce the translated model function f*(x f(x+ + +*) to simplify our henceforth definitions. Since explaining f(x) at x = x" is equivalent to explaining * (x) at x 0, we have
Φ = (61,..,, (f,=)))))))=)))===--())(_)))>>)) (f E
(1)
Similarly, let E be function that quantifies the reliability loss in explaining the prediction with an arbitrary attribution (#1...., aN) E I as below.
E(a) = = .a) 20 0
(2)
where the lower value implies the more reliable attribution in explaining the prediction. We note that a = (a1,..,yy) aN) can be arbitrary, not necessarily Ø
2.2 GROUP ATTRIBUTION AND ITS RELIABILITY LOSS
A group attribution attributes a score to each group of explanatory components, where the compo- nents within each group treated
shared variable. Formally, let G [G1,..., Gm } be grouping (partition) of the component set [x1,..., 'N Then the group-mapped function fö assigns each group-variable 9: its corresponding components variables of X, defined as
(g). .....99 f (9) ... .o(())))))))))))))))))))()))
where a is the group map such that x; E Go(i) for each 1 < < N. For example, the group-mapped function of of f(a, b, c) with the grouping G = {fa, b). {c}} is fa 92) = f(gı, 91. 92).
Once have a grouping G. its group attribution dc is defined the attribution scores of få. which is
ΦG = (#a .... )GG)) + (e 0) RM
(4)
By definition, each group score ΦG; indicates the co-contribution of their components 2a Gj- Note that it is not necessarily equal to the sum of the component scores in general.
Similarly, we can derive the reliability loss of group attribution &G as below,
{(ØG) E(fã.
(5)
which tells that how reliable in explaining the prediction with the group attribution. From the def- inition, we can say a group attribution Φc is more reliable than the component attribution &p {(PG) < {()) and less reliable if {(pG) > {(Φ).
2.3 COMPONENT-INTERPRETTOO
OF A GROUP ATTRIBUTION AND ITS RELIABILITY LOSS
As discussed in the introduction, group attribution does not attribute scores to their belonging com- ponents that its component-ing inntepeettiterrpeteed scores would have larger reliability loss than the original component attribution. To address this, we need to first formalize the score-interpreting function which interprets group attribution &G in terms of the individual components, denoted with the tilde as below.
6c = (01, (o...... ØN) = (($G) E N
It is notable that the interpreting policy can vary depending the component score's semantics but must not utilize any information of the prediction. For example, defining ((0G) = & is not acceptable. Consequently, the component-wise reliability loss of a group attribution ΦG is given as below.
E(Φa) [("", .G))
3

---

Under review as a conference paper at ICLR 2023
2.4 FORMULATING THE OPTIMIZATION PROBLEM OF THE RELIABILITY ORADE-OFF
In order to formalize the trade-off of dual reliability of a group attribution, we first normalize the improvement and deterioration of the reliability losses.
As the reliability loss of a group attribution {(ØG) is expected to be lower than the that of the original component attribution {(Φ), we consider {(Φ) as the baseline and define the normalized score for group attribution's 's reliability (NGR) G as the ratio of the improved amount to the baseline, given as
((G) = {(Φ) {(OG)
(8)
{(Φ)
follows that higher 9 is better: it becomes if {(OG) - 0 (maximum improvement) and if {(éG) E(Φ) (no improvement). It can be negative if the grouping is ill-chosen.
On the other hand, the component-interpreted reliability loss of a group attribution {(GG) is ex- pected to be higher than the original {(Φ). Since the most uninformative grouping Gail that merges all components into one group i.e., Gaul = {G1} = {f21..., 2y} }} is expected to have the largest component-imtepprteed component-interpreted reliability (NCR) as the ratio of the less-deteriorated (saved) amount to Gan to the gap, given as
component interpreted reliability loss, define the normalized score for group attribution's 's
C(G) = 5(0Gm) - E(Φc)
(9)
E( {(Φ)
also follows that higher is better: it becomes (if(( = {(Φ) (no deterioration), (f E(ØG... (deteriorated as Gai).
Since there two singular cases that should be avoided for searching the grouping, which singleton grouping (no-grouping) and the all all-grouping Gall As their (G.C) scores are (0,1) 1) and (1, O), respectively, define the optimization objective L the geometrie mean of two scores, given as
[
+ + -
(C(G) +
++3
((G) max
1+E -0
max
1+E .0
(10)
where e 2 0 is the tolerance hyperparameter for dealing with negative G and C values and B E [-1, 1/2] the balancing hyperparameter that positive weighs more than and vice versa. It follows that larger C implies better group attribution.
3 APPLICATION TO SHAPLEY VALUE
3.1 SHAPLEY VALUE AND ITS RELIABILITY LOSS
Shapley value has originated from cooperative game theory, indicating the fair division of given reward to each player. It has been utilized as the axiomatic attribution method for post-hoc model explanations (Lundberg & Lee, 2017), where the players and the reward corresponded to the binary explanatory components and the output difference of the model prediction, respectively.
Formally, let Z {.,,.., 2N be the set of binary explanatory components and 2 13 be the set of the all possible coalition states z = (Z), 2N), where each z; = 1, indicates whether zi is involved in the coalition not, respectively. Once model function Z - R is given, contribution of 2 at a coalition state z E Z is defined as
(2) = f(z
f(z)
(11)
where the notation 2 means z assigned with =; = 1. Since (z) is trivially zero when z, = 1. we restrict the domain of hú to Z == {z Z = 0).
Then Shapley value of 24 is given as the weighted sum of contributions h4 at all possible coalition states, which is
k!(N k 1)!
Φi =
(x)) WN (k)
(12)
N!
ZE Z
4

---

Under review as a conference paper at ICLR 2023
where |z] is termed coalition size, the number of 1s in
Since follows that Eeez. "x([z)) = 1, a Shapley value Φ, can be considered as the expected value h; by regarding the weights wN (|z]) as the probability, i... di = Ez [b,]. This perspec- tive of defining Shapley values naturally leads to measure the expected L2 error of the contributions, given as
(a:)) = Ez,,ol(h; a;)
Σ
WN (12))((((2) - -a;)2
(13)
2E2:=0
which is named Shapley error of attributing 2j with score dj. Now we define the reliability loss of Shapley attribution
(aaa) Σ (a.)
(14)
......... QN) and (ai,... a.). Similar to the property of ordinary mean and variance, follows that (?(a,) = (o.) + (a: - Φ.)² so that the reliability loss consequently satisfies
where Φ (#1....,
(a) = (0) + 2
(15)
which implies that the reliability loss minimized to (o) when a Φ, showing the optimality of Shapley value attribution.
3.2 SHAPLEY GROUP ATTRIBUTION
Let G = [G1. G2, .... GMr} }o be partition (grouping) of the component set Z = {=....., NN } with non-empty groups. Then group-wise coalition state E under the grouping G is restricted the cases that components in each group Gj are all involved or not, denoted as z G[ 1, G 0. respectively. Since each group G, has binary involvement state, the coalition state has M degree of freedom and can be represented as a M-dimensional binary vector.
First, contribution of G coalition state z E Z defined as the output difference of f by switching all 2; G2 to 1, given
ha, (z) = f(zc,=1) f(z)
(16)
where the notation 2G,=1 means that is assigned with = 1 for all z; E Gj. Similar to h4, we consider the domain of lig, as Zc,=o. defined as z E satisfying z(G,] 0 and z z/,,.. {0, for all 1 < m + j < M. Consequently, the Shapley value and error of a group Gj defined as
Φc, = Eec
Eτ2 (ac,) = Ezay-ol(he, ag,)
(17)
is notable that the expectation operation Ezc,= is not compatible with the component-wise case Ez,-o since the dimension of coalition states chagned from N to M, which implies that Shapley value of group does not equal to the sum of its components" Shapley values.
3.3 G-SHAP: ALGORITHM FOR SHAPLEY GROUP ATTRIBUTION
Since the number of groups M vary from [1, NV. it is difficult to evaluate the group-wiss sooup-wise Shapley terms from component-win wise terms. Moreover, group-wise coalition states depend on not only the number of groups but also the grouping itself. It implies that the optimization problem is more challenging than the set-partitioning problem, where the target value of each subset fixed.
However, as (Guanchu 2022 has shown the effectiveness, Shapley statistics be approximated by excluding the components or groups which have little effect on the target. Despite the incompat- ibility of Shapley weights, the weight decomposition property w (k) ww (k) wx(k 1)
5

---

Under review as a conference paper at ICLR 2023
In all ue peen Beidd
x= (e" of +f, 41
Necgedi (1+ rig
Riw ((4 [ (2
Indggnor Inaeo.M.ck
@ragiig progrss
Eachskg groupin
Illustration of the G-SHAP algorithm with corresponding NGR-NCR graph: (a) initial stage: grouping starts with the singleton grouping (equal to component-atrrbutton)) where (G, (0, 1), (b) an arbitrary middle step, (c) the very next step: the optimal grouping of the core subset K applied (d) last step: remaining number of group is less than the core set size. Dotted curve indicates contour line of the objective L and star mark indicates the best Shapley group attribution which G-SHAP finally returns.
Figure 2:
suggests that di and E can be decomposed with 2, cy-condito contributions as
N-
uN (U)he(Zz==)) + w (x((()))
==|=|
(18)
N
Σ
WN (1)(h) oi)² wN(1 ()14(())=)) 02)²
It implies that hz(2z=o), h; z (2(2) then #= Pdjj=1 and Eai== Eilj=1 & so that the appropriate exclusion would yield more accurate approximation for Shapley statistics.
where 2j * 2; can be chosen arbitrary.
(0 x;)² the heuristic for the exclusion. Components or groups with top-k e; values considered the core subset K of given grouping G. Once we have K then we observe all binary states 1 of K to derive the optimal grouping of K, where the excluded components groups are fixed. Then apply the optimal grouping and continue the progress until |G| < k. Overall progress is illustrated in the Figure 2.
In our method G-SHAP, we take e; [ x,)²
4 EXPERIMENTAL RESULTS
While there have been existing group or cluster wise explanation methods (Masoomi et al. 2020 Singh et al., 2018) their grouping criteria and objective are different from ours that comparing those explanations with ours would not be resonable. Therefore, as mentioned in the introduction, have focused verifying the explanatory benefits of our group attribution (G-SHAP) through comparison with the corresponding component attribution (SHAP).
4.1 EXPERIMENTAL SETUP
As mentioned in the introduction, we have applied the proposed method on the validation datasets of Flower5 (Mamaev] 2018). MS COCO 201 (multtii-labbel) (Lin et al., 2014). and Pascal vOC 2012(multi-laka) (Everingham et al. 2012) with ImageNet 2usskuskyy et al. (2015) pretrained ResNet-50 (He et al. 2016) model, where Flower5 stands for subset of Flower dataset with distinctive classes (daisy, dandelion, rose, sunflower, tulip). We have fine-tuned the model to each datasets the average prediction accuracy is 94.7%, 82.9%, and 90.3%, respectively. For MS COCO and Pascal VOC models, we have taken logit value of top-1 label as the output.
Since our heuristic of choosing the subset requires conditional Shapley values which have ( terms, we have considered superpixels image as explanatory components. We have ex- perimented with two superpixel methods, quick-shift (Vedaldi & Soatto 2008) and graph-based

---

Under review as a conference paper at ICLR 2023
COME
SIMPN--1 13
9N- -OA0 cutt
-0EN N.D. SI GLCC WI- SUNE
Figure 3: G-SHAP results for the image classification task, taken from MS COCO, Flower5, Pascal VOC dataset, where superpixels chosen graph-baseds graph-based, and quick-shift, for each image, respectively. The 3-5th columns stand for B = 0.25, 0.00 and -0.25, respectively. For each image, heatmap of the upper attribution reliability. Heatmaps are area-normalized ratio to their base values, which are their sum divided by entire area of the image.
indicates the attribution score and the lower row indicates the
Felzenszwalb & Huttenlocher 2004) segmentation method, which existing attribution methods LIME (Ribeiro et al.J (016) and XRAI (Kapishnikov et al., 2019) use. As Shapley value consid- binary input states, have defined the input map mean-color masking function that x = 0, corresponds to the mean-colored and the original superpixel, respectively. We also define the score estimating function ( distributing group according to pixel area, i.e., wy
[()G)), Exyco wy &G for E G. We set core subset dimension = 10, B = 0.0, and e = 0.1 as default.
4.2 OPTIMIZATION EFFECTS OF G-SHAP
We have observed NGR, and NCR verify the improved and saved Shapley of G-SHAP attribution for 3 = -0.25, 0.00, 0.25, stated at Table (left). As NGR, NCR indicates normalized ratio of the amounts, tells that 75% 91% of the baseline E(Φ) are resolved through grouping while 62% ~ 68% of the bound gap E(ōG.) - E(Φ) is saved for = 0.00 case. It has also been observed 3 considerably affect the NGR and NCR scores such that positive weighs NCR much than NGR, whereas the negative B weighs NCR much than NGR, agreeing with our expectation.
Figure 3 shows that the balancing effect can also be verified qualitatively, telling that higher 3 results in higher NCR that the heatmap of G-SHAP is closer to the component attribution (SHAP) but lower NGR that Shapley errors less improved. On the other hand, lower results in the opposite well

---

Under review as a conference paper at ICLR 2023
Figure 4: Comparison results of G-SHAP with various heuristic methods, where the images are taken from COCO, Flower5, and Pascal VOC dataset, and the superpixels are chosen from quick- shift, quick-shift, and graph-based method, respectively.
Superpixels
Datasets
NGR
NCR
NGR
NCR
NGR
-0.25
NCR
Methods
NGR
Quick-Stift
NCR
NGR
Graph-based
NCR
COCO
0.396
0.832
0.799
0.679
0.939
0.667
2-grouping
0.786
0.552
0.818
0.558
Quick-shift
Flower5
0.593
0.831
0.758
0.642
0.919
0.616
3 3'grouping
0.696
0.646
0.754
0.635
VOC
0,602
0.829
0.824
0.670
0.942
0.680
K-means grouping
0.476
0.701
0.493
0.646
COCO
Adjacency
0.443
0.720
0.423
0.711
Graph based
Flower5
0.603
0.779
0.840
0.605
0.950
0.653
G-SHAP groody
0.678
0.584
0.648
0.561
VOC
G-SHAP porposed
Table Reliability scores of the G-SHAP for 3 = 0.25, 0.00 and -0 0.25 (left) and comparison with baseline heuristics, where scores averaged on the datasets (right)
also tells that G-SHAP attribution consists of a few groups with salient superpixels. It implies that our method needs to be compare with baseline grouping methods to validate our approach as the sanity-check, discussed in the later subsection.
4.3 VALIDATING THE GROUPING APPROACH
In order show the validity of our grouping strategy, have compared G-SHAP with various grouping heuristics which would likely yield the similar results, stated in the Table (right) and illustrated in the Figure The details for each grouping heuristics are described below.
First, we have employed 2-grouping and 3-grouping methods as G-SHAP with lower 3 yields few groups. The 2-grouping method sorts the components by Shapley values and splits them into two groups by merging the top 1 < k < N components and the others, and returns the best one. Similarly, the 3-grouping method considers middle-(N- - - -m) components and picks the best one. While the results show that their NGR and NCR are slightly lower than ours (less than 0.1), 1.1), their explanation contains too minimal information since the most intermediate-salien superpixels are neglected.
grouping cases with of top-k, bottom-m, and the
We have also employed K-means grouping and setaacnncy-greede grouping methods to test the grouping performance of merging components with closer attribution scores, the salient superpix- els G-SHAP attribution are often attributed with higher or closer Shapley values. The K-means grouping method clusters of the components into < l 10 groups and returns the best group- ing, where the distance metric is given as the difference of normalized Shapley value (divided by superpixel area). The Adjacency-greedh method iteratively merges two groups with the closest
8

---

Under review as a conference paper at ICLR 2023
min-deletion
max-deletion
random-deletion
Superixel method
Attribution
Flower5
COCO
VOC
Flower5
COCO
VOC
Flower5
COCO
VOC
SHAP
0.476
0.566
0.671
0.444
0.438
0.515
0,446
0,465
0.587
Quick-shift
G-SHAP
0.171
0.158
0.204
0.159
0.144
0.191
0.147
0.131
0.169
SHAP
0.532
0.674
0.786
0.458
0.494
0.563
0.469
0.537
0.646
Graph-based
G-SHAP
0.223
0.176
0.223
0.191
0.161
0.190
0.177
0.149
0.180
Table 2: Estimation game result of component attribution (SHAP) and group attribution (G-SHAP)
normalized Shapley values, and returns the best grouping. As these strategies are expected to retain higher NCR scores, both methods show that NCR is simlar or slightly higher than ours, whereas NGR is clearly lower than ours. It implies that these two heuristics could not resolve group-wise Shapley errors as any interaction statistics are utilized.
In addition, we have employed G-SHAP without core subset searching the ablation study, named G-SHAP greedy, which instead greedily merges two groups which are expected to improve the L the most. As it shows that both NGR and NCR around 0.1 lower than method in average, implying that the optimization problem is challenging to solve with simple greedy approach so that optimal partition searching in the core subset is necessary.
4.4 ESTIMATION GAMES
Deletion game (Petsiuk et al. 2018, Wagner et al. 2019) is the main strategy to assess the attribution scores, which removes each com- ponent of input data in sequence and evaluates the model output drop through AUC of the curve. However, these methods usually rely on the ranking of the attribution scores so that it does not as- sess the reliability of the attribution in general. Therefore, we have employed this idea in different way, termed estimation game, which aims to measure the error of expected output changes to the actual one. As Shapley value indicates the expected model contri- bution, this assessment approach is intuitive to understand has also been utilized in (Guanchu 2022). For the deletion process, we have employed three types of deletions: min min-deletion, max-deletion, and random- -eleetnneee,ion, which deletes (fills mean-color) inputs increas- ing, decreasing, and random order of attribution score, respec- tively. Since model logits can be arbitrarily scaled depending on the prediction, we have normalized as follows: (1) we have linearly rescaled y-axis such that y = 0, stands for ground image (mean- colored image) and target image, respectively. (2) we have also linearly rescaled æ-axis as indicates the ratio of removed pixels to entire pixels. Therefore removal game always starts from (0, 1) and ends with (1, 0), illustrated in the Figure 5. As shown in Table 2, G-SHAP resolves around 60% to 70% of the L.2 estimation error of component attribution (SHAP), providing better understanding of the local behavior of the model.
ood light
Acro nupul
Excected
Contribdoo coor
Norrolzed
Figure 5: Illustration of the estimation game, measuring the error of expected drop to the actual drop
5 CONCLUSION
Though input-atribbttion methods provide clear interpretation as the explanations correspond to the data, non-linearity of deep models intrinsically hinders reliability of attribution. In this work, we have presented novel perspective of quantifying the reliability, attributing groups, and formulating it with the optimization problem. We have chosen Shapley value for the scoring policy to specify the terms and propose the grouping algorithm G-SHAP. We have shown the explanatory benefits of our group attribution multiple perspectives. Its improvement of a group attribution's 's reliability loss is clearly larger than deterioration of conponent-iteerpretee reliability loss, and also improved local explainability of a model's prediction. However, since our method utilizes Shapley conditional terms and search partition spaces with iteration, its computation cost is too high to start with pixel-wise components. Deeper analytical approach and utilizing the prior information of input components would improve the performance and feasibility, left as potential for future works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sercan O Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. arXiv preprint arXiv:1908. 07442, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
M. Everingham. L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.paccal 2012.
The
network.org/chan hwwp:///wwooboolle//////////0//0
Pedro Felzenszwalb and Daniel P Huttenlocher. Efficient graph- bashdbbaeddde image segmentation. Inter- national journal of computer vision, 59(2):167-181, 2004.
Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint arXiv: 190.::3990003199, 2019.
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory, 28(4)5547-565, 1999.
Yu-Neng Guanchu. Accelerating shapley explanation via contributive cooperator selection. In In-
ternational conference on machine learning. PMLR, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. J. Mach. Learn. Res., 22:104-1, 2021.
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry. Xrai: Better attributions through regions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4948 4957, 2019.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learming, pp. 2668-2677. PMLR, 2018.
Indra Kumar, Carlos Scheidegger, Suresh Venkatasubramannin, and Sorelle Friedler. Shapley resid- uals: Quantifying the limits of the shapley value for explanations. Advances in Neural Information Processing Systems, 34, 2021.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir Bourdev, Ross B. Girshick. James Hays, Pietro Perona, Deva Ramanan, Piotr Doll'a a r, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.00 0312, 2014. URL http::://eex....rrte/eeeee//
0312
Scott M Lundberg and Su-In Lec. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Alexander Mamaev. Flowers recognition. http::/www..eggg..cc..m/oooeeeee 2018.
Aria Masoomi, Chieh Wu, Tingting Zhao, Zifeng Wang, Peter Castaldi, and Jennifer Dy. Instance- wise feature grouping. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13374-13386. Curran As- sociates, Inc., 2020. URL wtR..ttttss.:/prceeeeii
cc/ wwwppaper///ecccccccccccccpe.e 202per/2020/f11ee -appaaacooe.tiiiaaaaeeaeereeeepe
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:1806.0111 0066074221, 2018.
10

---

Under review as a conference paper at ICLR 2023
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining. San Francisco, CA, USA, August 13-17, 2016, pp- 1135-1144, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fet ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 115())21112555 2015. doi: 10. 1007/s 1263-015-0816-y.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.600 2013.
Chandan Singh, W James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In International Conference on Learning Representations, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
C. Szegedy, Wei Liu, Yangqing Jia, Sermanet, Reed, D. Anguelov, Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. doi: 00.110/CVPR.2015..225577
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877 2020.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In International Conference on Learning Representations, 2018.
Andrea Vedaldi and Stefano Soatto. Quick shift and kernel methods for mode seeking. European conference on computer vision, pp. 705-718. Springer, 2008
Jorg Wagner, Jan Mathias Kohler, Tobias Gindele, Leon Hetzel, Jakob Thaddaus Wiedemer, and Sven Behnke. Interpretable and fine-grained visual explanations for convolutional neural net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9097-9107, 2019.
Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu. Interpretable convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8827- 8836, 2018


Reviews:
[{"review_id": "vdlQein142A", "reviewer": "Reviewer_BDjb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes to consider a reliability loss term to consider the differences between component-level explanations, group-level explanations and the actual explanations. \n\nBased on this idea, an optimization approach ( G-SHAP) tailored for Shapley values is proposed. \n\nExperiment on the image classification task on 4 datasets show the effectiveness of the proposed method", "summary_of_the_review": "\nThe proposed combination and reliability terms is sound and well motivated. To the best of my knowledge this is novel.\nThe manuscript has a good balance of verbal and formal descriptions. \nThe evaluation covers a good amount of different datasets which could help paint a wider picture of the capabilities of the proposed method.\n\nMy main concerns with the manuscript are the following:\n\nAs admitted in the paper, the proposed method seems to be computationally expensive, reason for which it might not scale to dense inputs, e.g. pixel-level attribution.\n\nIn Sec. 2.3, it is stated \"it is required not to utilize any information of given prediction, or the estimated score would contain unexpected information of the actual component-wise contribution\" \nCould you further elaborate on this statement?\n\nRegarding the grouping policies, is there a principled manner to select the grouping policy? \n\nCurrently the proposed method is only tested on the image classification task. It would strengthen the manuscript if pointers on how the proposed method could be applied to other tasks were provided.\n\nIndependently of its inner-workings, at the end of the day the proposed method is an explanation method. In this regard, it would have strengthen the manuscript if the proposed method was tested under the sanity checks proposed by [Adebayo et al., NeurIPS'18]. This way there could be guarantees on whether the generated heatmaps/attribution maps do constitute valid explanations.\n\nVery related to the previous point, the manuscript effectively show how integrating reliability and group attribution helps getting better attribution maps. However, it is hard to assess at this points where the proposed method is positioned, w.r.t. to the large pool of existing explanation methods.\nHere I wonder if applying the suggested optimization to a state of the art explanation method would further improve its performance, or whether is this just the case for Shapley-based methods..", "strength_and_weaknesses": "Strengths\n- The proposed ideas seem novel to me\n- Varied set of datasets\n- Code to be released\n- An ablation study is present.\n\nWeaknesses\n- High computational costs\n- Focus on a single task\n- Evaluation is not complete.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oOvD6gsHn-B", "reviewer": "Reviewer_ssJm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "With the increasing use of machine learning models, it becomes important to understand their decisions before using them in high-stakes real-world applications. Despite the effectiveness of feature-attribution methods in explaining model decisions, the non-linearity of ML models hampers the reliability of the explanations. In this work, the authors hypothesize that the reliability of feature attribution scores can be improved by attributing scores to groups instead of individual features. In particular, they present a novel perspective of group attribution, propose an optimization problem, and integrate it with Shapley value attribution to propose G-SHAP. Qualitative and quantitative empirical results on image classification tasks show that model predictions can be better understood with groups than with individual components.", "summary_of_the_review": "The paper presents a new perspective to attribute group features but lacks clarity and empirical evidence using state-of-the-art methods.", "strength_and_weaknesses": "**Strengths**\n\n1. A new perspective to attributing the importance of input features using *reliability loss* that indicates the discrepancy between predicted and actual contributions and *group attribution* that attributes scores to a group of explanations instead of the individuals.\n\n2. The proposed G-SHAP outperforms vanilla SHAP feature attribution methods and other baseline attribution methods across three real-world datasets.\n\n**Weaknesses and Open Questions**\n\n1. The paper argues about the usability of the explanations in Section 1 but does not discuss it further in the empirical analysis. Further, is usability directly related to the empirical performance of an explanation?\n2. The definitions of reliability loss and group attribution are agnostic to the choice of explanation method, but the paper limits its application to just Shapeley values.\n3. What is the intuitive interpretation of a group? What does it represent? Is it just a simple grouping of individual attributions, or does it aim to group components with similar attributions?\n4. The paper does not compare the results with state-of-the-art feature attribution methods like GradCAM, Integrated Gradients, LIME, etc.\n5. One main takeaway of the paper is that a model prediction is better understood with groups than individual components. The authors motivate the problem using the non-linearity of the model. In addition, grouping multiple features should intuitively lead to better performance as they may break the correlation of dependent features. It would be great if the authors could comment on this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "kHuueiseEM", "reviewer": "Reviewer_FuXt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a generalizable optimization strategy that codifies a way to trade-off between group attribution and component-wise attributions calculated from explanation methods. The end product is a group attribution version of a given explainer (they propose G-SHAP as an example for groupwise SHAP) that balances the reliability loss from a group attribution compared to component attribution. They also provide formalization for reliability loss (both component-wise and group-wise) with the total loss defined as the geometric mean of the two. ", "summary_of_the_review": "This is a promising line of work overall. But in my opinion is not quite ready for publication. The authors should consider taking their time and improving the motivation, writing and experiments provided in the paper. The paper in its current form will be a disservice to their own work if it gets published as it is.", "strength_and_weaknesses": "Strengths:\n- The paper provides and motivates an important problem. i.e. group attributions are important to capture \"groupings\"/non-linear interactions of features as they contribute to a prediction, something that component-wise attribution methods lack. On the other hand, just providing group attributions can result in in obfuscating contributions of individual components. Hence the trade-off that forms the basis of this paper.\n\nWeaknesses:\n1. The paper is extremely hard to read. Is littered with typos, unfinished sentences and overloaded notation. On average, there's a typo in every other sentence, too many for me to list here. The experiment section specially has too many typos to even count. I strongly advise the authors to read through there paper multiple times and improve its readability and remove typos. Even if the paper has an extraordinary contribution it becomes really hard to feel excited about it if it looks like it has been put together last minute.\n\nsome examples:\n-- right above eq (2) \"and an given scores\" (remove \"an\")\n-- above eq (7) \"an score estimation\" (\"an\" --> \"a\" is redundant)\n-- above eq (11) \"with their Once we have the\" (sentence doesn't make sense)\n-- section 3 1st line \"to each player involved in.\" (unfinished sentence)\n\n2. There are decisions being made here that are not clearly motivated, there's an overall lack of theoretical justification for decisions, though heuristic justifications can be made. For example:\n- under eq(3) the logic behind the conclusion that $a = \\xi$ is the global minimum of eq(3) doesn't make sense. Was this intended to be $a = \\phi$? . Why were equations (2) and (3) formulated the way they are formulated? It'd help if the line of thinking behind these equations is clearly articulated before the equations are presented.\n- Before equation (10), a seemingly arbitrary decision is made about the upperbound for CR when, as the authors say it doesn't have one. What's the reason behind choosing this particular upperbound?\n- below equation (8), \\psi can be arbitrarily defined, how arbitrary? does a \\psi that simply spits out random numbers work? e.g. it's unclear what the equivalent \\psi is in the G-SHAP formulation.\n\n3. The experiment section, in addition to be seemingly very rushed, also seems undercooked in terms of comparisons. There are no comparisons with other group-wise feature selection approaches (e.g. see \"instance-wise feature grouping\" by Masoomi et al. NeurIPS 2020 and the methods they compare against) and even the results compared to baselines don't show much improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "04BJxKR3Cvv", "reviewer": "Reviewer_boPc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a new explanation algorithm called G-SHAP that computes attributions of feature subsets and attributions of individual features within each subset. The motivation to do so is to overcome the problem of interactions between features, thereby grouping features that interact together into the same group. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "+ves: \n\n- A really good idea to automate the computation of group attributions of feature subsets that interact. Also using the G-SHAP heuristic to determine the optimal subset combination is novel. \n\n-ves: \n\n- Empirical results are shown for images alone, it would be good to show these on tabular / text datasets too. \n- The disentangling of group to individual features is less clear from the text and experiments in the sense as to how these differ from actual shapley values of all individual features and how the repeat problem of assigning individual attribution values within each subset does not occur. \n- Its unclear if the shapley axioms hold at group level based on the computed group attributions. \n- It might be good to build experiments where feature interactions/dependence are known a priori and evaluate if the computed subsets match ground truth. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject