The Effective coalitions of Shapley value For Integrated Gradients
Authors: ShuYang Liu
Changjie Fan
Yu Xiong
Meng Wang
Yujing Hu
Tangjie Lv
Zixuan Chen
Runze Wu
Yang Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explanation Shapley value

Abstract: Many methods aim to explain deep neural networks (DNN) by attributing the prediction of DNN to its input features, like Integrated Gradients and Deep Shap, which both have critical baseline problems. Previous studies pursue a perfect but intractable baseline value, which is hard to find and has a very high computational cost, limiting the application range of these baseline methods. In this paper, we propose to find a set of baseline values corresponding to Shapley values which are easier to be found and have a lower computation cost. To solve computation dilemma of Shapley value, we propose Effective Shapley value (ES), a proportional sampling method to well simulate the ratios between the Shapley values of features and then propose Shapley Integrated Gradients (SIG) to combine Integrated Gradients with ES, to achieve a good balance between efficiency and effectiveness. Experiment results show that our ES method can well and stably approximate the ratios between Shapley values, and our SIG method has a much better and more accurate performance than common baseline values with similar computational costs.

Under review as a conference paper at ICLR 2023
THE EFFECTIVE COALITIONS OF SHAPLEY VALUE FOR INTEGRATED GRADIENTS
Anonymous authors Paper under double-blind review
ABSTRACT
Many methods aim to explain deep neural networks (DNN) by attributing the pre- diction of DNN its input features, like Integrated Gradients and Deep Shap. which both have critical baseline problems. Previous studies pursue perfect but intractable baseline value, which is hard to find and has a very high computational cost, limiting the application range of these baseline methods. In this paper, we propose to find a set of baseline values corresponding to Shapley values which are easier to be found and have lower computation cost. To solve computation dilemma of Shapley value, we propose Effective Shapley value (ES), a propor- tional sampling method well simulate the ratios between the Shapley values of features and then propose Shapley Integrated Gradients (SIG) to combine In- tegrated Gradients with ES, achieve good balance between efficiency and effectiveness. Experiment results show that our ES method can well and stably approximate the ratios between Shapley values, and our SIG method has a much better and more accurate performance than common baseline values with similar computational costs.
INTRODUCTION
Deep Learning (DL) has exhibited significant success in various tasks, such computer vision and reinforcement learning. Unfortunately, under the curse of caansparency-pefformann trade-off, it's difficult to understand the intrinsic working logic of DL. Attributing the prediction of deep network to its input features is one of the most popular methods in DI. evaluation domain, such as Deepl.ft ift (Shrikumar al. (2017)), Integrated Gradients (Sundararajan et al, (2017) and Deep Shap (Lundberg & Lee (2017) 7)). All these methods have crucial problem how to choose a perfect baseline as a benchmark for input. As mentioned in Frye et al (2020), the quality of baselines determines the quality of explanations for DL. Ren et all (2021) considers there are two key requirements: (i) baseline values should remove all information represented by origin variable values and (ii) baseline values shouldn't i bring in new/abnormal information.
Some studies provide empirical baseline values based on actual experience. Ancona et all (2019) set baseline values zero, Dabkowski & Gal (2017) set baseline values mean value over many samples and usually people randomly select some samples from datasets. While other studies try to find more reasonable baseline value. Fong Vedaldi (2017 makes baseline values smoothed by blurring the input image with Gaussian noise. Frye et al, (21221 sets the baseline value of pixel with surrounding pixels. Ren et (202T) learns baseline values corresponding to set of features. Those methods try to approximate the perfect baseline value. However, it's difficult to find a baseline value that perfectly satisfies the two principles for various inputs in practice. For example, in the field of computer vision, zero baseline value is a common baseline value, seen as bringing no additive information. But in facial expression code task for Asian people whose eyes are black no longer suitable for zero bascline value, owing to the black area around the eyes bringing additive information. It's ideal to use a transparent image as baseline, which is impossible to achieve in computer. Therefore, though many methods try to find perfect baseline value, most people still use empirical baseline values based on experience, which leads to unsatisfactory and unstable results.
Instead of finding perfect baseline value, propose to find set of informative baseline values, which can be found easier and have much low computation. Shapley value (Shapley (I95T)) is computed as summation of marginal difference for all coalitions and Shapley value can

---

Under review as a conference paper at ICLR 2023
accurately reflect the contributions of features. at (2021) holds the view that Shapley value can explain the difference between prediction and global average prediction. What's 's more, most coalitions in Shapley value are informative and we simply remove some features to get an informative coalition, which means can be found easier. So we propose to find a set of informative baseline values associated with Shapley value.
However, the computation of Shapley value needs to iterate over all combinations, which is exponential with respect to the number of features. For the computation dilemma of Shapley value, we propose a proportional sampling method to approximate the ratios between Shapley value and propose Shapley Integrated Gradients (SIG) to combine Shapley value and Integrated Gradients, achieving a good balance between efficiency and effectiveness. Integrated Gradients has much faster computation process compared with Shapley value but we discover that though Ren et all (2022) has pointed out that Integrated Gradients is simulation of Aumann Shapley value alongside a special calculation path, Integrated Gradients takes a shortcut compared with calculation path of Shapley value and it will lead to an unsatisfactory and unstable explanation.
To verify the effectiveness of our sample methods, we conduct experiments on three typical tasks: human-defined function to verify the validity of our method to simulate ratio between Shapley value of players. facial expression code image classification, to verify that our methods have better performance compared to zero baseline method or mean baseline methods;
Our contributions can be summarized as follows: (I) We discover that Integrated Gradients takes shorteut compared with calculation path of Shapley value; (2) We propose effective proportional sampling method Effective Shapley value to approximate the ratios between Shapley values and design experiments verify the effectiveness and preciseness of our methods; (3) We propose Shapley Integrated Gradients which combines Integrated Gradients with Effective Shapley value and achieve a balance between efficiency and effectiveness.
2 RELATED WORK
Most previous studies focused on finding a perfect baseline value, while our proposed method try to find set of baseline values that are informative and easier to obtain. In our proposed methods, the selection of baseline values is based on the calculation path of Shapley value.
Shapely value. Shapley value (Shapley (1951)) was proposed to distribute contributions to players, assuming that they are collaborating. It's the only distribution with desirable properties, linearity, nullity, symmetry, and efficiency axioms. Aumann & Shapley (RUTS) extended the concept of Shapley value to infinite game. Some previous studies used Shapley value for model explana- tion. Lundberg & Lee (201Z) proposed Shapley Additive ex exllaaations(SHAP) a model explanation method with Shapley value. The SHAP regards the feature player in game, regards model as utility function, and uses chain rule to reduce computational complexity. Based on SHAP, Lundberg et all (2018) continued propose TreeSHAP, method for tree model reducing complexity from O(TL2M) to O(TLD2). It's worth noting that there is also baseline problem in SHAP. To simplify computation, Ghorbani & Zou (2019) used Monte Carlo Sampling and gradient-basee methods to efficiently estimate data Shapley values. Ancona all (201)) sampled from distribution of coalition size of k and then average all these marginal contributions as an approximation of Shapley value.
Ghorbam Zou (2000) proposed new multi-armed bandit algorithm to explain Neuron's Shapley
value.
Integrated Gradients. Integrated Gradients was proposed by Sundararajan all (2U1 to combine implementation invariance of gradients along with the sensitivity of techniques, which also needs crucial baseline value. [Merick& Taly (2020): Kumar et all (2020); Binder et all (206); Shrikumar et all (2017) provided their experiential guidance of selecting baseline values, without providing any theoretical guidance. And Chen et all (202I) regarded baseline values as background distribution, which is similar to our view.
3 PRELIMINARIES
Shapley value. Let us consider a game with n players and F S|S C 2" means all subsets of players. Game will return reward corresponding to the coalition S through utility function v.
2

---

Under review as a conference paper at ICLR 2023
For coalition S and player 2; where 2, & S, the marginal contribution of player x; with coalition is v(S U 24) v(S). Shapley value of player æ, is the summation of coalition E F/i with probabilityy of selecting coalition, follows:
|S
|S| 1)! (0(Suz))-
f(z,)
E
F
SEP/())
Aumann Shapley value. With the expansion to infinite game, Aumann Shapley value was proposed with definition that ds represents infinitely small player game, I represents the complete set of players and t! is perfect sampling with scale t. Aumann Shapley value can be written as follows:
(dss = S. v(t ds) o(tT)dt.
Integrated Gradients. Integrated Gradients was proposed to attribute the prediction of a deep network to input features. It integrates the gradient of the prediction with respect to input features along straight path between input z and baseline x'. Integrated Gradients for model F can be written follows:
f a==
F + olla xx')) do.
IntegratedGrrad (s) (x; x!)
(3)
azi
OUR METHOD
4.1 PROBLEM OF INTEGRATED GRADIENTS
4.1.1
THE CONNECTION BETWEEN INTEGRATED GRADIENTS AND SHAPLEY VALUE
P1
P2
(5- Pa
75.72
combination: X2
Xy Mau Eap
combination: 2318
Figure 1: (a) The calculation paths of Integrated Gradients and Shapley value in two features input. Path P1 is the calculation path of Shapley value while P2 is Integrated Gradients; (b) Orange path is the calculation path of Shapley value while blue path is the calculation path of Integrated Gradients; (c) The calculation path of permutations between x2) and (x 13).
We suppose that there are two dimension features shown in Figure 1(a), baseline sample x' (r1,72) and input sample = (51, 82). Integrated Gradients computes feature's 's contribution alongside path P2 according to Sundararajan et al. (2) Baseline sample x' is considered as not containing any information and all points in path Pg can be regarded as joining the calculation of contribution for each feature. We make the hypothesis that baseline sample x' is considered as
background in game and all points alongside P2 are considered as complete set in game. We can find that Integrated Gradients is the same to Aumann Shapley value under this hypothesis. The same

---

Under review as a conference paper at ICLR 2023
can be extended to n dimensions. Formally, Integrated Gradients is approximation of Aumann Shapley value along Path Ps. The Proof of theorem is shown in Appendix A.I.
Theorem 1: Integrated Gradients is approximation of Aumann Shapley value alongside path that starts from baseline value (T1, F2, ... Ta) and directly connect with end point (S1, S2, Sn).
4.1.2 THE LIMITS OF INTEGRATED GRADIENTS
Integrated Gradients simulates Atumann Shapley value among points in Pz. However, let's consider Shapley value. In Figure I(a), we regard baseline sample x' as Z in Shapley value and S as player and points (ri,r2), (ri, S2), (S1, S2) as coalition. Shapley value of player Sa is defined (ri,
as:
1 f(22) = 2 (v(11, s2) - 2211,r2)) +
2 (e1()), 02) - v(s1, r2)).
When computing Shapley value, we will walk through points (r), (r1, S2), (81, S2). We call points to be walked through when computing Shapley value calculation path of Shapley value. Path P2 is a shortcut for Shapley value, conflicting with path Pi.
Now, let's take closer look at computation on coalitions. As a matter of fact, marginal differ- ence between S U I; and S is made up marginal difference between permutations in S U.; and per- mutations in S. Points in combinations is permutations of combinations and factor |
can be seen as proportion of connections/lines between
24 and S for all connections/points since permutations wouldn't affect utility value due to nature of Shapley value. In Figure I(c), we simulate computation between (x1, 222) and (a1 22; xa). Factor 2T = means lines between (21,22) and (2,, Z2, æ3) take a proportion of j for all lines in the computation of Shapley value for Ty. For simplification, we hide the computation process of permutations, just show computation of combinations and set weight to combination according to the factor.
In conclusion, the calculation path of Shapley value is the weighted points walked through when calculating Shapley value. As shown in Figure (bb), the calculation paths of Shapley value presented with orange dotted lines, and the blue dotted curve represents the calculation path of Integrated Gradients.
4.2 EFFECTIVE SHAPLEY VALUE
For Shapley value of player x4, if we collect all coalitions without H;, we can get accurate Shapley value with the formula of Shapley value, which will bring a large computation. We notice that not all coalitions contribute to Shapley value for player z;. Coalitions that satisfies v(S U xz)) - v(S) = 0 actually don't It influence Shapley value for player z,, We call coalitions satisfy that U() - v(S) # 0 effective coalitions for player x;. So we just need to collect effective samples for player 2; and then can get accurate Shapley value. However, it still needs to iterate over all effective coalitions. still having an extensive computational cost. Considering that in actual cases we are usually concerned about which player/feature has more contribution than others, not the exact Shapley value of each player. We transform our goal from accurate Shapley value to ratio between Shapley value of each player. We can collect proportional coalitions from effective coalitions.
When we classify effective coalitions according to number of players in the coalition, in formula S E N and N is effective coalitions for player I;. we can easily simulate ratio be- tween Shapley value of players through proportional sampling. Let's consider the influence of factor - Fl.
In the analysis of section 3.1, factor represents proportion of the coali- tion's 's permutations in all permutations. In theory, we can directly multiply factor with marginal difference of coalition S. which will be slow if we iterate one by one. For set djk that satisfies v(S U x;) v(S)= j&l[s| k, S E djk, its contribution to Shapley value consists of factor 1S and number of coalitions in dzk- In order to seep up calculation, we hope to reflect influence of factors through numbers of sampling coalitions. Concretely, we set my as coalition set that satisfies v(S 2z)) v(S) j. normalization as normalization function and (T) as sample number for player I;. The sampling number Cýk of set djı is
Fl
my
[k
F|
k
1| [T|
Cjk = normalization!
(5)
Emj
mj
4

---

Under review as a conference paper at ICLR 2023
The overall algorithm of Effective Shapley value (ES) is shown in algorithm 1. We first randomly select partial coalitions A and then classify them by marginal difference v U xz) v(S).S A. recorded M (M,,... mj..... For m,, we then classify by players of coalition |S S E m,, recorded as D = (dij.... d,)...)). Lastly according to equation 1, sample from dzk- Theorem 2 gives the probability of correct results. The Proof of Theorem 2 is shown in Appendix A.2.
Theorem 2: Make ny considered be all combinations without player J. Ej considered to be effective coalitions in ni. by considered to be randomly selected coalitions in njj dj con- sidered be effective coalitions in bj. Omin min(a;) [] that the range of mini- mum of az. The correct probability of proportional sampling for Shapley value of player j is:
P Dk= k=1 II, 9o,,1es c 1 Inyl-1es! leyl
Algorithm Effective Shapley value
INPUT:
player set n, utility function and constant
OUTPUT: baseline coalitions for each player
Randomly select samples from subsets grouped by length for 5; in player set do
A + select effective sample set of player 2;
classify into m, according v0S 24)
=())=i SeA A
for my in M do
Classify according to end for
|S| = k 5 Sem my
Get D (dyl...., djx)
Randomly sample according to equation
end for
Algorithm 2 Shapley Integrated Gradients
INPUT: player set N, utility function and constant T
OUTPUT:
baseline samples for each player
Set m players single player and get new player set M (mi,..,,
for m, in player set M do
Sample effective coalitions D, for m, according to algorithm Set D; as baseline values and set coalition with all players as input
Compute Integrated Gradients over these baseline coalitions
end for
Average all values Integrated Gradients as each player's n e N contribution
4.3 SHAPLEY INTEGRATED GRADIENTS
As matter of fact, for coalitions sampled by algorithm 1, there is need apply Integrated Gradients, just directly computing marginal difference of coalitions and average it as contribution of player x;. However, when there are many players, like pixels in image, usually meaning 244*244 players, it's almost impossible to compute each player's 's sampled coalitions. So we set part of image player like 80*80 area as a player and apply Integrated Gradients compute each player's contribution. Concretely, We set these sampled coalitions as a set of baseline values and set input as origin input. Then we average contributions computed by Integrated Gradients and get each feature's contribution. From point of calculation path, this method starts from some points in calculation path of Shapley value and ends with input. It's worth pointing out that as the area decreases, the result will get closer to ratio between Shapley value. When the area decreases to 1, result will be close to ratio between Shapley value. The problem become that we set sampled coalitions as baseline values and coalition containing all players input. Under the guarantee of theorem 3, The Proof of theorem 3 shown in Appendix A.3, result will get close to ratio of Shapley value. Shapley value verify experiment will prove it.
The overall algorithm Shapley Integrated Gradients(SIG) is shown in algorithm 2. We first set players coalition single player, like 80*80 area as player in image. Then we select partial
5

---

Under review as a conference paper at ICLR 2023
6.50
Loadic Sandina
015
Stradhy
0.55
6:28
3290
3225
2MD
C
d
Figure 2: (a) represents human defined utility function (1): (b) represents human defined utility func- tion (2): (c) represents represents human defined utility function (3); (d) represents human defined utility function (4), Each row represents an independent experiment.
effective coalitions of these players according to algorithm 1. We set these selected coalitions as baseline values and set coalitions with all players input, like origin image to compute Integrated Gradients. Finally, we average these values as each player's contribution.
Theorem 3: Integrate Gradients between effective coalition S and coalition SUx, for player a; can exactly equal marginal difference in Shapley value under the hypothesis that in utility function coalition S directly connect coalition U z.--
While we can also set zero baseline value as the baseline value and set our sampled coalitions input. Interestingly, in ResNet(He.ee.a) all (2016), there isn' 't't much difference between these two methods as shown in Appendix B.I which is opposite in DLN. It's worth pointing out that our SIG computation process is much faster due the batch computation of gradients.

---

Under review as a conference paper at ICLR 2023
Random
Oure
Zemo
Morn
Random
Ous
Zeto
Mean
Figure 3: Integrated Gradients value produced with different baseline values on facial expression coding task. From results, we can obviously find that our SIG method can concisely and accurately locate relevant features.
Random
Ouns
Zem
Mean
Random
Ours
Zero
Meon
Figure 4: Comparison of distribution over positive values and negative values. The distribution of positive values is similar to negative values and our SIG achieves a better result on both distribution of positive and negative values.
EXPERIMENT
To prove the validity of EG, in other words, the simulation of ratio between Shapley value of players, we experiment with our ES method on human defined utility function, which has few features and much less coalitions than actual application. To verify the validity of SIG, we choose Facial expression coding & Image classification tasks, which have 244 features.
Env: (1) Human defined utility function, we define a utility function v with n players; (2) DLNNZhnng et all (202T)) model for Facial expression coding: model to learn compact and
7

---

Under review as a conference paper at ICLR 2023
identity-invann invariiat facial expression embedding by explicitly disentangling the identity attribute. The model takes two pictures as input and outputs distance between two pictures; (3) ResNet a classic computer vision model to classify image.
Baseline Sample. We choose three baseline methods that are widely used for various tasks: (1)Mean baseline values. The baseline value of each input variable is set to mean value of this variable over all samples; (2)Zero baseline values, baseline values of all input variables are set to Random) baseline samples, randomly selected baseline samples from dataset.
Shapley value verify. To verify the generality of EG, we define four utility function: (1) 12 players; (2) 12 players with different utility of coalitions; (3) 20 players; (4) 20 players with same utility function to (2). We assume that utility function will have much high computation cost corresponding actual cases so we use DNN model to simulate utility function. The DNN model can perfectly replace utility function with partial data, with results shown in Appendix B.2. In (2), we choose 50% coalitions as train dataset. In (3)&(4) we choose 40% coalitions as train dataset. We set coalitions with all players input and set our sampled coalitions as baseline values. Considering that we randomly sample partial coalitions firstly in ES, we conduct three independent experiments to avoid the influence of randomness.
random
OU1
2010
menn
-22
-3
rardom
OLTS
mean
-3a
SHPP wiku
-0.33
-C.15
-130
-0.55
1.55
0.00
632
0.30
rendoom
DAPS
zero
mean
+02
+13
ED
83
82
03
Figure 5: Experiments Resnet and ImageNet dataset with different baseline values. We can ob- viously find out that our SIG method is similar with random baseline which is the common baseline method but more concerned about areas in zero and mean baseline method.
8

---

Under review as a conference paper at ICLR 2023
As shown in Figure 2, can discover that in some cases, ES achieves similar results compared to other three baselines, but in other cases, the results of other three baselines are quite disastrous while ES shows a relatively accurate results. It shows that ES not only performs better than other three baselines but is more stable than other three baselines.
Baseline Verify. In computer vision tasks, if we treat each pixel player, there would be 244*244 players, which is acceptable computation. In order to reduce computation and compute each pixel's contribution, compute each player's contribution. We set our sampled baseline coalitions as baseline values and set coalition with all players as imput.
treat 80*80 pixel area as player and apply Integrated Gradients to
As shown in Figure 3. we experiment with our SIG method in the facial expression codeing task. We select positive points that values > 0 which will reduce the distance between two pictures in our views because negative points that values < 0 which have the opposite effect. And we discover that negative points have a similar distribution with positive points as Figure 4 shows. If we mix positive and negative values, this will lead to poor interpretability.
Compared with other three baseline methods, our SIG results are more concentrated near hu- man facial organs, such as eyes, lips, etc, more in line with human intuition both in negative and positive values. Although Zero and mean baseline methods can also identify human facial organs, they will also pay attention to other areas of human face, in some cases even focusing areas be- yond the scope of human faces. More excitedly, the time cost by our SIG method is close to other baseline methods. More details are in Appendix B1.
And as shown in Figure 5, we experiment with our SIG method in the image classification tasks based on ResNet. We set almost same configuration to facial expression coding task. The performance of our SIG method is similar to random baseline, but more focused on areas that zero baseline and mean baseline pay attention to while random baseline ignores.
CONCLUSION
In this paper, we discover that Integrated Gradients takes a shortcut compared with Shapley value, which leads to unsatisfactory and unstable results. We propose Effective Shapley to find out a set of coalitions, which corresponds to calculation of Shapley value. In order to reduce compu- tation and get more applicable DNN explainer, we propose Shapley Integrated Gradients which combines Integrated Gradients with Effective Shapley. Experiments show that our Effective Shapley value method can well approximate the ratios between the Shapley values of players and our Shap- ley Integrated Gradients method can achieve better and more stable performance than Integrated Gradients.
For future work, we will explore the connections among features, set related features as a single player, and try find more reasonable coalition set. What's more, will improve the random sampling method and get closer to Shapley value and apply Effective Shapley value to complex applications.
REFERENCES
Kjersti Aas, Martin Jullum, and Anders Løland, Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artificial Intelligence, 298: 103502, 2021.
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pp. 272-281. PMLR, 2019.
Robert J Aumann and Lloyd S Shapley. Values of non-atomic games. Princeton University Press, 2015.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin. Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local remormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.

---

Under review as a conference paper at ICLR 2023
Hugh Chen, Scott Lundberg. and Su-In Lee. Explaining models by propagating shapley values of
local components. Explainable Al in Healthcare and Medicine, pp. 261-270. Springer, 2021.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in neural information processing systems, 30, 2017.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437, 2017.
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability the data manifold. arXiv preprint arXiv:2006.01277 2020.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251. PMLR, 2019.
Amirata Ghorbani and James Y Zou. Neuron shapley: Discovering the responsible neurons. Ad- vances Neural Information Processing Systems, 33:5922-5932, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
I Elizabeth Kumar, Suresh Venkatasubramaniin, Carlos Scheidegger, and Sorelle Friedler. Prob- lems with shapley-value-based explanations as feature importance measures. In International Conference on Machine Learning, pp. 5491-5500. PMLR, 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv: 000.18002000888 2018
Luke Merrick and Ankur Taly. The explanation game: Explaining machine learning models using Conference for Machine Learning and Knowledge
shapley values. In International Extraction, pp. 17-38. Springer, 2020.
Jie Ren, Zhanpeng Zhou, Qirui Chen, and Quanshi Zhang. Towards game-theoretic view base- line values in the shapley value. 2021.
Lloyd S Shapley. Notes the n-person gameii: The value of n-person (195). Lloyd S Shapley, 1951.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145- 3153. PMLR, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328, PMLR, 2017.
Wei Zhang, Xianpeng Ji, Keyu Chen, Yu Ding, and Changjie Fan. Learning a facial expression embedding disentangled from identity. In Proceedings of the IEEE/VV conference computer vision and pattern recognition, pp. 6759-6768, 2021.
10

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM
A.1 PROOF OF THEOREM
First, we rewrite Aumann Shapley value's formula into gradient form as follows:
f(ds)
v(tI + ds) - v(tl)dtt
r11 + ds) v(tl)
dsdt
ds
ds /-² 4 (((l)) dt.
yss
Considered that coalition usually plays game on the basis of game's background playground. The utility of coalition can be regarded as the utility of coalition with background, v(tl) = v(tl + Ø). The Aumann Shapley can be written fs:)(ds) = ds Jo & v(er+0) dt. Integrating from the empty set/background of game Ø to the complete set I, we can get the Aumann shapley value of the complete set:
is
f(1) = 54)= f/' 4 v(ll 2))
dtds.
4p
Since in Integrated Gradients baseline sample x' is considered containing nearly zero infor- mation, we set baseline sample x' as the background of game. As complete set means all players that join the game, points alongside P2 participate in contribution allocation problem in Integrated Gradients. We make all points alongside P2 as T.
/'
Φ u(t(x - xx)+x')
f(1)
1
Jn
4P8
f-'
60 vlt(l - x') + x')
= -x- 20)
dtds.
6ss
Noticing that s a vector, for each feature, we can get Integrated Gradients of feature æe
f(s:) (x; - xt) //'' & v(t(z
x') x²)
dtds.
Proof over.
A.2
THE PROOF OF THEOREM 2
We can model the problem as independent samples, So For nj, the probability of selecting k effective sample
Pk C/e,, g -b
leal-
For perfectly proportional sampling, we limit the proportion without constraint of the number
of Omin
02 aN 62 i.... eN-
The probability of sampling correct number samples is
A P=
ΣI
c los | ley
11

---

Under review as a conference paper at ICLR 2023
A.3
THE PROOF OF THEOREM 3
The hypothesis means gradients between S and S U 2; are constant
gou(t( - x')++ v(1) v(0)
1 [0.1]
5ps
1
Following equation, we prove Theorem 3:
f(1) Ix vT v20)
1
v( - v()) v(1 + - v(0)
-((
B MORE EXPERIMENTAL RESULTS AND DETAILS
B.I DISCUSSION ABOUT BASELINE VALUES IN COMPUTER VISION
For ResNet, we choose origin image as input of Integrated Gradients and make coalitions se- lected by our sample method as baseline values, noted proposal 1. To test validation of our sampled coalitions, we choose zero baseline values as baseline values and choose our sampled coalitions as inputs and average contributions, noted proposal 2. To our surprise, there isn" 't much difference be- tween the two methods, as shown. In consideration computational efficiency we choose coalitions selected by method as baseline values and choose origin image input, since PyTorch performs optimization of gradient calculation and it's faster to set origin image as input. From left to right, it's proposal 2, proposal proposal proposal zero baseline, mean baseline, and random in each picture.
Figure 6: The effect of different configuration for ResNet ImageNet.
We set the same proposals to ResNet. Results shown in Figure 7. From left to right, it's proposal 2, proposal 1, zero baseline, mean baseline and random in each picture.
From the result, can obviously find that our method has similar explanation to proposal in ResNet, but our method has a better explanation for facial expression task. Moreover, our method much faster than proposal 2.
12

---

Under review as a conference paper at ICLR 2023
Figure 7: The effect of different configurations for DLN on facial expression task.
B.2 DDDITIVE EXPERIMENT FOR HUMAN DEFINED UTILITY FUNCTION.
B.2.1 VERIFY OF DEEP NEURAL NETWORKS
a
Figure 8: The effect our trained deep neural network.
We use trained deep neural networks to replace utility function the computation of Shapley value, as shown. Each row represents an independent experiment. It can be obviously found that our trained deep neural network can perfectly replace utility function, leading to almost same Shapley value. arepresents Env (1) represents Env (2); represents Env (3); d represents Env(4).
B.2.2 DIFFERENCE BETWEEN INTEGRATED GRADIENTS AND MARGINAL DIFFERENCE
We accumulate marginal difference and Integrated Gradients between S I; and S, where E F/i and x; represents player. From the result, view that U dr linearly connect S
13

---

Under review as a conference paper at ICLR 2023
in deep neural network and the result also proves that correctness of Theorem 2. orepresents Env (1); b represents Env (2); represents Env (3); d represents Env(4). Each row represents an independent experiment.
Figure 9: The difference between Integrated Gradients and marginal difference based on sampled coalitions.
14


Reviews:
[{"review_id": "SQndIA2S9k", "reviewer": "Reviewer_QQsg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes ideas to combine integrated gradients and shapely values effectively. The problem according to the paper is that shapely values  are too expensive to be computed directly. The paper wants to use the ideas from shapely values to find a proper baseline for integrated gradients. The paper proposes effective shapely value, an approximation of shapely values that can be computed more effectively.\n\nThe paper claims that experimental results show that a combination of shapely values with integrated gradients provides good results.", "summary_of_the_review": "The paper does not provide sufficient experimental justification of the proposed approach to be accepted. It ignores a lot of issues with interpretability methods and their evaluation that are discussed in literature. ", "strength_and_weaknesses": "Weaknesses:\n- Experimental validation\n  -  Validation in Fig 2.  The entire experimental setup is not provided. It is not possible to actually implement this experiment and it is not possible to verify why it would make sense. Additional detail is needed.\n - Figures 3, 4, 5: it is not obvious at all why these visualisations are better than any other interpretability method. Why is this method showing the right result?\n - Positioning the work relative to work looking at the limits and potential issues with interpretability methods. There is not discussion whether this method performs well w.r.t. to any of these papers discussing the effectiveness of interpretability methods and their potential limitations.\n    - e.g. the evaluation by Hooker et al. A benchmark for interpretability methods in deep neural networks\n    - Adebayo et al. Sanity checks for saliency maps\n    - Kindermans et al. The (un) reliability of saliency methods, \n   - Wilming et al. Scrutinizing XAI using linear ground-truth data with suppressor variables\n\n- Clarity\n  - On page 3 the shapely value is defined properly. However It is not clear how the Aumann Shapley value should be interpreted with a neural network. The citation for the Aumann Shapley value is not available in open access nor is it discussed properly on websites such as wikipedia. With the limited discussion here I am not able to properly understand the nuances of this. For example it is not clear how the players in the game map to an image classification problem.\n  - On Page 3 at the bottom: We make the hypothesis that baseline sample x′ is considered as background in game and all points alongside P2 are considered as complete set I in game. It is never defined what background in the game actually means. How this should be interpreted. \n  - Theorem 1: please provide an intuition in the main paper. There is also no sufficiently formal definition of all components in the theorem to actually understand the proof in the supplementary.\n  - Page 5: partial coalition appears twice in the document and is not defined. It is not clear from the context what it actually is.\nStrengths:\n - Algorithm 1 and 2: make sure everything is defined properly. It is not clear for example how you can sample according to equation 1 as mentioned in algorithm 1. What is shown there is a sum, not a distribution.\n  - \"We first set players coalition as a single player, like 80*80 area as a player in image.\" This sentence is incredibly confusing. How do you change between a player contribution or not.\n\nStrengths:\nN/A", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "PWBU0OW2mG8", "reviewer": "Reviewer_cr3F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a proportional sampling method to estimate the ratios between Shapley values by introducing effective coalitions. Then, it combines this method and the existing Integrated Gradients to propose the Shapley Integrated Gradients that can explain a deep neural network (DNN) prediction by attributing the prediction to its input features.\n", "summary_of_the_review": "It is challenging to evaluate the soundness of the proposed approach due to the clarity and quality of the paper. Furthermore, the proposed baseline values are not motivated properly. As a result, I believe this paper requires a major revision.\n", "strength_and_weaknesses": "In my view, the biggest weakness of the paper is the clarity which requires a major revision.\n\n1. The contribution of the paper is not clear. In the 3rd paragraph of the introduction, the paper states that its main contribution is to \"propose to find a set of informative baseline values\". However, this paragraph discusses the Shapley value as a measure of the feature contribution without discussing how baseline values will be chosen in this paper (as opposed to existing baselines: e.g., zero baseline values).\n\n2. The Integrated Gradient approach is motivated by 2 axioms including the implementation invariance. Thus, it is not convincing that this work criticizes Integrated Gradients by proposing an approach (Shapley value) that not only does not satisfy the implementation invariance but also requires more computation than the Integrated Gradients.\n\n3. Furthermore, while Section 4.1.2 is about the limits of Integrated Gradients, it only discusses the Shapley value. I do not understand the limits of Integrated gradients after reading this section.\n\n4. The paper criticizes that \"Integrated Gradients takes a shortcut compared with calculation path of Shapley value and it will lead to an unsatisfactory and unstable explanation\". However, it uses the Integrated Gradients in Algorithm 2, instead of computing the marginal contribution of coalitions. Thus, why the \"unsatisfactory and unstable\" of the Integrated Gradients is prevented in this work?\n\n5. After section 4.2, I still do not understand how to estimate the ratio between Shapley values even though it is the main objective of this section. In particular, how does the proposed method \"transform the goal from accurate Shapley value to ratio between Shapley value\"?\n\n6. In Theorem 2, P is not in the range of [0,1] so why is it even a probability?\n\n7. In the first paragraph of Section 4.3, it discourages the computation of the marginal difference of sampled coalitions by Algorithm 1 because it is impossible for a large number of players. However, algorithm 1 requires computing these marginal differences when it classifies S into m_i. Does it mean that algorithm 2 is very computationally intensive since it performs computing these marginal differences when invoking algorithm 1? Moreover, this work should include analyses of the computation complexities of the two algorithms.\n\n8. The paper also mentions the 2 key requirements of baselines in the first paragraph of the introduction, but it never says how the proposed baseline values satisfy (or do not satisfy) these 2 requirements.\n\n9. Why does the factor for d_jk not change even though we have an additional requirement that v(S cup x_i) - v(S) = j? According to (1), the factor only depends on the size |S|.\n\n10. In Theorem 2, the explanation is confusing: is a_j a set of coalitions or just a coalition (\"a_j considered to be effective coalitions in bj\")? What does it mean by min(a_j)?\n\n11. Algorithm 2 is confusing because it says the output is baseline samples for each player (in the 2nd line). However, the last line of the algorithm outputs the average of all values of IG.\n\nThere are other minor issues.\n\n12. In the line below equation (4), where is the point (s1,r2)?\n\n13. Typo in C_{|ej|}^{|aj|} in Theorem 2.\n\n14. \"seep up\" -> \"speed up\"\n\n15. D = (d_1j, ..., d_jk, ...) is confusing due to the notation j.\n\n16. What is \\phi in equation (3)?\n\n17. Figure 1.b, there is not any orange color.\n\n18. In the 2nd line of the paragraph after Figure 1, x=(s1,s2) should be x=(S1,S2).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C5sBNLLDk9", "reviewer": "Reviewer_y3yp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on the task of attributing the prediction of DNN to its input features and try to address the baseline issues. The work proposes to find a set of baseline values corresponding to Shapley values.To solve the computation dilemma of Shapley value,  a proportional sampling method (Effective Shapley value, ES) is proposed to well simulate the ratios between the Shapley values\nof features. Besides, the submition proposed Shapley Integrated Gradients (SIG) to combine Integrated Gradients with ES, to achieve a good balance between efficiency and effectiveness. I am not an expert in this filed. However, as a reader, this work is unreadable and requires a lot of prior information. Besides, there are a lot of undefined notations or terms.", "summary_of_the_review": "I cannot recommend accepting this work. Although I am no working in this field, this work is far from a ready work for publish.", "strength_and_weaknesses": "This method combines Shapley values with the Integrated Gradients method for better accuracy of locating relevant features while causing a trade-off to the efficiency of the original Integrated Gradients method. The novelty of such a combination is limited. The sampling strategy to reduce computation of SHAPLEY VALUE is also not novel.  Moreover, it's not clear how to set baseline values when calculating v(S) in Algorithm 1. It's also not clear how to directly set Di as baseline values in Algorithm 2. Why the authors use Integrated Gradients is also questionable. Each player's contribution is also calculated in the algorithm1. Why is the Integrated Gradients method necessary to calculate the contribution? The overall writing of the methodology is confusing. \n\nMoreover, there are a lot of notations confusing readers: what is v in Eq (1)? What is the difference between S1 and s1? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject