Can We Faithfully Represent Absence States to Compute Shapley Values on a DNN?
Authors: Jie Ren
Zhanpeng Zhou
Qirui Chen
Quanshi Zhang
Venue: ICLR 2023 poster (2023)
Keywords: explainable AI
attribution methods
deep neural networks

Abstract: Masking some input variables of a deep neural network (DNN) and computing output changes on the masked input sample represent a typical way to compute attributions of input variables in the sample. People usually mask an input variable using its baseline value. However, there is no theory to examine whether baseline value faithfully represents the absence of an input variable, i.e., removing all signals from the input variable. Fortunately, recent studies (Ren et al., 2023a; Deng et al., 2022a) show that the inference score of a DNN can be strictly disentangled into a set of causal patterns (or concepts) encoded by the DNN. Therefore, we propose to use causal patterns to examine the faithfulness of baseline values. More crucially, it is proven that causal patterns can be explained as the elementary rationale of the Shapley value. Furthermore, we propose a method to learn optimal baseline values, and experimental results have demonstrated its effectiveness.

Published as a conference paper at ICLR 2023
CAN WE FAITHFULLY REPRESENT MASKED STATES TO COMPUTE SHAPLEY VALUES ON A DNN?
Jie Ren, Zhanpeng Zhou, Qirui Chen, Quanshi Zhang Shanghai Jiao Tong University
ABSTRACT
Masking some input variables of a deep neural network (DNN) and computing output changes on the masked input sample represent typical way to compute at- tributions of input variables in the sample. People usually mask an input variable using its baseline value. However, there is no theory to examine whether baseline value faithfully represents the absence of input variable, i.e., removing all sig- nals from the input variable. Fortunately, recent studies (Ren et al. 2023a Deng et al. 2022a) show that the inference score of DNN can be strictly disentangled into set causal patterns (or concepts) encoded by the DNN. Therefore, we pro- pose to causal patterns to examine the faithfulness of baseline values. More crucially, it is proven that causal patterns can be explained as the elementary ra- tionale of the Shapley value. Furthermore, we propose method learn optimal baseline values, and experimental results have demonstrated its effectiveness.
INTRODUCTION
Many attribution methods (Zhou et al. 2016 Selvaraju ct al. 2017 Lundberg and Lee 2017 Shriku-
mar al.) 2017) have been proposed to estimate the attribution (importance) of input variables to the model output, which represents an important direction in explainable AI. In this direction, many studies (Lundberg and Lee 2017 Ancona et al. 2019; Fong et al.) 2019) masked some input vari- ables of a deep neural network (DNN), and they used the change of network outputs on the masked samples to estimate attributions of input variables. As Fig. T shows, there are different types of baseline values to represent the absence of input variables.
Theoretically, the trustworthiness of attributions highly depends on whether the current baseline value can really remove the signal of the input variable without bringing in new out-of-distribution (OOD) features. However, there is no criterion to evaluate the signal removal of masking methods. To this end, we need to first break the blind faith that seemingly reasonable baseline values can faithfully represent the absence of input variables, and the blind faith that seemingly 0OD baseline values definitely cause abnormal features. In fact, because DNN may have complex inference logic, seemingly OOD bascline values do not necessarily generate OOD features.
phenomenon. The core challenge of thcoretically guarantecing or examining whether the baseline value removes all or partial signals of an input variable is to explic- encoded by DNN in countable manner. To this end,
Concept/causalityyeennn
itly define the signal/conccep/knooleee Ren et al. (2023a) have discovered a counter-intuitive concept-emerging phenomenon in a trained DNN. Although the DNN does not have a physical unit to encode explicit causality concepts, Renn] et al. (2023a): Deng et al. (2022a) have surprisingly discovered that when the DNN is sufficiently trained, the sparse and symbolic concepts emerge. Thus, we such concepts new perspective to define the optimal bascline value for the absence of input variables.
As Fig. [ shows, each concept represents an AND relationship between a specific set S of input variables. The co-appearance of these input variables makes a numerical contribution Us to the network output. Thus, we can consider such a concept as a causal patter of the network output,
'Quanshi Zhang the corresponding author. He is with the Department of Computer Science and Engi- neering, the John Hopcroft Center, at the Shanghai Jiao Tong University, China. zqa 1 1022@sjtu edu. cn. Note that in this paper, the causal pattern means the extracted causal relationship between input variables and the output encoded by the DNN, rather than the intrinsic causal relationship hidden in data.

---

Published as a conference paper at ICLR 2023
PIO
explained as
value: Introduce additooal information (grey dots)
Blumning image: cannot remove low- www.freeuency features
Based on distribution wir.. contexts, Le. p(* p(*((8)
Zero introduce additional Information tblack dots)
remove old cousal patterns
Introduce new cousal pattems
Figure I: (Left) Previous masking methods may either introduce additional signals, cannot move all the old signals. (Right) The inference of the DNN on images masked by these baseline values be well mimicked by causal patterns.
and Us is termed the causal effect. For example, the concept of a rooster's head consists of the forehead, eyes, beak, and crown, L.e., S= (forehead, eyes, beak, crown) {f, e,,,,, c3 for short. Only if input variables J. e, b, and e co-appear, the causal pattern is triggered and makes an effect Us on the confidence of the head classification. Otherwise, the absence of any input variables in the causal pattern S will remove the effect.
Ren et al. 2023a) have extracted
of sparse causal patterns (concepts) encoded by the DNN More importantl. the following finding has proven that such causal patterns can be considered as elementary inference logic used by the DNN. Specifically, given input sample with n variables, we can generate 2n different masked samples. We can use relatively small number of causal patterns to accurately mimic network outputs on all 2n masked samples, which guarantees the faithfulness of causal patterns.
Defining optimal baseline values based on causal patterns. From the above perspective of causal patterns, whether baseline values look reasonable and fit human's intuition is no longer the key factor to determine the trustworthiness of baseline values. Instead, we evaluate the faithfulness of baseline values by using causal patterns. Because the baseline value is supposed to represent the absence of an input variable, we find that setting an optimal baseline value usually generates the most simplified explanation of the DNN, i.e., may extract a minimum number of causal patterns to explain the DNN. Such an explanation is the most reliable according to Occam's 's Razor.
We prove that using incorrect baseline values makes single causal pattern be explained as an exponential number of redundant causal patterns. Let us consider the following toy example, where the DNN contains causal pattern 8=(f,e, b, c} with considerable causal effect F. on the output. If an incorrect baseline value by of the variable f (forehead) just blurs the image patch, rather than fully remove its appearance, then masking the variable f cannot remove all score E. The remaining score E-U(,,e. will be explained as redundant causal patterns Ufa,), Ufe,)) Ufe,e.) etc.
Furthermore, incorrect baseline values may also generate new patterns, For example, if baseline values of (f.e, b, e} are set as black regions, then masking all four regions may generate new pattern of black square, which a new causal pattern that influences the network output.
Therefore, we consider that the optimal baseline value, which faithfully reflects the true inference logic, usually simplifies the set of causal patterns. L.e., it usually reduces the overall strength of exist- ing causal effects most without introducing new causal effects. However, we find that most existing masking methods are not satisfactory from this perspective (see Section 3.2 and Table 11, though the masking method based on conditional distribution of input variables (Covert et al., 2020b; Frye et al. 2021) performs bit better.
In particular, notice that Shapley values also be derived from causal patterns in theory, i.e., the causal patterns are proven to be elementary effects of Shapley values. Therefore, we propose a new method learn optimal baseline values for Shapley values, which removes the causal effects of the masked input variables and avoids introducing new causal effects.
Contributions of this paper be summarized follows. (1) We propose metric to examine whether the masking approach in attribution methods could faithfully represent the absence state of input variables. Based on this metric, find that most previous masking methods are not reliable.
2

---

Published as a conference paper at ICLR 2023
(2) We define and develop an approach to estimating optimal baseline values for Shapley values, which ensures the trustworthiness of the attribution.
2 EXPLAINABLE AI THEORIES BASED ON
INTERACTIONS
This paper is typical achievement on the theoretical system of game-theoretic interactions. In fact, our research group has developed and used the game-theoretical interaction as a new perspective to solve two challenges in explainable AI, i.e., (1) how to define and represent implicit knowledge encoded by DNN as explicit and countable concepts, (2) how to use concepts encoded by the DNN to explain its representation power or performance. More importantly, we find that the game- theoretic interaction is also a good perspective to analyze the common mechanism shared by previous empirical findings and explanations of DNNs.
Explaining the knowledge/concepts encoded by DNN. Defining interactions between input variables of DNN in game theory a typical research direction (Grabisch and Roubens 1999. Sundararajan et al. 2020). To this end, we further defined the multi-variate interaction (Zhang et al. d) and multi- orderrrrderrre interaction (Zhang et al. 2021bb to represent interactions of different com- plexities. Ren et al. (2023a) and Li and Zhang (2023) first discovered that could consider game-theoretic interactions as the concepts encoded by a DNN, considering the following three terms. (1) We found that trained DNN usually only encoded very sparse and salient interactions, and each interaction made a certain effect on the network output. (2) We proved that we could just use the effects of such small number of salient interactions well mimic/predict network outputs on an exponential number of arbitrarily masked input samples. (3) We found that salient interac- tions usually exhibited strong transferability across different samples, strong transfcrability across different DNNs, and strong discrimination power. Thus, the above three perspectives comprised the solid foundation of considering salient interactions as the concepts encoded by DNN. Furthermore, Cheng et al. (202Tb) found that such interactions usually represented the most reliable and prototypical concepts encoded by a DNN. Cheng et al. (2021a) further analyzed the different signal-processing behaviors of DNN in encoding shapes and textures.
The game-theoretic interaction is also a new perspective to investigate the representation power of DNN. DeNg et al.) (2022a) proved counter-intuitive bottleneck/difficully of DNN in representing interactions of the intermediate complexity. Zhang et al. (2021b) explored the effects of the dropout operation interactions to explain the generalization power of DNN. Wang et al. 2021a b): Ren et al. (2021) used interactions between input variables to explain the adversarial robustness and adversarial transferability of a DNN. (Zhou et al. (2023) found that complex (high- order) interactions were more likely to be over-fitted, and they used the generalization power of different interaction concepts explain the generalization power of the entire DNN, |Rnn et al. (2023)) proved that Bayesian neural network (BNN) was less likely to encode complex (high- order) interactions, which avoided over-fitting.
Game-theoretic interactions are also used to analyze the common mechanism shared by many empirical findings. [Dng et al. (2022b) discovered that almost all (fourteen) attribution meth- ods could be re- forormulated as reallocation of interactions in mathematics. This enabled the fair comparison between different attribution methods. Zhhang et al. (2022) proved that twelve previous empirical methods of boosting adversarial transferability could be explained as reducing interactions between pixel-wise adversarial perturbations.
PROBLEMS WITH THE REPRESENTATION OF THE MASKED STATES
was first introduced in game theory to measure the contribution of each player in game. People usually use Shapley values to estimate attributions of input variables of a DNN, Let the input sample x of the DNN contain n input variables, i.e., x [.i...., ... Tn). The Shapley value of the i-th input variable ø, is defined as follows.
The Shapley value (Shapley 1953
6. - -s []S][(n JS] - 1)1/m!] [u( - e(xs))
(I) where c(ss) E R denotes the model output when variables in S are present, and variables in N masked. Specifically, v(wa) represents the model output when all input variables are masked. The Shapley value of the variable is computed as the weighted marginal contribution of when the variable is present w.r.f. the case when the variable i is masked, i.e. e())s(())) - -v(=s).
3

---

Published as a conference paper at ICLR 2023
(itgg) the income is than 50k
Table 1:
The ratio R of the remaining and newly intro- duced causal effects in the masked inputs. A small value of R meant that baseline values removed most original causal effects and did not introduce many new effects.
Ug 0.72
+0.58
11.15
0.95
00.58
+0.52
0.52
relation
ediran
ship
status
on
SeX
Re(men))
(2
(ifonditional)
R!um)))
Figure 2: Causal patterns that explain the inference on a sample in the income dataset.
MNIST CIFAR-I0 10
1.1736
0,3043
0.4159
0.3780
0.2185
0.6630
0.8042
0.7288
0.4027
0.1211
The Shapley value is widely considered a fair attribution method, because it satisfies the linearity, dummy, symmetry, and efficiency axioms (Weher) (988) (please refer to Appendix D). However, when we explain DNN, typical challenge how to faithfully define the absence of an input vari- able. The most classical way is to use baseline values (or called reference values) 6 [b,, by...., b.] to mask variables to represent their absence. Specifically, given an input sample x, wy denotes a masked sample, which is generated by masking variables in the set N S.S.
i E S, (æs): zs; otherwise, (ars): by We aim to learn optimal baseline values & to faithfully represent absent states of input variables.
Decomposing a DNN's output into sparse interactions. Given a trained DNN v and an input a with n input variables, Ren et al.] (2023a) have proven that the DNN output v((z)) can be decomposed into effects of interactions between input variables. Specifically, let S C N denote subset of input variables. The interaction effect between variables in S is defined as the following Harsanyi dividend (Harsanyi 1982).
Us ( -15"1 v(Zs))
(-1)88
(3)
Based on this definition, we have v(xr) = EsgN Us.
Sparse salient interactions can be considered causal patterns (or concepts) encoded by the DNN. Theorem [ and Remark prove that most interactions have ignorable effects Us ss 0, and people can use a few salient interactions with non-ignorable effects to well approximate the inference scores on 2d different masked samples. Thus, we can consider such interactions as causal p or concepts encoded by the DNN. Accordingly, we can consider the interaction effect Us as the causal effect. Besides, Remark has been verified on different DNNs learned for various tasks by experiments in both Appendix G. and (Ren et al., 2023a).
(023a) and Appendix E. Let us consider a DNN Theorem (Faithfulness, proven by Ren el al.
e and an input sample æ with n input variables. We can generate 2n different masked samples, i.c.
{as)S S N). The DNN's 's outputs on all masked samples can always be well mimicked as the sum of the triggered interaction effects in Eq. (3). i.e., VSCN, e(#s) = Esscs Us.
Remark (Sparsity) Interaction effects in most DNNs are usually very sparse. Most interaction effects are almost zero, i.e., Us O. few most salient interaction effects in Q (less than 100 interaction effects in most cases) are already enough to approximate the DNN, i.., VS S N, (,v()))> Estenss's Ug. where [90] < 2".
Each causal pattern (concept) S represents an AND relationship between input variables in S. For example, the head pattern of a rooster consists of (forehead, eyes, beak, crown]. the forehead, eyes, beak, and crown of the rooster co-appear, then the head pattern S = {forehead, eyes, beak, crown} is triggered and makes causal effect Us on the output. Otherwise, any part is masked, the causal pattern S will not be triggered, and the DNN's ' inference score v(xx) will not receive the causal effect Us. In sum, when we mask an input variable i, is supposed to remove all causal effects of all AND relationships that contain the variable i. Please see Appendix F for the proof.
3.1 EXAMINING THE FAITHFULNESS OF BASELINE VALUES USING CAUSAL PATTERNS
We use salient causal patterns (or concepts) evaluate the faithfulness of masking methods. Specifi- cally, we examine whether baseline values remove most causal effects depending on D, and whether baseline values generate causal effects.
The evaluation of the masking methods based on salient causal effects is theoretically supported from the following three perspectives. First, Theorem [ and Remark [ prove that the inference score of a DNN can be faithfully disentangled into a relatively small number of causal patterns. Second, Theorem 2 shows that Shapley values can be explained re-allocation of causal effects input
4

---

Published as a conference paper at ICLR 2023
variables. Therefore, reducing effects of salient patterns means the removal of elementary factors that determine Shapley values. Besides, in order to verify that the reduction of causal patterns can really represent the absence of input variables, we have conducted experiments to find that salient patterns triggered by white noise imputs were much less than those triggered by normal images.
Please Appendix G.2 for details.
(proven by Harsanyi 1982) and Appendix E.2 We can directly derive Shapley values from the effects Us of causal patterns. The Shapley value can be considered as uniformly allocating each causal pattern S's effect Us to all its variables, i.e. Φx Esg SgNN(() |s|+1 Usl(()
Theorem 2
Third, an incorrect baseline value b; will make partial effects of the AND relationship of the variable i be mistakenly explained as an exponential number of additional redundant causal patterns, which significantly complicates the explanation. Therefore, the optimal baseline value is supposed to generate the most sparse causal patterns the simplest explanation of the DNN. Compared to dense causal patterns generated by sub-optimal baseline values, the simplest explanation removes as many existing causal effects as possible without introducing additional causal effects.
Remark 2 (proof in Appendix E.33 Let us consider function with single causal pattern f(=))= ws 11 (,,,), Accordingly, ground-trun baseline values of variables are obviously {5;), because setting any variable vj E S, 2, will deactivate this pattern. Given the correct baseline values b3 6 we can use a single causal pattern to regress [ i.e., Us = [(ws), V S + S, Us = O.
Theorem 3 (proof in Appendix E.3) For the function f(zs)= w 11 -,)), we use m' incor- rect baseline values (U, +
5 to replace correct ones to compute causal effects, then the function will be explained to contain at most 2n causal patterns.
Theorem (proof in Appendix E.3 we use m' incorrect baseline values to compute causal effects in the function (xx)) = wi (ay - 65), a total of 4m))84mm²) causal patterns of the k-th order emerge, k 2 15|-m'. A causal pattern of the k-th order means that this causal pattern represents the AND relationship between k variables.
Specifically, Remark 2 Theorems 3 and 4 provide new perspective to understand how incorrect baseline values generate new causal patterns. Remark [ shows how correct baseline values explain toy model that contains single causal pattern. Theorems and 4 show that incorrect baseline values will use an exponential number of redundant low-order patterns to explain a single high-order causal pattern. For example, we are given the function f(x) = (1)xp-6))(z, 5,) s.l. H, = 3, F. = 4, = 5, = 3. If we use ground-truth baseline values {ōp, ). then the function explained as simple as a single causal pattern Ω = ((p..) [p..)) which yields correct Shapley values o, Φ,- = 0.5 w. according to Theorem 2 Otherwise, if we use incorrect baseline values {% 1. K = 1}. then this function will be explained as four causal patterns n= {0, {p), (g}, {p. h Le,, f(x) U.Co (p) + Uto + ((pee) where U, 2wr U(w) Aww Uto) -3w, and U(p.z) = 6w are computed using incorrect baseline values. Incorrect baseline values increase complicated causal patterns and lead to incorrect Shapley values op ww, d, 0. In fact, the existence of most newly introduced causal patterns is due to that the effects of a high-order causal pattern not fully removed, and that OOD causal patterns (new OOD edges shapes) may be caused by incorrect baseline values.
3.2 PROBLEMS WITH PREVIOUS MASKING METHODS
In this subsection, we compare causal patterns the masked sample with causal patterns in the original sample to evaluate the following baseline values.
(1) Mean baseline values. As Fig. [ shows, the baseline value of each input variable is set to the mean value of this variable over all samples (Dabkowski and Gal 2017). i.e. 6, Ez [æs]. However, empirically, this method actually introduces additional signals to the input. For example, mean values introduce massive grey dots to images and may form new edges abnormal causal patterns. This has been verified by experiments in Table [ Experimental details will be introduced later.
(2) Zero baseline values. Baseline values of all input variables are set to zero (Ancona et al. 2019 Sundararajan et al. (017), f.e. Vi E N, b, = 0. As Fig. shhows, just like mean baseline values, zero baseline values also introduce additional signals (black dots) the input (verified in Table
5

---

Published as a conference paper at ICLR 2023
Table 2: Analysis about previous masking methods.
Baseline values constant
Different sumples share the same bascline values or not
Setting of buseline values
Shortcomings
Mean baseline value
introduce additional signals
Zero baseline value
introduce additional signala
Blurring baseline value
X
not remove low-frequency componeuts
Marginal distribution
assume leature independence
Conditional distriburion
X
balance between the computationa post and the accuriicy high computational cost
Ours
both are ok
(3) Blurring input samples. Fong and Vedaldi (2017) and Fong et al. (2019) blur image pixels T, using a Gaussian kernel as its masked state. [Covert et al. (2020a); Sturmfels et al. 2020) mentioned that this approach only removed frequeeyy signals, but failed to remove low- www-frqquency signals.
(4) For each input variable, determining a different baseline value for each specific context S. In- stead of fixing baseline values as constants, some studies use varying baseline values to compute m(rs) given œ which are determined temporarily by the context S in æ. Some methods (Frye et al.) 2021 Covert et al.] 2020b) define v(xs) by modeling the conditional distribution of variable values in , given the context S. i.e. e(æs)) = Bppa' las) [model(ats [mmdelis L 2 ssyss)]. The operation means the concatenation of x's 's dimensions in 5 and x's dimensions in N \ S. By assuming the independence between input variables, the above conditional baseline values can be simplified to marginal baseline values (Lundberg and Lee 2017), i.e. v((s)) = Ep(=') [model(azg L æ s))-
We conducted experiments to examine whether the above baseline values remove all causal pattems in the original input and whether baseline values introduce new causal patterns. We used the metric [Us|)] to evaluate the quality of masking. We gen- crated a set of samples based on æ, where a set of input variables were masked, and U's denote the causal effect in such masked samples. Us denote the causal effect in the original sample æ, which was used for normalization. Ur(maie)) denotes the causal effect in white noise input, and it represents the unavoidable effect of huge amounts of noise patterns. Thus, we considered the U (-fauise) term as an inevitable anchor value and removed it from R for a more convincing evaluation. The masking method would have two kinds of effects on causal patterss. (1) We hoped to remove all existing salient patterns in the original sample. (2) We did not expect the masking method to introduce new salient patterns. Interestingly, the removal of existing salient patterns decreased the R value, while the triggering of new patterns increased the R value. Thus, the It metric reflected both effects. A small value of R indicated a good setting of baseline values.
R= Ex
(EsgN D'sl D)
We used 20 images in the MNIST dataset (LeCun et al. 1998) and 20 images in the CIFAR-10 dataset (Krizhevsky et al., 2009) to compute R. respectively. We split each MNIST image into 7 7 grids and split each CIFAR-10 image into 8 8 grids. For each image, we masked the central 4 x 3 grids using the zero baseline, mean baseline, blur baseline, and the baseline based on the conditional distribution, and computed the metric of R(amm) R(mem)) (M and (ccomitionn)) respectively. Table shows that the ratio by using previous baseline values were all large. Although the masking method based on conditional distribution performed better than some other baseline values, our method exhibited the best performance. It indicates that previous masking methods did not remove most existing patterns and/or trigger new patterns.
3.3 ABSENCE STATES AND OPTIMAL BASELINE VALUES
In the original scenario of game theory, the Shapley value was proposed without the need to define the absence of players. When people explain DNN, we consider that the true absence state of variables should generate the most simplified causal explanation. Remark 2 and Theorem 3 show that correct baseline values usually generate the simplest causal explanation, i.e., using the least number of causal patterns to explain the DNN. In comparison, if an incorrect baseline value b, does not fully remove all effects of AND relationships of the variable then the remained effects will be mistakenly explained as a large number of other redundant patterns.
The above proof well fits Occam's Razor, i.e., the simplest causality with the minimum causal patterns is more likely to represent the essence of the DNN's inference logic. This also lets us consider the baseline values that minimize the number of salient causal patterns (ie., achieving the simplest causality) as the optimal baseline values.
Therefore, the learning of the baseline value by of the i-th variable can be formulated to sparsify causal patterns in the deep model. Particularly, such baseline values supposed to remove existing

---

Published as a conference paper at ICLR 2023
Table 3: Examples of generated functions and their ground-truth baseline values.
Functions (VI N. E (0.1])
The ground truth of haseline values
00 -0.185m; (22 + 23) 12 242550277
67 0 for i E 3,
2222 sigmoid( 25,27 + 2.50) 2y
b for (4.5, 6. 7), = 0 (1,23,8..)
-iggmoid 4xg 4zg- 6.00) P443560T T
B for i= = 2. by 0 for (1.3.445.6..88...
causal effects without introducing many new effects.
arg min, Eg [(x)), subject to ((x) {$ S [ >
(4)
where Us (z(b) denotes the causal effect computed the sample by setting baseline values
ESTIMATING BASELINE VALUES
derive Eq. 44) learn optimal baseline values, but the computational cost of enumerating all causal patterns is exponential. Thus, we explore an approximate solution learning baseline values. According to Theorem 4 incorrect baseline values usually mistakenly explain high-order causal patterns where the order m of the causal effect Ug is defined as the cardinality of S, m = |S).
Based on Theorem
unnecessarily large number of low-order causal patterns,
Thus, the objective of learning baseline values is roughly equivalent to penalizing effects of low- order causal patterns, in order to prevent learning incorrect baseline values that mistakenly represent the high-order pattern exponential number of low-order patterns.
min L(b), subject to L(b) = -.E SEN.18ISK SEM [+>(]]]]] (z]
(5)
An approximaee-ye--efcciiett solution. When each input sample contains huge number of vari- ables, e.g., an image sample, directly optimizing Eq. (5) NP-hard. Fortunately, we find the multi- order Shapley value and the multi-order marginal benefit in the following equation have strong connections with multi-order causal patterns (proven in Appendix
H. as follows.
(x=b) ESCNY( [u( b) - u(xs, 6)] ESC 18
18|=m
(6)
Av: (S) b) (xssu()) b) b(xs, b) ULUC)))))
where 6fm²) (z b) and (S) b) denote the m--order Shapley value and the m-order marginal benefit computed using baseline values b, respectively, where the order m given as m=|S).
According to the above equation, high-order casual patterns Us are only contained by high-order Shapley values m! and high-order marginal benefits Av; Therefore, in order to penalize the effects of low-order causal patterns, we penalize the strength of low-order Shapley values and low-order marginal benefits, respectively, as an engincering solution to boost computational efficiency. In experiments, these loss functions were optimized via SGD.
Lihhppey (b) Σ ΣV
Σ
16(
(m) (x|b)), Immrginal (b) Σ ΣΣ
E SCNN (Sa b) 18
Um. PEX KEN
m--Uni mex IEN
where m denotes the maximum order be penalized. We have conducted experiments to verify that baseline values b learned by loss functions in Eq. 77) could effectively sparsify causal effects of low-order causal patterns in Eq. 5). Please see Appendix G.3 for results.
Most importantly, we still used the metric R in Section 3.2 to check whether the learned baseline values removed original causal patterns in the input while not introducing new patterns. The low value of ((uu)) in Table 1 shows that baseline values learned by our method successfully removed existing salient causal patterns without introducing many new salient patterns.
EXPERIMENTS
5.1 VERIFICATION OF CORRECTNESS OF BASELINE VALUES AND SHAPLEY VALUES
Correctness of baseline values synthetic functions. People usually cannot determine the ground truth of baseline values for real images, such as the MNIST dataset. Therefore, we con- ducted experiments synthetic functions with ground-truth baseline values, in order to verify the
7

---

Published as a conference paper at ICLR 2023
Table 4: Accuracy of the learned baseline values.
initialize as O
initialize as 0.5
initialize
initialize as O
initialize as 0.5
initialize
Synthetic functions
98.06%
98.70%
98.70%
98.06%
18%
98.14%
Functions in (Tang 2018
88.52%
91.80%
90.16%
86.89%
91.80%
10.16%
Table 5: Accuracy of Shapley values on the extended Addition-Mulliplccationn dataset using different
settings of baseline values.
Zero
Mean
Baseline values in SHAP
Kemel SHAP 33.88%
Frye et al. (2021
Ours
Accuracy 82.88%
72.63%
81.25%
66.0%%
100%
Table 6: An example of Shapley values
Baseline values
The computed Shapley values (or
computed on different baseline values. The function is model(ar)
Truth/Ours
[0.0. 5. 0.06, 0.10. 0.06, 4.23)
62.1--55- (11 -
Zero baseline
(0,0, -..99 1.15, 0.87, -4.23]
(. 1...-000111111 and the input is æ-
Mean bascline
(1.31,0, -2.00, -0.74, 0.58, 0.19. -2.113
Setting in SHAP
f0.014.0, -0.011, -0.003, 0.003, -0.010
correctness of the learned baseline values We randomly generated 100 functions, whose causal pat- terns and ground truth of baseline values could be easily determined. This dataset has been released The generated func- tions were composed of addition, subtraction, multiplication, exponentiation, and the sigmoid opera- tions (see Table 3. For example, for the function y= sigmoid((3z12, -223 3 rggr +0.25(#6 + E (0. 1), there were three causal patterns (i.e. {a1, (2, 233, [14, (xz,,))) ), which were activated only if x2 = for e (1, 2, 4,5, 6, 7} and xy =0. In this case, the ground truth of baseline values was bt = 0 for i = 21, 2, 4, 5, 6, 7 and bg = 1. Please see Appendix G.4 for more discussions about the setting of ground truth baseline values.
at whtttp::/ hgb. com /ttp///pp11//// faithful- baslllbaeelll leeeennievvvaae
We used our method to learn baseline values on these functions and tested the accuracy. Note that |b.-b]|e [0, I] and 6, E0, 13. If [b, -b, 0.5, we considered the learned baseline value correct. We set 0.5n in both Lshuptey and Lmarginat. The results are reported in Table
4 and are discussed later.
Correctness of baseline values on functions in (Taag et al. 2018). Besides, we also evaluated the correctness of the learned baseline values using functions in Tsang al. (2018). Among all the 92 input variables in these functions, the ground truth of 61 variables could be determined (see Appendix (i.4). Thus, we used these annotated baseline values to test the accuracy. Table 4 reports the accuracy of the learned baseline values on the above functions. In most cases, the accuracy was above 90%, showing that our method could effectively learn correct baseline values. A few functions in (Tsang et al. (2088) did not have salient causal patterns, which caused errors in the learning. Besides, in experiments, we tested our method under three different initializations of baseline values (i.e., O, 0.5, and 1). Table 4 shows that baseline values learned with different initialization settings all converged to similar and high accuracy.
Correctness of the computed Shapley values. Incorrect baseline values lead to incorrect Shap- ley values. We verified the correctness of the computed Shapley values on the extended Addition- Multiplication dataset (Zhang et al., (021c). We added the subtraction operation to avoid all base- line values being zero. Theorem of each causal pattern to its compositional variables. This enabled us determine the ground-truth Shapley value of variables without baseline values based on causal patterns. For example, the func- tion f(x)= 2 + 52zzи4 I5 s.t. [1, 1, 1, contained three causal patterns, according to the principle of the most simplified causality. Accordingly, the ground-truth Shapley values were m===2/3//2 - 5/2, and Øs =1. See Appendix G.5 for more details. The estimated Shapley value @i was considered correct if (d, - do1 < 0.01; otherwise, incorrect. Then, we computed the accuracy of the estimated Shapley values as the ratio of input variables with correct Shapley values.
2 considers the Shapley value as a uniform assignment of effects
Discussion on why the learned baseline values generated correct Shapley values. We computed Shapley values of variables in the extended addition-Mullipliccnn values, and compared their accuracy in Table 5 The result shows that our method exhibited the highest accuracy. Table [ shows an example of incorrect Shapley values computed by using other baseline values. Our method generated correct Shapley values in this example. For the variable xs, due to its negative coefficient 1198, its contribution should be negative. However, all other baseline values generated positive Shapley values for 2s. The term -4.23z, showed the significant effect of the variable 2; on the output, but its Shapley value computed using baseline values in SHAP was just -0.010, which was obviously incorrect.
dataset using different baseline
8

---

Published as a conference paper at ICLR 2023
Values of input sample
Zero baseline
Mean baseline values
Basaline SAGE
Ous mmarginal zeroinit
in SHAP
stupply zero-init
Naliva-courtry
Houre-per-wek
Capital-oas
Caplal-gann
Sex
Using Lauupky (zero-ent)
Race
Raiatonship
Satapay
Shepley
Ocoupation
(mcan ini))
Martta-status
Lsinng Emangtnn (zero int) Using puurgeat (mconintt)
Education
3411
Workeloas
Ago
40 60 60
130
*gt jaogaoge
Shapley value
Figure 3: The learned baseline values (left) and Shapley values computed with different baseline val- ues (right) on the income dataset. Results on the MNIST, the CIFAR-1O, and the credit datasets are shown in Appendix and G.7
5.2 RESULTS AND EVALUATION ON REALISTIC DATASETS AND MODELS
Learning baseline values. We used our method to learn baseline values for MIPs, LeNet (Lee- Cun al. 1998), and ResNet-20 (He et al.] 2016) trained on the UCI South German Credit dataset (namely credit dataset) (Dua and Graff 2017), the UCI Census Income dataset (namely 2017), the MNIST dataset (LeCun et al. 1998). and the CIFAR-10
income dataset) (Dua and Graft
dataset (Krizhevsyy et al.) 2009 respectively. We learned baseline values by using either Lshuply or
3-p(y pyy ig)" In the [1, where i((s)) denotes the output feature of the penultimate layer given the masked input æs, in order to boost the efficiency of learning. We set 0.22 for the MÑIST and the CIFAR- 10 datasets, and set x= 0.55 for the simpler
Tmaaniinal as the loss function. In the computation of Tshhupey, we set ((xs) ) = log p(y" mn)
computation of Lmarginal, [S] was set to (5) =
data in two UCI datasets. Given baseline values, we used the sampling-based approximation (Castro et al. 2009) to estimate Shapley values
We used two ways to initialize baseline values before learning, i.e. setting baseline values or mean values over different samples, namely zero-init and mean-init, respectively. Fig. 3 (left) shows that baseline values learned with different initialization settings all converged to similar base- line values, except for very few dimensions having multiple local- minimmm solutions (discussed in Appendix G.7.. which proved the stability of our method.
Comparison of attributions computed using different baseline values. Fig. B shows the learned baseline values and the computed Shapley values on the income dataset. We found that attributions generated by zero/mean baseline values conflicted with the results of all other methods. Our method obtained that the occupation had more influence than the marital status on the income, which was somewhat consistent with our life experience. However, baseline values in SHAP and SAGE some- times generated abnormal explanations. In this top-right example, the attribute capital gain was zero, which was not supposed to support the prediction of "the person made over 50K year." How- ever, the SAGE's baseline values generated a large positive Shapley value for capital gain. In the bottom-right example, both SHAP and SAGE considered the marital statuy important for the pre- diction. SHAP did not consider the occupation as an important variable. Therefore, we considered these explanations not reliable. Attribution maps and baseline values generated on the CIFAR- 10 and the MNIST datasets are provided in Appendix G.6 Compared to zero/mean/blurring baseline values, our baseline values were more likely to ignore noisy variables in the background, which were far from the foreground in images. Compared to SHAP, our method yielded more informative attributions. Besides, our method generated smoother attributions than SAGE.
6 CONCLUSIONS
In this paper, we have defined the absence state of input variables terms of causality. Then, we have found that most existing masking methods cannot faithfully remove existing causal patterns without triggering new patterns. In this way, we have formulated optimal baseline values for the computation of Shapley values those that remove most causal patterns. Then, we have proposed an eef-cffcceentt method to learn optimal baseline values that represent the absence states of input variables. Experimental results have demonstrated the effectiveness of our method.

---

Published as a conference paper at ICLR 2023
ETHIC STATEMENT
This paper aims to examine the masking approach in previous explaining methods. We find that previous settings of the masking approach cannot faithfully represent the absence input variables, thereby hurting the trustworthiness of the obtained explanations. Therefore, we propose new method to learn optimal baseline values to represent the absence of input variables. In this way, the trustworthiness of explanations of the DNN is further boosted. There ethical issues with this paper.
REPRODUCIBILITY STATEMENT
We have provided proofs for all theoretical results in Appendix and Appendix H We have also provided experimental details in Section 5 and Appendix G Furthermore, we will release the code when the paper is accepted.
ACKNOWLEDGEMENT
This work is partially supported by the National Nature Science Foundation of China (62276163) National Key R&D Program of China (2021ZD01 1EEDD0116000 1602), Shanghai Natural Science Foundation (2IJC1403800,222R 434600), National Nature Science Foundation of China (U19B2043). This work is also partially supported by Huawei Technologies Inc.
REFERENCES
Kjersti Aas, Martin Jullum, Anders Løland. Explaining individual predictions when features are dependent: More accurate approximations to shapley values. Artif. Intell., 298(C), sep 2021. ISSN 0004-3702. doi: 10. 110.101 1016606j (0016//.rtiit.2200 a02 103502.
Chirag Agarwal and Anh Nguyen. Explaining image classifiers hy removing input features using generative models. In Hiroshi Ishikawa, Cheng-Lin Liu, Tomas Pajdla, and Jianbo Shi, editors, Computer Vision ACCV 2020, pages 101-118, Cham, 2021. Springer International Publishing. ISBN 678-3-030-69544-6
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with polynomial time algorithm shapley value approximation. Kamalika Chaudhuri Ruslan Salakhutdinov. editors, Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pages 272-281. PMLR, 09-15 Jun 2019.
Robert J. Aumann and Lloyd S. Shapley. Values of Non-Atomic Games. Princeton University Press, Princeton, 2015. ISBN 9781400867080. doi: doi: 10ii:1..11159999 400867080.
Peer Bork, Lars J Jensen, Christian von Mering, Arun K Ramani, Insuk Lee, and Edward M Marcotte. Protein interaction networks from yeast human. Current Opinion in Structural Biology. 14(3):292 299299929999922 2004. ISSN 0959-440X. doi: https://00..og//1.11010.......b
Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5):1726-11 1730, 2009. ISSN 0300--0548. doi: htts://doi.org/ 10. 1016/ j.cor.2008.04 04.0004. Selected papers presented the Tenth International Symposium Locational Decisions (ISOLDE X).
Xu Cheng, Chuntung Chu. Yi Zheng, Jie Ren, and Quanshi Zhang. A game-theoretic taxonomy of visual concepts in dnns. arXiv preprint arXiv:2106. 10938038......... 2021a.
Xu Cheng, Xin Wang. Haotian Xue, Zhengyang Liang, and Quanshi Zhang. A hypothesis for the aesthetic appreciation in neural networks. arXiv preprint arXiv:: 02646, 2021b.
Covert, Scott Lundberg, and Su-In Lee. Feature removal unifying principle for model explanation methods. arXiv preprint arXiv:2011.03623, 2020a.
Ian C. Covert, Scott Lundberg, and Su-In Lee. Understanding global feature contributions with additive impor- tance measures. In NeurlPS 2020. ACM, December 2020h.
10

---

Published as a conference paper at ICLR 2023
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6967-697777 1696, 2017.
Huiqi Deng, Qihan Ren, Hao Zhang, and Quanshi Zhang. Discovering and explaining the representation bottleneck of DNNS. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event April 25-29, 2022. OpenReview.net, 2022a.
Huiqi Deng. Na Zou, Mengnan Du, Weifu Chen, Guocan Feng. Ziwei Yang. Zheyang Li, and Quanshi Zhang. Understanding and unifying fourteen attribution methods with taylor interactions. arXiv preprint, 2022b.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In 2016 IEÉE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 4829 4837. IEEE Computer Society, 2016.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In 2019 IEEF/CVP International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pages 2950-2958. IEFE, 2019. doi: 10.1 10911199..222190000000
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In IEEE International Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 3449-3457, IEEE Computer Society, 2017. doi: 10.1 10.11.///
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in co- operative games. International Journal game theory, 2844)5547- 1999, doi: 10. 10.1000/000 02075001110
Ulrike Grömping. Estimators of relative importance in linear regression based on variance decomposition. The American Statistician, 61(2):139 147, 2007.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti.
Local rule-based explanations of black box decision systems. arXiv preprint 1rXiv: 10820, 2018.
Michael Harradon, Jeff Druce, and Brian Ruttenberg, Causal learning and explanation of deep neural networks via autoencoded activations. arXiv preprint 1rXiv:1802.00544 00541, 2018.
John C. Harsanyi. A Simplified Bargaining Model for the Pereooo Cooperative Game, pages 44-70. Springer Netherlands, Dordrecht, 1982 ISBN 978-94-017--552--9 doi:
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
Alon Keinan, Ben Sandhank, Claus C. Hilgetag, Isaac Meilijson, and Eytan Ruppin. Fair attribution of func- tional contribution in artificial and biological networks. Neural Comput.. (6/9):1887-1913 2004. doi: 10. (001166200997666433338777
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, 1lya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In Peter L.. Bartlett, Fernando C. N. Pereira, Christopher C. Burges, Léon Botton, and Kilian Q Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevado, United States, pages 1 1106-11 14, 2012.
I. Elizabeth Kumar, Suresh Venkatasubramaniin, Carlos Scheidegger, and Sorelle A. Friedler. Problems with shapley-value-base based explanations as feature importance measures. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Muchine Learning Research, pages 5491-5500. PMLR, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEFE, 86(1 ():277777722, 1998. doi: 10.1 109/5.726791.
11

---

Published as a conference paper at ICLR 2023
Mingjie Li and Quanshi Zhang. Does neural network really encode symbolic concept? arXiv preprint arXiv:2302.13080 2023.
Scott M. Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4765-4774, 2017.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv:1802.03880 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. 6th International Conference on Learning Repre- sentattins, ICLR 2018, Vancouver, BC, Canada, April 30 OpenReview.net, 2018.
May 3, 2018, Conference Track Proceedings.
Luke Merrick and Ankur Taly. The explanation game: Explaining machine learning models using shapley values. In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar R. Weippl, editors, Machine Learning and Knowledge Extraction- 4th IFIP TC 5, TC 12, WG 8.4 4, WG 8.9, WG 12.9 International moss-Domain Conference, CD-MAKE 2020, Dublin, Ireland, August 25-28, 2020, Proceedings, volume 12279 of Lecture Notes in Computer Science, pages 17-38. Springer, 2020. doi: 100799799 3-030-0333333 57321-8)22
Alexander Mordvintsev. Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. In Google Research Blog, 2015.
Jie Ren, Die Zhang, Yisen Wang. Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang. Meng Zhou, Jie Shi, and Quanshi Zhang. Towards unified game- theoretic view of adversarial perturbations and robust- ness. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 3797-3810. Curran Associates, Inc., 2021.
Jie Ren, Mingjie Li, Qirui Chen, Huiqi Deng, and Quanshi Zhang. Towards axiomatic, hicrarchical, and symbolic explanation for deep models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2023a.
Qihan Ren, Huiqi Deng, Yunuo Chen, Siyu Lou, and Quanshi Zhang. Bayesian neural networks tend to ignore complex and sensitive concepts. arXiv preprint arXiv:2302. 13095, 2023b.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In IEEE Inter- national Conference Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 618-626. IEEE Computer Society, 2017, doi: 10.110/CCC222..7......
Pranab Kumar Sen. Richard H. Lindeman, Peter F. Merenda, and Ruth Z. Gold. Introduction bivariate and multivariate analysis. Journal of the American Statistical Association. 76:752, 1981
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games,
1953.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3145-3153. PMLR, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualis- ing image classification models and saliency maps. In Yoshua Bengio and Yann LeCun, editors, 2nd In- termational Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
Xi Song, Tianfu WU, Yunde Jia, and Song-Chun Zhu. Discriminatively trained and-or tree models for object detection, In 2013 IEFE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA, June 23-28, 2013, pages 3278-3285. IEFE Computer Society, 2013.
Erik Štrumbelj and Igo Kononenko. Explaining prediction models and individual predictions with feature con- tributions. Knowledge and information systems, 41 41(3):647 665, 2014, doi: 10. 10.100///1111--00 0--011-0677--0
Erik Strumbelj. Igor Kononenko, and M Robnik Šikonja. Explaining instance classifications with interactions of subsets of feature values. Data & Knowledge Engineering, 68( (8(10):886904 2009. doi: 60.1016/..atal 2009.01.000
12

---

Published as a conference paper at ICLR 2023
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Proceedings of the 37th International Conference Machine Learning. ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceedings of Machine Learning Rexearch, pages
PMLR, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference on Machine Learning, ICMI, 20/7, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 331 3319-3328, PMLR, 2017.
Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9259-9268. PMLR, 2020.
Michael Tsang, Dehua Cheng, and Yan Liu. Detecting statistical interactions from neural network weights. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
Xin Wang, Shuyun Lin, Hao Zhang, Yufci Zhu, and Quanshi Zhang. Interpreting attributions and interactions of adversarial attacks. In 2021 IEEE/CVF International Conference on Computer Vision, 10C1 Montreul, QC. Canada, October 10-17, 2021. pages 1075-1084, IEEE, 2021a.
Xin Wang, Jie Ren. Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang. A unified approach to interpreting and boosting adversarial transfcrability. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b.
Robert James Weher. Probabilistic values for games, page 101-120. Cambridge University Press, 1988. doi: 1010077/CB00700111534446666
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. arXiv preprint 1506.0650006677 2015.
Die Zhang, Hao Zhang, Huilin Zhou, Xiaoyi Bao, Da Huo, Ruizhao Chen, Xu Cheng, Mengyue Wu, and Quanshi Zhang. Building interpretable interaction trees for deep NLP models. In Thirty-Fifh AAAI Confer- ence on Artificial Intelligence, AAAI 2021, Thirt-Thire Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 4328-11337.. AAAI Press, 2021a.
Hao Zhang, Sen Li, Yinchao Ma. Mingjie Li, Yiehen Xie, and Quanshi Zhang, Interpreting and boosting dropout from game- theoretic view. In International Conference Learning Representations, 2021b.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 10877- 10886. AAAI Press, 2021c.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence. EAAI 2021, Virtual Event, February 2-9, 2021, pages 10877- 10886. AAAI Press, 2021d.
Quanshi Zhang, Xin Wang, Jie Ren, Xu Cheng, Shuyun Lin, Yisen Wang, and Xiangming Zhu. Proving common mechanisms shared by twelve methods of boosting adversarial transferability. arXiv preprint arXiv:2207.J 1694, 2022.
Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene enns. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Bolei Zhou, Aditya Khosla. Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 2921-2929. IEEE Computer Society. 2016.
Huilin Zhou, Hao Zhang, Huiqi Deng, Dongrui Liu, Wen Shen, Shih-Han Chan, and Quanshi Zhang. Concept- level explanation for the generalization of dnn. arXiv preprint 2r32v:2302. 13091, 2023.
13

---

Published as a conference paper at ICLR 2023
Table Accuracy of Shapley values computed on the pre-defined decision tree, which was based on the MNIST dataset.
The learned baseline values could re- cover original samples from adversarial examples.
Table 8:
Baseline
baseline SHAP
MNIST LeNet
2.33
0.43
value
MNIST on AlexNet
2,53
1.15
Accuracy 92.99% 92.93%
93.37%
96.84%
CIFAR- 10 on ResNet-20
1.19
1.11
A RELATED WORKS
No previous methods directly examined the faithfulness of the masking methods. Instead, we made a survey in a larger scope of attribution methods and other explainable Al studies, and put them in the appendix. Nevertheless, we will put this section back to the main paper if the paper is accepted.
In the scope of explainable AL, many methods (Simonyan et al. 2014] Yosinski et al.) 2015: Mord-
vintsev et al., 2015| Dosovitskiy and Brox, 2016 Zhou et al. 2015) have been proposed to explain
the DNN. Among all methods, the estimation of attributions for each input variable represents
classical direction (Zhou et al., 2016 Selvaraju et al. 2017; Lundberg and Lee, 20177] Shrikumar ct al. 2017). In this paper, we mainly focus on attributions based on Shapley values.
Shapley values. The Shapley value (Shapley 1953) in game theory was widely considered as a fair distribution of the overall reward in game to each player (Weber 1988). (Sen et al. 1981) and (Grömping 2007) used the Shapley value to attribute the correlation coefficient of a linear regression to input features. (Strumbelj et al. 2009; Strumbelj and Kononenko 2014) used the Shapley value to attribute the prediction of a model to imput features. (Bork et al. 2004) used the Shapley value to measure importances of protein interactions in large, complex biological interaction networks. (Keinan et al. (004)) employed the Shapley value measure causal effects in neurophys- ical models. (Sundararajan et al. 2017) proposed Integrated Gradients based on the AumannShap- ley(Aumann and Shapley 2015) cost-sharing technique. Besides above local explanations, (Covert et al. 2020bb focused on the global interpretability.
In order to compute the Shapley value in deep models efficiently, (Lundberg and Lee 2017) pro- posed various approximations for Shapley valus in DNNs. (Lundberg et al. (2188) further computed the Shapley value on tree emsembles. (Aas al. 202I) generalized the approximation method in (Lundberg and Lee| 2017) to the case when features were related to each other. (Ancona et al. 2019) further formulated a polynomial-tine approximation of Shapley values for DNNs.
Baseline values. In terms of baseline values of Shapley values, most studies (Covert et al. 2020a Merrick and Taly, 2020 (Sundararajan and Najmil [020)] Kumar et al.] 2020) compared influences of baseline values on explanations, without providing any principles for setting baseline values. Shrikumar et al. (2017) proposed DeepLIFT to estimate attributions of input variables, and also mentioned the choice of baseline values. Besides, AAgawall and Nguyen (2021) and Frye et al. (2021) used generative models to alleviate the -ut-o--disttibution problem caused by baseline values. Unlike previous studies, rethink and formulate baseline values from the perspective of game- theoretic causality. We define the absent state of input variables, and propose a method to learn optimal baseline values based on the number of causal patterns.
B QUANTITATIVE EVALUATION OF ATTRIBUTIONS FOR IMAGE CLASSIFICATION
In order to quantitatively evaluate Shapley values computed by different baseline values the MNIST dataset, we constructed an And-Or decision tree following (Harradon et al. 2018), whose structure directly provided the ground-truth Shapley value for each input variable. Then, used different attribution methods to explain the decision tree. Table shows that our method generated more accurate Shapley values than other baseline values.
We constructed a decision tree (Song et al. 2013) for each category in the MNIST dataset. Specifi- cally, for each category (digit), we first computed the average image over all training samples in this category. Let ((r) R denote the average image of the c-th category. Then, we built a decision tree by considering each pixel as an internal node. The splitting rule for the decision tree was de- signed as follows. Given an input in the category c, the splitting criterion the pixel (node) x; was
14

---

Published as a conference paper at ICLR 2023
designed as ((E! > 0,5)&((z; > 0.5) { If (क) > 0.5)&((; > 0.5) = True, then the pixel value I, was added to the output; otherwise, H2 was ignored. In this way, the output of the decision tree was f(x) = -iev 2i, where V = {i E N (x > (z, > 0.5) = True} denote the set of all pixels that satisfied the above equation. For inference, the probability of æ belonging to the category c was p(c|z) = sigmoid(y(f(x))) 5)), where 7 = 40 was a constant and B x EiEN 1 >0.5 In
this case, we defined u(xxx) = log p(c/æ)
I-P(c|2) Thus, the co-appearing of pixels in V formed causal pattern to contribute for v(xx). In other words, because Vi E N, 22 2 O, the absence of any pixel small probability p(c[2z) < 0.5 and a small v. This pattern can also be understood as an AND node in the And-Or decision tree (Song et al.) 2013).
in V might deactivate this pattern by leading
In the above decision tree, the ground-truth Shapley values of input variables (pixels) were easy to determine. The above decision tree ensured that the absence of any variable in would deactivate the causal pattern. Therefore, according Theorem 2 in the paper, the output probability should be fairly assigned to pixels in V. i.e., they shared the same Shapley values Φ; = IVT pixels that were not contained in the output, their ground-truth Shapley values were zero
"(=w) For other
We estimated Shapley values of input variables in the above decision tree by using zero baseline val- ues, mean baseline values, baseline values in SHAP, and the learned baseline values by our method, respectively. Let Φ; denote the estimated Shapley value of the variable i. If o a S 0.01, we con- sidered the estimated Shapley value Φi correct; otherwise, incorrect. this way, we computed the accuracy of the estimated Shapley values, and Table shows that our method achieved the highest accuracy.
C REMOVING ADVERSARIAL PERTURBATIONS FROM THE INPUT
Let æ denote the normal sample, and let ah (Madry et al. 2018). According to (Ren et al.) 2021), the adversarial example a:d mainly created out-of-distribution bivariate interactions with high-order contexts, which were actually related to the high-order interactions (causal patterns) in this paper. Thus, in the scenario of this study, the adversarial utility was owing to out-of-distribution high-order interactions (causal patterns). The removal of input variables was supposed to remove most high-order causal patterns.
x + 5 denote the adversarial example generated by
Therefore, the baseline value can be considered as the recovery of the original sample. In this way, used the adversarial example " initialize baseline values before learning, and used Lmorginal to learn baseline values. If the learned baseline values b satisfy ||b==|| ||>""><<]| we considered that method successfully recovered the original sample some extent. We conducted experiments using LeNet, AlexNet (Krizhevsky et al. 2012). and ResNet-20 on the MNIST dataset (| S 32/255) and the CIFAR- I [[T]] <8/ 8/255). Tsooes that our method recovered original samples from adversarial examples, which demonstrated the effectiveness of our method.
D AXIOMS OF THE SHAPLEY VALUE
The Shapley value (Shapley 1953) was first introduced in game theory, which measures the con- tribution of each player in a game. Actually, given an input æ with n input variables, i.e., Zn). [.i,...,Zu] we can consider deep model as game with players N 21, n). Each player i is an input variable v, (e.g. an input dimension, a pixel, or a word). In this way, the problem of fairly estimating attributions of input variables in the DNN is equivalent assigning the total reward in the game to each player. The Shapley value is widely considered fair attribution method, because it satisfies the following four axioms (Weber 1988).
the problem of fairly
(1) Linearity axiom: If two games can be merged into a new game u(#s) = v(ms) + =w(#s) then Shapley values in the two old games also can be merged, i.e. Vi E N, di, = di.. + Di.w.
(2) Dummy axiom and nullity axiom: The dummy player is defined as a player without any inter- actions with other players, i.e. satisfying AS N \ {i), v(zsu(())
v(xs) + v( Then, the dummy player's 's Shapley value is computed as di = (1) The null player i is defined as a player that satisfies VS C N \ (i), v( v(ws). Then, the null player's Shapley value is di = 0.
For Table 7, the splitting criterion was designed (5 > 0.5).
15

---

Published as a conference paper at ICLR 2023
(3) Symmetry axiom: VS _ N (i.j), v( v(wsu())) then Φi -
(4) Efficiency axiom: The overall reward of the game is equal to the sum of Shapley values of all players, Le. v(æx) v(zo) Eien dr.
E PROOFS OF THEOREMS
This section provides proofs of theorems in the main paper.
E.1 PROOF OF THEOREM
Theorem (Faithfulness, proven by Ren et al. (2023a)) Let us consider a DNN and an input sample with input variables. We generate 2" different masked samples, iec. {æs|S C N}. The DNN's outputs on all masked samples can always be well mimicked as the sum of the triggered interaction effects in Eq. 3 i.e., VS C N, v(æs) Es'gs Us-
Proof: According the definition of the Harsanyi dividend, we have VS C N,
Σ
Ug!
Σ
Σ
(-1)
11118'11111 v(x))
$'58
8'S8 LCS'
Σ
Σ
L.C SS
88
Σ
Σ
1-1)''1'1
8'88:32 =
181-14
S - |4]
((LL
(-1) = v(xs)
m
E.2 PROOF OF THEOREM 2
Theorem Harsanyi dividends can be considered as causal patterns of the Shapley value.
Σ SCNVAC S Usu(i)
(8)
In this way, the effect of causal pattern consisting of m variables can be fairly assigned to the variables. This connection has been proved in (Harsanyi 1982).
Proof:
right
Σ
Usuta
|S|
91+1
SENVE
Σ
Σ
(11)
Σ (
|S| +1
SENVI)
458
= Σ
(-1)) 15| 14| u(L -
|S|+1
SCNVi)
LCS
(-1)
Σ
Σ
[0 (1))
x(D)
= L ((LU(HH) x(L)
K +++1++1
LENUO) KENAAMO
1-16
(-1)*
(n-1-14)) 111)
Σ
Σ
% Let = |K]
b +
LENVO)
km
[T|1(n
!TA))!
Σ
(u(LU v(L)]
by the property combinitorial number
n!
LCN(()
0 left
16

---

Published as a conference paper at ICLR 2023
Table 9: Comparison between ground-truth baseline values and incorrect baseline values. The last column shows ratios of causal patterns of different orders fm ES [Usl
sc [el We consider in- teractions of input samples that activate causal patterns. We find that when models/functions contain a single complex collaborations between multiple variables (i.e. high-order causal patterns), incor- rect baseline values usually generate mixture of many low-order causal patterns. In comparison, ground-truth baseline values lead to sparse and high-order causal patterns.
Functions (Vi N,a {J,1])
Baseline values
Ratios
ground truth: 6" = [0,0,000000
f()) = 21 22
0.5 incorrect: ((3) =
600
I 1.1,1
incorrect: 8(2) 0.., 0.0.0.1
incorrect: 6(3) 0.3,0.1,03,,,5,001
ground truth: b* 0,
[(x) 5i
incorrect: 8(1) [0.5, 0.5, 0.5
504 7.5)
incorrect: ((2) [0.6, 0.4, 0.3
x 1,1,111]
incorrect: ((3) [0.3, 0.6, 0.5,
5"
6(3)
6(21
6(9)
ground truth: 6" [0,0,0, 0,
f(x) = my (22 + tæ 24)
incorrect: ((1) = [0.5, 0.5, 0.5, 0.5]
== 1,00
incorrect: ((2) 10.2,033 0.6, 0.1
incorrect: b( [L.0,0.3. 0.1
h'
6()
(()
69
E.3 PROOF REMARK2, THEOREM 3, AND THEOREM 4
Remark 2 Let consider function with single causal pattern f(ms) = ws [ -55)- Ac- cordingly, ground-truth baseline values of variables are obviously {5,}, because seiting any variable Vj E S, 2, 65 will deactivate this pattern. Given the correct baseline values bj = 85, we can use single causal pattern to regress (. i.c., Us f(xs), V St S.Ug =
Theorem 3 For the function (#s) = ws [g - 85). if we use m' incorrect baseline values (U,16] &3) to replace correct ones to compute causal effects, then the function will be explained to contain at most nd causal patterns.
Theorem we use m' incorrect baseline values to compute causal effects in the function f(xs) = ws II ( (5)), total of (4-19(=mm)) causal patterns of the k-th order emerge, -S]-m'. causal pattern of the k-th order means that this causal pattern represents the AND relationship between k variubles.
Theoretical proof: Without loss of generality, let us consider an input sample æ, with Vj E S, o 8j. Based on the ground-truth truth baseline value {6,), ). we have
(1) u(as) = f(xs) ws II,ss 6,) +
(2) VS' (, ws ILes (x, 5,) I Tres (fr 6x) 0,
Accordingly, we have Us = (-1) S1- v(w5)) = v(es) # O. For S' Ç S, we have Us.
(-11 Ergg 0=
VS) let LU M, where I. C.C. S and M 0.8 Then, have
17

---

Published as a conference paper at ICLR 2023
Uge = Σ
(-1)1s"1-7
758
Σ
(-1)1891112 v((L))) + Σ (-1)1s"1-aa") v(war))
M'CM
L'#B
M'#0
E (-1))81-111-1M)) ((LLUN)) +(-1)18) e((2r))
+
EM
M'
=v(()
Σ (-1)18"-114 ((x)))
E (-1) (-11|11-1'11-1 ((00))1))((()))))))
LCL
UCLM'CM
1/70
=( (0l18'1-()u((ss)) + Σ (-11)88111811111 u(Es) % )((2)) + 0 Only if L' = S
M'CM M
Σ (-1)811-188--10000 =
M'CM
Therefore, there is only one causal pattern with non-zero effect Us. In comparison, if
m' incorrect baseline values (8]), where Ejes Lo m', then the function will be explained to contain at most 2 causal patterns. For the simplicity of notations, let
8= f1, ..., m). and 8i 61 E1, ... 8mm sm em', where E Eml # 0. Let T {1, 2,........ m' }. In this case, we have
(1) v(xs)= f(xs) #
(2) VS' S. m m' v(as)) = ws IT (xy 6;) Hrs (8) - 8t). Because |S| - |S > m'
there is at least one variable with ground-trum truth baseline value in S S Therefore, v(æs:) 0. Furthermore, Us (-1)8"
=0
(3) AS' S, k 2 m m', u(xs)) ) = ws [lses (x, lhes,, (8) 5t). If ( C S', then S CT and v(ass) # O. Otherwise, v(xs)) = O. Then,
Us Σ LEg
(-1))881-1410001
Σ
(-1)15 ((01) +
Σ
(-1) -14
LGS [
LCS'.L2m-m
+
Σ
Σ
(-1)1811-144 ((2))))))
LCS LES-L2M-mt LLSST
LGS L LES/T
Σ
LS8' L228T
If the above Ug. = O, it indicates that S g S". In this case, there is no subset S S' s.t. SVT C L. In other words, only if C * 0. In this way, a total of (x-()))mm)) causal patterns of the k-th order emerge, where the order & of a causal pattern means that this causal pattern S' contains k = | variables. There are totally -m= (1 = 2n causal patterns in æ.
For example, if the input æ is given as follows,
d 2e,,
T = (1....,m') 6, 4 G iES {m' +1...,,m?
2,
where e, o are arbitrary non-zero scalars. In this case, we have VS S T, Usru(m't..... 61 + O. Besides, if {m' + m) & S", we have Ug. = O. In this way, there are totally 2nd causal patterns in a.
Experimental verification: We further conducted experiments to show that the incorrect setting of baseline values makes a model/function consisting of high-order causal patterns be mistakenly explained as a mixture of low-order and high-order causal patterns. To show this phenomenon, we compare causal patterns computed using ground-truth baseline values and incorrect baseline values
18

---

Published as a conference paper at ICLR 2023
in Table 9. and the results verify conclusion. We find that when models/functions contain com- plex collaborations between multiple variables (i.e. high-order causal patterns), incorrect baseline values usually generate fewer high-order causal patterns and more low-order causal patterns than ground-truth baseline values. In other words, the model/function is explained as massive low-order causal patterns. In comparison, ground-truth baseline values lead sparse and high-order salient patterns,
F PROVING THAT MASKING INPUT VARIABLES REMOVES CAUSAL EFFECTS
In this section, we prove that for the causal pattern S if the input variable is masked, then the causal effect wi 0.
Proof:
let S = S' U {ij. if S is masked, then 7L s.t. L, Lz Therefore, ((L = vL). According to the definition of Harsanyi dividend (Harsanyi 1982). we have
Us Σ (-1) v(L)
LCS
Σ
(-1) (11)
(.g(8)((1))
(-1)
11111144-444001 + Σ
(1))
LC8
LCS
(-1)
(-1)184(41-10 ((()) Σ (-1)157--11004)
LES!
L55
Σ
(-1) (-1) (-1)15"1-u11) v(L)
LES!
(-1+1)(-1)8
-1-
LCS'
0
Note that the causal pattern not containing i will not be deactivated by the masking of i. For example, [eyes, beak) is not deactivated by the absence forehead, because this pattern represents the AND relationship between eyes and beak, and it does not contain forehead.
G MORE EXPERIMENTAL DETAILS AND RESULTS
G.1 VERIFICATION OF THE SPARSITY OF CAUSAL PATTERNS
In this subsection, conducted experiments to verify the sparsity of causal effects, which is intro- duced in Remark [ To this end, we computed causal effects Us of all 2" causal patterns encoded by DNN. Specifically, effects in the model. Figure 4 shows the distribution of absolute causal effects [Us| of causal pat- terns in the first five samples of each category of the income dataset. These results show that most causal patterns had insignificant causal effects, Ug s 0. Only a few causal patterns had salient causal effects.
trained three-layer MLP the income dataset and computed causal
Moreover, we also conducted experiments to demonstrate the universality of this phenomenon. We trained the five-layer MLP, CNN, LSTM, ResNet-32, and VGG-16 on the UCI census income dataset, the UCI TV news channel commercial detection dataset, the SST-2 dataset, and the MNIST dataset, respectively. Figure 5 shows the absolute causal effects Us in the descending order. These results show that various DNNs learned on different tasks could be explained by set of sparse causal patterns.
G.2 VERIFICATION OF USING CAUSAL PATTERNS TO EXAMINE THE STATE OF INPUT VARIABLES
In this subsection, we conducted experiments to verify that causal patterns reflect the states re- moving existing patterns. Given causal effects Us in the normal input image and causal effects
19

---

Published as a conference paper at ICLR 2023
S ample
Sample
Sample 3
Sample
Sample
orcusss panteme
audl
2007
100
Category
1200
cfuicaa prcccpalpttnne patteme
2100
1180
Category
1100
200
Figure 4: Histograms of absolute causal effects of causal patterns encoded by the three-layer MLP
trained on the income dataset.
strength of causa
16
trained
60
5
40
CNN trained on SST-2 dataset salient
16
LSTM trained SST-2 dataset
ResNet-32 trained on MNSST dataset salient
effects [Ug]
income dataset
TV news dataset
on MNIST dataset
salient
salient
salient
salient
noisy
noley
noiay
22
2
Index causal patterns
Figure 5: Absolute causal effects of different causal patterns shown in descending order, which shows that sparse causality is universal for various DNNs.
(noise) in the white noise input, we compared their distributions in Figure Note that assumed that the white noise input naturally contained information for classification than the normal input image. We found that most causal effects in the white noise input were close zero, and there were few salient causal patterns. Besides, we computed the average strength of causal effects in the above two inputs. the normal input, the average strength of causal effects EsçN Us| = 5.5285, while in the white noise input, the average strength was much smaller, Escx|U results indicated that salient causal patterns could reflect the information encoded in the input.
US
(noise) (adis)) 0.2321. These
Figure 6: The distribution of causal effects Us in the normal input, and the distribution of causal effects US (noise)) in the white noise input.
Number of causal patterms
normal input
white noise input
cm0
Us
G.3 EFFECTS OF THE PROPOSED METHOD ON MULTI-ORDER SHAPLEY VALUES AND MULTI-ORDER MARGINAL BENEFITS
In this section, we conducted experiments to verify that baseline values learned by the proposed loss function in Eq. 7 could effectively reduce causal effects of low-order causal patterns in Eq. (5).
To this end, we computed the metric E. (: (Us))((((N) v(xa))] to measure the relative strength of causal patterns of specific order m, order evaluate the effectiveness of baseline values. Fig. (aa) shows that compared to zero baseline values, our method effectively reduced low-order causal patterns. addition, Fig. g and Fig. [])) verify that the loss Lisapla in Eq. (7) reduced the number of salient causal patterns in Ω which means Lstaply avoided the exponential number of causal patterns caused by incorrect baseline values.
G.4 DISCUSSION ABOUT THE SETTING OF GROUND- TRUTH BASELINE VALUES.
This section discusses the ground truth of baseline values of synthetic functions in Section 5.] of the main paper. order to verify the correctness of the learned baseline values, we conducted experiments synthetic functions with ground-truth baseline values. We randomly generated 100
20

---

Published as a conference paper at ICLR 2023
1 +00 v(0)
Number of causal patters in qi (S NIJ( 21
Zero baseline values Count
Leamed baseline values
compused
the cessus dataset
the deddi
8 xio'
Count in the log space
Learned bao2 waaes Cownr
Zano bessire
10
10
Lag nomber
10
10
Log rumzer
values
7am biseline
4
10
10*
101
10
Leamed baseliee values fotuple
0
10 -10
0
40
11
Figure 7: Our method successfully (a) reduced effects of low-order causal patterns on the MLP trained on the census dataset, and (b) reduced the number of salient patterns.
Figure 8: Distribution of causal ef-
fects Us of causal patterns in 20 samples in the credit dataset.
Table 10: Examples of synthetic functions and their ground -ruth baseline values.
Fanctions (W N. [0.1])
The ground truth hoseline values
-0. -0.185rr + axy))-55 -appstghrttaoooppp
1 = for 0...e
-iigmodd - Azy lay - Liy +2 0.011rx(22 +
au ja30 jaa 0.17211921/24 2s) jasks 0,171 2g/2.50 0.003/2r aso anr -sigmold((kr) Sagaagit 7.5) zigmoudi Satdt Sxy 4) aypasteett
8, 1 for é (1,2,3). = 0 fori (4.....11 0 [I.....
6 for
1.3g
3,4, 10
-apeett 0.156(55 as I ng) #7o0ptFii
0fford i (le... 10)
functions whose causal patterns and ground truth of baseline values could be easily determined. As Table 10 shows, the generated functions were composed of addition, subtraction, multiplication, exponentiation, and sigmoid operations.
The ground truth of baseline values in these functions was determined based on causal patterns be- tween input variables. In order to represent the absence states of variables, baseline values should activate as few salient patterns as possible, where activation states of causal patterns were consid- ered as the most infrequent state. Thus, we first identified the activation states of causal patterns of variables, and the ground-truth of baseline values was set as values that inactivated causal pat- terns under different masks. We took the following examples to discuss the setting of ground-truth baseline values (in the following examples, Vi E N, xr E (0, 1} and by E (0, 1))-
fax x u2 + sigmoid(zz + 25 0.5) Let us just focus on the term of x2 in f (ax). The activation state of this causal pattern is J 42443- = when Vi E [1,2, x = 1. In order to inactivate the causal pattern, we set Vi E (1,2,3), by 0.
f(x) = - 22233 + (I1 + xz5) + ................................ Let us just focus on the term of -2 ((x) The activation state of this causal pattern is - 22233 -1 when vi E (1,2,3), x, = 1. In order to inactivate the causal pattern, we set Vi E [1,2,3), b = 0.
f(x) (x1 x2 xz)² +.... Let us just focus on the term of (α1 + 22 xz)² in f(x). The
activation state of this causal pattern is (x) + I2- - x333 = 8 when x1 = -22 = 1, 2y = 0. In order
to inactivate the causal pattern under different masks, we set bj bq = 0,
- 3x3 - 1.5) + ..... Let us just focus on the term of sigmoid(3an 222 3.0g 1.5) in f(x). In this case, æ1; 22, 33 form a salient causal pattern because sigmoid(33s (3 22
f(x) = sigmoid
3zg 1.5) > 0.5 only if d1 32 = and Z3 = O. Thus, in order to inactivate causal patterns,
ground-truth baseline values are set to bj 65 0, 1.
Ground-truth baseline values of functions in Tsang et al. 2018). This section provides more details about ground-truth baseline values of functions proposed in (Tsang et al.) 2018). We evalu- ated the correctness of the learned baseline values using functions proposed in (Tanng et al. 2018). Among all the 92 input variables in these functions, the ground truth of 61 variables could be deter- mined and are reported in Table IT] Note that some variables cannot be 0 or 1 (e.g. Ig cannot be zero in the first function), and we set Vi N. v2 E {0.001, 0.999) for variables in these functions instead. Similarly, we set the ground truth of baseline values Vi E N, Dy {0.001, 0.999). Some variables did not collaborate/interac with other variables (e.g. 24 in the first function), thereby hav- ing no causal patterns. We did not assign ground-truth baseline values for these individual variables, and these variables are not used for evaluation. Some variables formed more than one causal pattern with other variables, and had different ground-truth baseline values w.E.t. different patterns. In this case, the collaboration between input variables was complex and hard to analyze, so we did not consider such input variables with conflicting patterns for evaluation, either.
21

---

Published as a conference paper at ICLR 2023
Table 11: Functions in Tsang et al. 2018) and their ground baseline values.
Functions (Wi Nar, E 10.00100.000
The ground truth of buseline values
w w ai "(xa) logles + as)
65 - 0.900 for E (5)8,10),44 - 0.000 1 (1,2,
they 2 and - sin "(0.5e lnglles oup ley say +t (eyyta - -2th + logle) H of + Haa exp lay m ana + lrgaral - af (syan) + loglet + + ad) tha 1art bngieggny + vioplr + mi) Ino + + ingrptt eaplizuas) 1) exp(ie, + au + 1) confas da and (acctaa(as) t =cttaffs))) + madiisa it 20 mecounoennn
M O 909 for
5,6 0.001 for (1.2,7.9)
M 0.508 to ( (3.8.
B 0.999 fori (3.5.7.8)
1 0.988 e (1, (1,2,3), 0.001 for (4,6,8,9, 10)
15 a
0g 11.59 for (8.9, 0.001 for (1,2.3
0g 0,199 for i 9. 0.001 Ioe (1.23,44...
runy ges gestesnaytir sintrty dintra + sal) (r0ooe/0)f111)))
0.001 for 31,2,3,4,5,0)
touh(ruus asady// + explis + as) + kog((rrre + 1) more remml mofes seefarete)
M = 6.001 for e 7,,7, 9.10
sinblay 22) arcosstanniee
M for 3.00 0.001 for (1.2.4)
G.5 DISCUSSION ABOUT THE SETTING OF GROUND-TRUTH SHAPLEY VALUES.
This section discusses the ground truth of Shapley values in the extended Addition- Audition--uutiplicaatinn dataset (Zhang al., (202cc), which is used in Section |5. of the main paper. In order to verify the correctness of the Shapley values obtained by the optimal baseline values in this paper, we conducted experiments on the extended Addition ciattioonMulipplicationn dataset (Zhang et al. 2021c) with ground-truth Shapley values.
The
dataset in (Zhang al. 2021c) contained functions that only con- sisted of addition and multiplication operations. For example, f(x) Z1d2 + Z3I1 where each input variable 2, E {0, 1} was a binary variable. Given x = [1, 1, the function contained two salient causal patterns, i.e. {x1,42} and (I3,14), and their benefits to the output were U(g U(gg.aa)
1, respectively. According to (Harsanyi, 1982). the Shapley value is a uniform distribution of attributions. Therefore, the effect of causal pattern was supposed to be uniformly assigned to variables in the pattern. Thus, the ground-trutt Shapley values of variables were 01 = 22 = 1/2, and d3 = ô. = 1/2. However, if the input x = [1, 0, 1, 1], then the pattern {2 was deactivated and U [z),==) = O. In this case, Φ1 Φ2 = 0 while Φ3 = 2o = 1/2.
According to the analysis in Appendix G.4 ground-truth baseline values in the Addition- Multiplication dataset were all zero. Then our method is equivalent to the zero baseline values. Therefore, in order avoid all ground-truth baseline values being zero, we added the subtraction operation. We also added coefficient before each term in the function to boost the diversity of functions. For example, (xx) 3. 3.201 + 1.50r - 1). This funetion also contained two causal patterns, but the ground-truth baseline values of variables were different from the aforementioned
function. Here, bn = bi = b3 = 0 and bj = 1. Given the input x = [1. 1, 1, O). all patterns 1.7. Ground-truth Shapley 3.2 + (-1.5)
were activated and f(x) _2 + U [m.
values of input variables were Φ1 = 82 = 3.2/2 and 63 = d. = -1.//2. However, for the input
= [1, 1, 1, 1], the pattern {x3, } was deactivated, thereby d3 = Φ1 = 0. Note that the above function can also be considered
3.2r14² + 1.50x344 - 1.5xg)) According to Occam's Razor, we follow the principle of the most simplified interaction to recognize causal patterns the function, iee. using the least number of causal patterns. Thus, we consider the above function fxx = 3.2 + (. - 1) containing two salient causal patterns.
contain three patterns (f(x)
Based on the extended Addition Muttippiaultiiiiiiiinnnn dataset, we randomly generated an input sample for each function in the dataset. Each variable x; in input samples were independently sampled following the Bernoulli distribution, i.e. p(r, = 1) = 0.7. Therefore, for the mean baseline, baseline values of different imput variables were all 0.7. For the baseline value based on the marginal distribution, which was used in SHAP (Lundberg and Lee 2017). p(x() ~ Bernoulli Then, compared the accuracy of the computed Shapley values of input variables based on zero baseline values, mean baseline values, baseline values in SHAP, and the optimal baseline values defined in this paper, respectively. The result in Table 4 of the main paper shows that the optimal baseline values correctly generated the ground-truth attributions/haapley values of input variables.
G.6 EXPERIMENTAL RESULTS ON THE MNIST AND THE CIFAR- 10 DATASETS
Experimental results on the MNIST datasets. Experimental settings on the MNIST dataset have been introduced in Section 5.2 of the paper. Figure 9 shows the learned baseline values the
22

---

Published as a conference paper at ICLR 2023
Using the loss Lihaplly.
Using the loss marginnal
Zero-init
Mean-ini
Zero-init
Mean-init
1.00
0.00
Figure 9: The learned baseline values on the MNIST dataset (better viewed color).
Zero Baseline value
Mean Baseline value
Baseline in SAGE magnitude x2
Ours Ishaapey zero-init
Ours hihapley mean-init
Ours Iarrgnna zero-init
Ors Lmarginal mean-init
Input image
Blurring image
Baseline
in SHAP
0.3
-0.3
Figure 10: Shapley values computed with different baseline values the MNIST dataset.
MNIST dataset by using Shhpley and Imarginal in Section the paper, respectively. Figure 10 shows the computed Shapley values using different baseline values the MNIST dataset. Compared to zero/mean/blurring baseline values, the learned baseline values by our method removed noisy variables the background, which were far from digits in images. Compared SHAP, our method yielded more informative attributions. Shapley values computed using baseline values in SAGE were dotted. In comparison, our method generated smoother attributions.
Experimental results on the CIFAR-10 datasets. First, let us clarify the experiment settings on the CIFAR-10 dataset. We split each image into 8 8 grids. Each grid contained 4 4 pixels. which shared specific RGB color as their baseline value. Thus, we needed to learn total of x 8 x color values as baseline values. We learned baseline values by using the loss /Shplley Then, we computed Shapley values for grids by using zero baseline values and computed Shapley values by using the learned baseline values for comparison. Figure [T] shows that Shapley values computed with the learned baseline values mainly focused objects in images. In comparison, Shapley values based on zero baselines values mainly focused on the background.
Input image
Figure 11: Shapley values com- puted with different baseline val- ues. Images were selected from the CIFAR-10 dataset.
using baseline values learned by Lshapley
using zero baseline values
G.7 EXPERIMENTAL RESULTS ON THE UCI SOUTH GERMAN CREDIT DATASET.
This section provides experimental results on the UCI South German Credit dataset (Dua and Graff 2017). Based on the UCI datasets, we learned MI.Ps following settings in (Guidotti et al. (018). Figure 12 compares Shapley values computed using different baseline values. Just like results on the UCI Census Income dataset, attributions (Shapley values) generated by learned bascline values are similar to results the varying baseline values SHAP and SAGE. However, the zero/mean baseline values usually generated confticting results with all other methods.
Besides, we also noticed that baseline values learned with different initialization settings (zero-init and mean-init) all converged to similar baseline values, except for very few dimensions having multiple local-minimum solutions, which proved the stability of our method. More specifically, input variable might have different optimal baseline values in real applications from different
23

---

Published as a conference paper at ICLR 2023
Zero baseline values
Meon baseline values
Values of sample
SHAP
Foreign worter
Telephonne
Peopl: kablc
Houaing
Other installmenn plans
Age
Prooert
Presentrrsstongg
BAGE
Lihhaplyy
Iuurginal zoro-int
Punscrulstarzz-ae
Inahiiment--nee
Employment duration
Sovings
Amount 0.011
Status
Shapley value
Figure 12: Shapley values computed with different baseline values on the UCI South German Credit dataset.
perspectives. For example, the function of ( the variable a had two equivalent baseline value +0.5.
G.8 COMPUTATIONAL COMPLEXITY OF THE PROPOSED LOSS FUNCTIONS
In this section, we will introduce both the thcoretical complexity of the loss function and the real complexity of approximating the loss real applications.
In terms of the theoretical complexity, the complexity of the loss functions Lshapley and Lmarginal in Eq. (7) is Dn - (*),, where denotes the number of samples in the dataset. n is the number of input variables in the input. Σ (m) denotes the computational complexity of all terms 2. (m) (zr(b) of different orders m < A.
In terms of real implementation, mentioned in Section 5.2 used the sampling-based approxi- mation (Castro et al. (209)) to compute the loss function. We randomly sampled K" input variables in each epoch, and we randomly sampled T contexts of each order m w.E. each variable to ap- proximate Φ. (m))) (x(b). Thus, the real computational cost of Lshaplcy and Lmarginal was reduced to DK -E T. where set T 100 in implementation.
We conducted an experiment to show the time cost of learning the optimal baseline values. We used the MLP, LeNet, and ResNet-20 trained on the income dataset, the credit dataset, the MNIST dataset, and the CIFAR-10 10 dataset. For the MNIST and the CIFAR-10 datasets, we randomly sampled K 100 pixels the variable set N compute the Loss Ishapley and optimize their baseline values. Table 12 reports the time cost of optimizing Shapley for an epoch w.r.f DNNs trained on different datasets. The time cost was measured by using PyTorch 1.10 Ubuntu 20.04, with the Intel(R) Core(TM) i9- 10-10900X CPU @ 3.70GHz and one NVIDIA Geforce RTX 3090 GPU.
Table 12: Time cost of learning baseline values for an epoch.
Dataset
Number input varialles
per epoch
computational until convergeniee
5-layer MLP
UCT Census Income dataset UCI South German Credit daraset
12
3.81s
Ia5min (1000 epochs)
5-layer MLP
5.58s
1h33min (1000 epochs)
LeNet
MNIST CIFAR-10
2k:2mmn opochs)
ResNer-20
32x
64.44s
3h44min (200 epachs)
H MULTI-ORDER SHAPLEY VALUES AND MARGINAL BENEFITS
In Section of the main paper, we propose the Shapley values of different orders of (m) and marginal benefits different orders (S). Furthermore, we find that high-order causal effects are only con- tained by high-order Shapley values and marginal benefits. Actually, the loss function marginal
24

---

Published as a conference paper at ICLR 2023
benefits Imargina is more fine-gra fine-grained than the loss function the multi-order Shapley value Shhppley This section provides proofs for the above claims.
First, the Shapley value Φ can be decomposed into the sum of Shapley values of different orders (m)) and marginal benefits different orders Av,(S), as follows.
n-1
Φ,
(9)
m==0
m==0
where the Shapley value of m-order of (m) marginal benefit (s;(s) def ((assui))- vxxs)
EsCNMO0sI== (((osuu))) =(xs)) and the
Proof:
[S][(n 1
[u( v(xs)]
n!
SCN
[S 118!) (u( -((x))
re!
m m=0 SC N,18|= =m
[S 1 (S!!)
(u(osu(t)) - c(20s)
(n - 1)!
SCN.SS =m
Esc = (((ss)))) - -((k))
(m)
Connection between multi-variate interactions and multi-order marginal benefits. The m- order marginal benefit can be decomposed as the sum of multi-variate interaction benefits, as fol- lows. Therefore, high-order causal effects Us are only contained in high-order marginal benefits.
Av;(S) Σ ULU() LCS
(10)
Proof:
right
ULU()
LC8
((xx)+ + (-11)44-(4leaaxur))))
LCS
KCL
KCL
E
(-1)
[u( x(xx)]
Σ
KC8 K
Σ
(-1))P)
P= K
PESK
[S|-|K]
[S |1s| [K]
Σ
Σ
(-1)²)
Au;(K)
% % [P]
RCB
p
= Σ
[1+ (-1)))1-1R1)) Auz(K)
% %Lep [P]
RCS
25

---

Published as a conference paper at ICLR 2023
18
[s| [K]
0- (v])+ + RSS
Σ
Σ
Σ pdd
(-1)p Sei(K)
= (S) = left
Connection between multi-order interactions and multi-order Shapley values. The m-order Shapley value can also be decomposed the of interaction benefits, as follows. Therefore, high-order causal effects Ug are only contained in high-order Shapley values.
o!
EsC (a) Sm
Σ Unueo
(11)
LES
Proof:
((m)
s|=m (8:
Unu(a
128
Σ
ULU()
LG3
26


Reviews:
[{"review_id": "zNBkRuDb3A", "reviewer": "Reviewer_5Nqt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Aiming at investigating how to represent the absence of input variables and verify the faithfulness of baseline values, this paper proposes to use causal patterns to examine whether the masking method faithfully removes information encoded in input variables and a method to learn the optimal baseline value. Experimental results have demonstrated the effectiveness of proposed method.", "summary_of_the_review": "The absence of input variables and the faithfulness of baseline values of Shapley value are good points.", "strength_and_weaknesses": "Strong Points:\n[1]It is a challenging and crucial task that how to represent the absence of input variables and verify the faithfulness of baseline values of Shapley value for interpretable ML.\n[2]The approach of estimating optimal baseline values for Shapley values can ensure the trustworthiness of the attribution.\n\nWeak Points:\n[1]The authors did not properly back up their claims with evidence somewhere. For instance, “However, we find that most existing masking methods are not satisfactory from this perspective.” and “However, empirically, this method actually introduces additional information to the input.” which lacks key references to support the claim.\n[2]In this paper, how to initialize the  in Eq. (2) which is used to mask variables to represent the absence is not report.\n[3]How to evaluate the accuracy of Shapley value? There need more details about the metric.\n[4]The computational time and details of software environment are not reported, especially the time cost or complexity of computing base value via the approximation-yet-efficient solution.\n[5]Many of the reference should be further improved. For example, few references miss page. In addition, the authors may try to discuss the existing work in published papers (rather than a number of preprinted references from arxiv).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GP2bW1oYyF", "reviewer": "Reviewer_bMFX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel idea based on minimizing causal patterns to find the optimal baseline values in feature attribution. ", "summary_of_the_review": "The paper proposes novel ideas with relatively weak theoretical support, but good empirical results. The proposed algorithm can be computationally intensive. ", "strength_and_weaknesses": "- Strength:\n1. The paper proposes a novel idea to find the optimal baseline values in feature attribution. Empirically it achieves very good results compared to other existing methods such as zero/mean baselines. \n- A side weakness here: Why was conditional method not compared for CIFAR-10?\n\n- Theoretical weakness: \n1. The authors propose their criteria of choosing baseline values based on the observation that \" incorrect baseline values generate\nnew causal pattern\". While this observation is quite intuitive, why is the converse true? That is, it is a bit hard to see why minimizing the number of causal patterns would provide a good baseline value. \n- Clarity weakness: \n2. A minor point: Theorem 3 needs to be more concise. Is it possible to move part of it into a different property, or remark, or derivations ahead of the formal theorem? \n- Complexity of the algorithm: There is a trade-off between complexity and accuracy when we approximate the optimization. In practice, the proposed algorithm could be computationally intensive to get good baseline values, in particular for complex problems with higher orders of interactions. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9uT38i-xonk", "reviewer": "Reviewer_EX6v", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper examines whether the masking method faithfully removes the information encoded in the input variables. Then the authors propose a method to remove the effect encoded in the input variables by learning optimal baseline values for Shapley values. Experimental results demonstrate the effectiveness of the method.", "summary_of_the_review": "This paper finds the optimal baseline values that help to remove the causal effect encoded in the masked variables. I do admit the contribution of this paper, but there are some clarification issues that need to be resolved.", "strength_and_weaknesses": "Strength\n \n- The experiments are well-established on different datasets. The performance shows the effectiveness of the proposed method.\n\n- The approximate-yet-efficient solution is nice to reduce the low-order causal patterns and the number of salient causal patterns.\n\nConcerns\n\n- The problem is not defined clearly. Could you clarify the relationship between the absence of states of the input variables and the absence of the causal effect of the input variables at the beginning? It seems that the masking method you seek is actually aiming to remove the causal effect of the masked variable. It's confusing to not have a clear description of the problem right off the bat of what you are to remove by masking.\n\n- The presentation and some clarification issues. Some fundamental concepts need to be clarified. For example, could you please explain what it means by ‘removing causal pattern’ and ‘introducing causal pattern’ when you first mention it? Also, in this paper, the masking method is considered as faithfully representing the absence states of input variables if it faithfully removes old causal patterns without introducing new causal patterns. But first, to make it clearer, could you please present this definition more formerly; second, for me, it is not that straightforward to define it from this perspective. For example, I am confused about why masking $forehand$ should remove the information of the whole pattern $\\\\{forehand, eyes, beak\\\\}$. Even the causal effect of $forehand$ can be removed, is there any proof here? Moreover, could you please give some explanation on why the masking method should be considered in this way?  Is there a loss of information except for the masked variable? If yes, is there any metric for such a loss?\n\n- What is the computational complexity of the proposed method to estimate the optimal baseline values?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster