Shapley Explanation Networks
Authors: Rui Wang
Xiaoqian Wang
David I. Inouye
Venue: ICLR 2021 Poster (2021)
Keywords: Shapley values
Feature Attribution
Interpretable Machine Learning

Abstract: Shapley values have become one of the most popular feature attribution explanation methods. However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training. Thus, we propose to incorporate Shapley values themselves as latent representations in deep models thereby making Shapley explanations first-class citizens in the modeling paradigm. This intrinsic explanation approach enables layer-wise explanations, explanation regularization of the model during training, and fast explanation computation at test time. We define the Shapley transform that transforms the input into a Shapley representation given a specific function. We operationalize the Shapley transform as a neural network module and construct both shallow and deep networks, called ShapNets, by composing Shapley modules. We prove that our Shallow ShapNets compute the exact Shapley values and our Deep ShapNets maintain the missingness and accuracy properties of Shapley values. We demonstrate on synthetic and real-world datasets that our ShapNets enable layer-wise Shapley explanations, novel Shapley regularizations during training, and fast computation while maintaining reasonable performance. Code is available at https://github.com/inouye-lab/ShapleyExplanationNetworks.

Published as a conference paper at ICLR 2021
SHAPLEY EXPLANATION NETWORKS
Rui Wang Xiaoqian Wang
David I. Inouye
School of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47906
fruiw, joywang, dinouye @euuruue. edu
ABSTRACT
Shapley values have become one of the most popular feature attribution explanation methods. However, most prior work has focused on post-hoc Shapley explanations, which can be computationally demanding due to its exponential time complexity and preclude model regularization based on Shapley explanations during training. Thus, we propose to incorporate Shapley values themselves latent representa- tions in deep theeesy making Shapley explanations first-class citizens the modeling paradigm. This intrinsic explanation approach enables layer- wise explanations, explanation regularization of the model during training, and fast expla- nation computation at test time. We define the Shapley transform that transforms the input into a Shapley representation given a specific function. We operationalize the Shapley transform neural network module and construct both shallow and deep networks, called SHAPNETS, by composing Shapley modules. We prove that our Shallow SHAPPNES compute the exact Shapley values and our Deep SHIAPNETS maintain the missingness and accuracy properties of Shapley values. We demon- strate on synthetic and Shapley explanations, novel Shapley regularizations during training. and fast com- putation while maintaining reasonable performance. Code is available at hhtpp 7/github. cwm htom/innuyy hnw.//noooeeooeeeee Sab/Shap eeeepylllaaaae lanat woNks
datasets that our SHAPNETS enable layer-wise
INTRODUCTION
Explaining the predictions of machine learning models has become increasingly important for many crucial applications such as healthcare, recidivism prediction, or loan assessment. Explanations based on feature importance are one key approach to explaining a model prediction. More specifically, additive feature importance explanations have become popular, and in Lundberg & Lee (201)). the authors argue for theoretically- grounded additive explanation method called SHAP based on Shapley values way to assign credit members of a group developed in cooperative game theory (Shapley) 1953). Lundberg & Lee (2017) defined three intuitive theoretical properties called local accuracy, missingness, and consistency, and proved that only SHAP explanations satisfy all three properties.
sereiically-roundded properties, exact Shapley value computation has expo- nential time complexity in the general case. To alleviate the computational issue, several methods have been proposed to approximate Shapley values via sampling (Strumbelj & Kononenko| (2010) and weighted regression (Kernel SHAP), a modified backpropagation step (Deep SHAP) (Lundberg &
Despite these elegant
Lee 2017). utilization of the expectation of summations (Ancona et al. 2019). or making assumptions on underlying data structures (Chen al., 2019). To avoid approximation, the model class could be restricted to allow for simpler computation. Along this line. Lundberg et al. (2020) propose method
for computing exact Shapley values for tree-based models such as random forests or gradient boosted trees. However, even if this drawback is overcome, prior Shapley work has focused post-hoc explanations, and thus, the explanation approach cannot aid in model design or training.
On the other hand, Generalized Additive Models (GAM) explored in Lou et al. (2012 2013); Caruana et al. (2013) (via tree boosting), Chen et al. 201 7a) (via kernel methods), and Wang et al. (2018); Agarwal al. (2020) (via neural networks) exposes the exact Shapley explanation directly. In particular, the output of GAM model is simply the summation of interaction free functions: faAM(a)) = E, fx(x),, where fa (-) are univariate functions
be seen interpretable model class that

---

Published as a conference paper at ICLR 2021
(2)
=
Ma, 22) = sin day
Impul
Stapley Reprigentatonn
Input
Shapley Representation
222
22
1a 62(1,
120
24
133
080
08
1.45
OCD
00-
200
-04
-16
-04
-2
05
211
0x(S.2)
(1.)
Figure 1: Shapley representations span beyond one-dimensional manifolds and depend on both the inner function and the reference values. In both groups, the gray-scale background indicates the respective function value while the rainbow-scale color indicates correspondence between input (left) and Shapley representation (right) along with function values- red means highest and purple means lowest function values. The red cross represents the reference values. More details in subsection 2.1
that can be arbitrarily complex. Interestingly, the Shapley explanation values, often denoted db, (at, J), for GAM are exactly the values of the independent function, i.e., Vs, (s, fGAM) the prediction and the corresponding exact SHAP explanation can be computed simultaneously for GAM models. However, GAM models are inherently limited in their representational power, particularly for perceptual data such as images in which deep networks are state-of-the- Thus, prior work either post-hoc (which precludes leveraging the method during training) or limited in its representational power (e.g., GAMs).
fs(x)). Hence,
To overcome these drawbacks, we propose to incorporate Shapley values themselves as learned latent representations (as opposed to post-hoc) in deep models-thereby making Shapley explanations first-class citizens the modeling paradigm. Intuitive illustrations of such representation are provided in Fig. and detailed discussion in subsecctiionn 2.1 We summarize our core contributions follows:
We formally define the Shapley transform and prove simple but useful linearity property for constructing networks.
We develop novel network architecture, the SHAPNETS, that includes Shallow and Deep SHIAPNETS and that intrinsically provides layer-wise wise explanations (i.e., explanations at every layer of the network) in the same forward pass as the prediction.
We prove that Shallow SHAPNET explanations are the exuct Shapley values-thus satisfying all three SHAP properties and prove that Deep SHAPNET explanations maintain the missingness and local accuracy properties.
To reduce computation, we propose an instance-specific dynamic pruning method for Deep SHAPNETS that skip unnecessary computation.
We enable explanation regularization based Shapley values during training because the explanation is latent representation in our model.
We demonstrate empirically that our SHAPNETS can provide these new capabilities while maintaining comparable performance other deep models.
Dedicated Related Works Section We present related works above and in the text where appro- priate. Due to space limit, we refer to Appendix for a dedicated literature review section.
2 SHAPLEY EXPLANATION NETWORKS
Background We give short introduction to SHAP explanations and their properties as originally introduced in Lundberg & Lee 2017). Given a model f Rdd ト IR that is not inherently interpretable (e-g., neural nets), additive feature-atrribtiion methods form linear approximation of the function over simplified binary inputs, denoted z E (0, 1)d, indicating the "presence" and "absence" of each feature, respectively: i.e., local linear approximation 77(z) dg + Σ different ways to model "absence" and "presence". in this work, we take a simplified viewpoint: "presence" means that keep the original value whereas "absence" means replace the original value with a reference value, which has been validated in (Sundararajan Najmi (2020) as Baseline Shapley. If we denote the reference vector for all features by r, then define simple mapping
2i=1 a,Zi. While there
2

---

Published as a conference paper at ICLR 2021
(1 2) O r, where denotes element- wise product (eg. (2, O, 1,1)) = [1, 22, 73, 24. Ø3)). A simple generalization is to group certain features together and consider including or removing all features in the group. Lundberg & Lee (2017) propose three properties that additive feature attribution methods should intuitively satisfy. The first property called local accuracy states that the approximate model 1) z should match the output of the model at the corresponding x, i.e., f(z) = 7(11 Σn= az. The second property called missingness formalizes the idea that features that are "missing" from the input æ (or correspondingly the zeros in 2) should have zero attributed effect on the output of the approximation, i.., 2, = 0 = 0, O. Finally, the third property called consistency formalizes the idea that if one model always sees a larger effect when removing a feature, the attribution value should be larger (see (Lundberg & Lee (017) for full definitions of the three properties and their discussions).
function between and x (2)
Definition (SHAP Values (Lundberg & Lee 2017)). SHAP values are defined as:
di (1,
Σ #EZ,
1|2111 (1-11)
d
[f (VV,, a,r U (i))) - (Var (2))),
d
where ||=|| is the & norm of z given a set 2,- {z (0. 1) 21 0) and = U (i} represents setting the i-th element in z to be I iisteed of 0.
2.1 SHAPLEY TRANSFORM AND SHAPLEY REPRESENTATION
For the sake of generality, will define the Shapley transform in terms of tensors (or multi- dimensional arrays). Below in Def. 2 we make a distinction between tensor dimensions that need to be explained and other tensor dimensions. For example, in image classification, usually explanation (or attribution) value is required for each pixel (i.c., the spatial dimensions) but all channels are grouped together (i.e., an attribution is provided for each pixel but not for each channel of each pixel). We generalize this idea in the following definitions of explainable and channel dimensions.
Definition 2 (Explainable Dimensions and Channel Dimensions). Given tensor representation E DxC = (di xd2xx- (((2 xe2x."), the tensor dimensions D = di x d2 that require atriiutiion will be called explainable dimensions and the other tensor dimensions C = C1 x 02 x will be called the channel dimensions.
As simplest example, if the input is a vector æ E [_dx1 then D = d and C = i.e., we have one (since C = 1) importance value assigned to each feature. For images in the space RRAx Axxe where h, w denote the height and width, respectively, the explainable dimensions correspond to the spatial dimensions (i.e., D = h x w) and the channel dimensions correspond to the single channel dimension (i.e., C' c). While the previous examples discuss tensor dimensions of the input, our Shapley transforms can also operate on latent representations (e.g., in a neural net). For example, our latent representations for image models could be in the space IR xe. Xex where the explainable dimensions correspond to spatial dimensions (i.e., D = h x w) and there are two channel dimensions (i.e., C = c1 x c2). Given this distinction between explainable and channel dimensions, we can now define the Shapley transform and Shapley representation.
Definition 3 (Shapley Transform). Given an arbitrary function f F.F:DDxC + OOO the Shapley transform Q: RDC x F + RRDXX' is defined as:
f)k = (x
Vi E
(2)
where In denotes the set of all possible indices for values captured by the explainable dimensions D (and similarly for ) and I, denotes the scalar function corresponding to the j-th output of f-
Definition 4 (Shapley Representation). Given function f E F as in Def. 3 and tensor B DxC,
we simply define the Shapley representation be: Z 2 2(x f) E R DxC
Notice that in the Shapley transform, we always keep the explainable dimensions D unchanged. However, the channel dimensions of the output representation space, i.e., C², are determined by the co-domain of the function f. For example, if f is a scalar function (i.e., C" 1), then the attribution for the explainable dimensions is scalar. However, if f is a velued function, then the attribution is a vector (corresponding to the vector output of f). A multi-class classifier is a simple example of an f that is vector-valued function (e.g., C" 10 for 10-class classification tasks). In summary, the Shapley transform maintains the explainable tensor dimensions, but each explainable element of D may be associated to tensor of attribution values corresponding to each output of f.
3

---

Published as a conference paper at ICLR 2021
2.2 SHALLOW SHAPNET
We now concretely instantiate our SHAPNETS by presenting our Shallow SHAPNET and prove that it produces exact Shapley values (see I and Theorem 2) proofs in/Appendix A).
Definition 5 (Shallow SHAAPNET). A Shallow SHAPPET G: IR I is defined as
9y(x) = sum!o) DD g(z, sum D) o (),
where g( S(x,) f) denotes the Shallow SHAAPNTT explanation, sum!Pi denotes the summation over explainable dimensions D. f BD + TRC is the underlying function, and C' is the required output dimension for the learning task (e.g., the number of classes for classification).
Remark 6 (Sparsity in active sets curbs computation). By itself, this definition does not relieve the exponential computational complexity in the number of input features of computing Shapley values. Thus, alleviate this in SHAPNETS, that depends only on a subset of input features, (e.g., ʃz (1), T2, F3) = 5x22 + 2g only depends on 22 and 2z3); we will refer to such subset the active set, A(f)) (A(f)) = {2,3) for the example above). In principle, we would like to keep the cardinality of the active set low: |.A(f}) < [Ipl for j E Icy, so that we can limit the computational overhead because features in the complementary set of A(fz) (i.e., the inactive set) have trivial Shapley values of O's for any input.
promote sparsity in the computation. Consider function fa
To enforce the sparsity by construction, propose simple but novel wrapper around arbitrarily complex scalar function fj (which can be straightforwardly extended to vector-valued functions) called Shapley module F that explicitly computes the Shapley values locally but only for the active ser of imputs (the others are zeros by construction and need not be computed) as in Fig. (left)
Definition 7 (Shapley Module). Given max active set size k < (Ipl (usually 2 S k < 4), an arbitrarily complex function f: R:DxC + parameterized by neural network such that [A(f) S by construction, and reference vector r E IR a Shapley module F; RDCC + Dx1 is defined as: F(x ((r) -
x))² ()..J 3; E A()) and (f(f.r) = O otherwise (by construction).
where Φ5 (,({,x) is computed explicitly via Def. 1 if
While the sparsity of the Shapley modules significantly reduces the computational effort, only small subset (i.e., the active set) of features affect the output of each Shapley module. Thus, to combat this issue, we consider combining the outputs of many Shapley modules that cover all the input features (i.c., A(f,) = Ip) and can even be overlapping (c.g.. A(fi) = A(f2) = 3)). We first prove a useful linearity property of our Shapley transforms. Without loss of generality, we can assume that the explainable dimensions and the channel dimensions have been factorized so that D= = djdzd3... ... and C = C10203 ... in the following lemma.
(2,3). (2,3),A(Ja)) A(fa) =
Lemma 1 (Linear transformations of Shapley transforms are Shapley transforms). The linear trans- form, denoted by matrix A RC" xC", , of a Shapley representation Z & f) DXC' is itself a Shapley transform for a modified function f:
AZT A (9(x,))) = (2 (xx (2 (z.:))) where f(x)= = Af(xx.
(4)
Given this linearity property and the sparsity of the output of Shapley modules (which only have k non-zeros), we can compute a simple summation (which is a linear operator) of Shapley modules very efficiently. In fact, never need to store the zero outputs of Shapley modules in practice and can merely maintain the output of f TR. DxC + C via summation aggregation. For this sparse architecture, we can rewrite to emphasize the linear transformation we use:
9y(z) ! Ω fz. sum o g(x;; / sum!?! o (AQ (M,
(5)
where f' is a vector-valued function, each f have small active sets (i.e., |A(fg)) < k), and the Shapley transform of fj is computed via Shapley modules. For any Shallow SHAPNET (including our sparse architecture based on Shapley modules and linear transform), we have the following:
Theorem 2. Shallow SHAPNETS compute the exact Shapley values. i.e., g(x; f) o())
We use Shapley modules inside both Shallow SHAPNETS and Deep SHAPNETS (subsection 2.3) to make our model scalable. This is similar in spirit with L-Shapley (Chen al. 2019), which computes local Shapley values as an approximation to the true Shapley values based on the assumed graph structure of the data, but we build such locally computed Shapley values directly into the model itself.
4

---

Published as a conference paper at ICLR 2021
Shapley Module, Fj
Shallow SHA.NET.
Deep SHAPNET, H
do Us,
FolFii
Fata
of
Fy(e))
Patas
Loyl an
ta
Lary
Magter
gm
By cnntmnctioon
Shapley Reperscuttation Zina d'a ahows. s12) 6.00.)))
toa Rutee
Us (s) =0
Edge weight
Edge weight of
Edge weight
Edge weight of
Figure 2: example of how we construct SHAPNET from simple Shapley modules (left) which explicitly compute the SHAP explanation values for the arbitrary function f5. Shallow SHAPNETS (middle) are based on computing many Shapley modules in parallel-in in particular, we show comput- ing all pairs of features-ad and then applying summation to aggregate the modules (where implicit zeros are not computed). Deep SHAPNET (right) are composed of Shapley transform blocks where the output explanation of the previous layer is used as input for the next layer. We show example using disjoint subsets for each Shapley module within a Shapley transform and then use a butterfly permutation (as in the Fast Fourier transform (FFT) algorithm) to enable complex dependencies as discussed in subsection 2.3
The edge weights shown here are direct results of Eqn. with d = 2.
2.3
DEEP SHAPNET
While our Shallow SHAPNET is quite powerful, does not enable inter-layer Shapley representations within deep model. To enable this, we construct Deep SHAPNETS by cascading the output of Shapley transform from Def. 3) with the only restriction that the output of the last Shapley transform the right dimensionality for the learning task (e.g., the output space the last Shapley transform is DDe, where e is the number of classes). Note that effectively the Shapley representation Z in Def. 4 from a Shapley transform can be seen input æ to the next Shapley transform, making it possible to cascade the representations.
Definition (Deep SHAPNET). Deep SHAPNET H is based on a cascade of Shapley transforms:
H. sum [D] h sum (2(4) (((-1) (2 (2(1)
where h(x)) E IR Dxc is the Deep SHAPNET explanation, c is the number of output classes or regression targets, (2 is the 1-th Shapley transform with its own underlying function f() and all the reference values are set to 0 except for the first Shapley transform (whose reference values will depend on the applicatioo).
To ground our Deep SHAPNETS, we present the following theoretical properties about the explanations of Deep SHAPNETS (proof in Appendix B).
Theorem 3. The Deep SHAPPET explanation h((x) defined Def. provides an explanation that
satisfies both local accuracy and missingness with respect to H.
Deep SHAPNET with Disjoint Pairs and Butterfly Connections is immediately obvious that the computational graph determined by the choice of active sets in Shapley modules dictates how the features interact with each other in a Shallow SHAPPET and (since Shapley transform) a Deep SHAPNET. For our experiments, we focus on one particular construction of a Deep SHAPNET based on disjoint pairs (active sets of Shapley modules do not overlap: ne A(G)) = O) in each layer and butterfly permutation across different layers to allow interactions between many different pairs of means that the cardinalities of the active sets of all the Shapley modules are set to 2, making the overhead for computing the Shapley values roughly 4 x that of the underlying function. An example of this construction can be seen on the right of Fig. 2 We emphasize that this permutation one of the choices that enable fast feature interactions for constructing Deep SHAPPEES from Shapley transforms. We do not claim it is necessarily the best but believe it is a reasonable choice if no other assumptions are made. One could also construct SHAPNETS based prior belief about the appropriate computational graph (an example for images below). Another possibility is to learn the pairs by creating redundancy and following by pruning discussed in subsection 2.5 and Appendix F.
to the Fast Fourier Transform (FFT) butterfly construction. This also
5

---

Published as a conference paper at ICLR 2021
Deep SHAPNET for Images Here describe one of the many possibilities to work on image datasets with SHAPNETS, inspired by works including Desjardins et al. (2013); [Dinh et al. (2017); Kingma & Dhariwal (2018). Still, the Deep SHAPNET here consists of different layers of Shapley transforms. We begin by describing the operations in each of the consecutive Shapley transforms, and then cascade from stage to the next (see subsection 1.3) for full details).
The canonical convolution operation is composed of three consecutive operations: sliding-window (unfolding the image representation tensor acquire small patches matching the the filter), matrix multiplication (between filters and the small patches from the images), and folding the resulting representation into the usual image representation tensor. We merely 1) replace the matrix multipli- cation operation with Shapley modules (similar to that in Desjardins et al. (2013)) and 2) put the
corresponding output vector representation in space Ie' back in the location of the original pixel (and performed summation of the values overlapping a pixel location just as canonical convolution operation would), since we have the same number of pixels. To create hierarchical representation of images, we use â-trous convolution (Yu & Koltun 2016) with increasing dilation as the Shapley transform goes from one stage to the next, similar to Chen et al. (2017b); Chen et al. (2018). To reduce computation, we can choose a subset of the channel dimensions of each pixels to take as inputs, similar to Dinh et al.] (2017); [ingma & Dhariwal (2018). For the final prediction, we do global summation pooling so that the final representation conforms to the number of output needed.
2.4 EXPLANATION REGULARIZATION DURING TRAINING
One of the important benefits to SHAPNETS is that regularize the explanations during training. The main idea is to regularize the last layer explanation so that the model learns to attribute features in ways aligned with human priors-e.g.. sparse smooth. This is quite different from smoothing the explanation post-hoc as in saliency map smoothing methods (Smilkov et al. 2017; Sundararajan ctt al. 2017 Yeh et al. 2019). but falls more into the lines of using of interpretations of models as in Ross et al. (2017) ; Liu & Avci (2019); Erion et al. (211)); Rieger et al. (2020); Tseng et al. (2020): Noack et al. (2021) ). For & regularization, the main idea similar to sparse autoencoders in which we assume the latent representation is sparse. This is related to the sparsity regularization in LIME (Ribeiro et al. 2016) but is fundamentally different because actually changes the learned model rather than just the explanation. Note that the explanation method stays the same. For foo regularization, the idea is to smooth the Shapley values so that none of the input features become too important individually. This could be useful in a security setting where the model should not be too sensitive to any one sensor because each sensor could be attacked. Finally, if given domain knowledge about appropriate attribution, different regularizations for different tasks can be specified.
2.5 CONDITIONAL DYNAMIC PRUNING
Corollary allows for pruning during inference time and learning during training time (Appendix El
(E) then zm!) 0 for all m l, where i is an index of certain certain,feature, and 2 and r are the output and reference value respectively for the E-th Shapley transform.
Corollary (Missingness in Deep SHAPNETS). If 2 -r,")
Corollary 4 (with proof in Appendix Cp is a simple extension of Theorem 3 if a feature's importance becomes O at any stage, the importance of the same feature will be 0 for all the stages that follow. In particular, the computation
a Shapley module could be avoided if the features in its active set are all zeros. In other words, we can dynamically prune the Shapley modules if their inputs are all zeros. In practice, we set threshold close to zero, and skip the computations involving the input features with importance values lower than the threshold. Note that the importance values for the features are conditioned on the input instances instead of a static pruned model (Ancona et al.) 2020). allowing more flexibility. Moreover, with the help of & regularization discussed in subsection 2.4 (which can applied on each ( we can encourage more zeros and thus avoid even more computation. An illustration can be found in Fig. where we mask the pixels with small & norm after normalization, and pruning results shown against overall performance of the model in Fig. 5] We draw similarity to prior works including (Sun et al. (2013); Zeng et al.] (2013); Han et al. (2015); [Baco]| (2015);
loannou et al. (2016); [engii al. (016)) heodorakopoulos al. (2017)) Bolukbasi et al. (2017)

---

Published as a conference paper at ICLR 2021
Image
Stage
Stage
Stage 2
Stage
Stage 4
Stage
Stage
Regoihe
2
2
2
Output
2
2
2
Layerf
2
2
1.00
0.75
-0.50
0.25
0.00
0.25
0.50
0.75
1.00
Layer-wise Shapley values (absolute value summation normalized)
Figure 3: A visualization of Corollary 4 and pruning in action: Notice 1) that once a pixel get to close to reference values at an earlier stage, it will stay that way for the rest of the computational graph and 2) & regularization promotes the sparsity and hence decreases the amount of computation required. The first row is retrieved from regularly trained model, the second row from model with & regularization on the last layer discussed in subsection 2.4 and the last row from the model with E regularization applied on all inter-layer representations.
EXPERIMENTS & VISUALIZATIONS
We will (I) validate that our SHAPNET models can be quite expressive despite the intrinsic explanation design, (2) demonstrate that our intrinsic SHAPNET explanations perform comparably or better than post-hoc explanations, and (3) highlight some novel capabilities. More details in Appendix !
Datasets First, we create a synthetic regression dataset by sampling the input from the uniform distribution where the true (synthetic) regression model is f II 0; + Σ, i 0.05c, where is sampled from a standard normal distribution. Second, we choose five worldworld datasets from Dua & Graff (2017): Yeast (d = 8) and Breast Cancer Wisconsin (Diagnostic) (d = 30).
[-0.5, 0.5
MNIST (LeCun & Cortes 2010), FashionMNIST (Xiao et al.) 2017). and Ci (Krizhevsky 2009) datasets to validate S SHAPNET on higher-dimensionll datasets.
SHAPNET model performance We first validate that our Deep and Shallow SHAPNET models can be comparable performance to other models-i.e. that our Shapley module structure does not significantly limit the representational power of our model. Thus, we define DNN models that are roughly comparable in terms of computation or the number of parameters- since our networks require roughly four times much computation as vanilla DNN with comparable amount of parameters. For our comparison DNNs, we set up a general feedforward network with residual connections (He et al. 2016). Note that for the vision tasks, we compare with two other models with our Deep SHAPNETS: a comparable convolutional-laye based model which shares the same amount of channels, layers, and hence parameters with only LeakyReLU activation (Xu et al. (0155) with no other tricks other than normalizing the input and random cropping after padding (same as our Deep SHAPNETS)) and the state-of-the-art
each dataset (Byerly et al., 202T Tanveer et al. 2020 Foret et al. 2021. The performance of the models are shown in Table and Table 2 in which the loss is shown for the synthetic dataset (lower is better) and classifcation accuracy for the other datasets (higher is better). While there is very slight degradation in performance, our lower-dimensional models are comparable performance even with the structural restriction. While there is some gap between state-of-the-arr for image data, our image-based Deep SHAPNET models can indeed do reasonably well even on these high-dimensional non-linear classification problems, but we emphasize that they provide fundamentally new capabilities explored later. See Appendix
Explanation quality We now compare the intrinsic SHAPNET explanations with other post-hoc explanation methods. First, because our intrinsic SHAPPEE explanations are simultaneously produced with the prediction, the explanation time is merely the cost of a single forward pass (we provide a wall-clock time experiment in Appendix F). For the limee-dimensional datasets, we validate our intrinsic explanations compared to other SHAP-based post-hoc explanations by computing the difference to the true SHAP values (which can be computed exactly or approximated well low dimensions). Our results show that Shallow SHAPNET indeed gives the true SHAP values up to
7

---

Published as a conference paper at ICLR 2021
Table 1: Model performance (loss for synthetic or accuracy for others, averaged over 50 runs).
Models
Deep SHAPNET
DNN (eq. comp.)
DNN (og. param.)
Shallow SHAPNET
GAM
Datasets
Synthetic (loss)
3.37e.3 0.585
3.93e.3
6.62e.3
3.1
3.36c-3
Yeast
0.576
0.575
0.577
0.597
Breast Cancer
0,959
0.966
0,971
0,958
0.969
Table 2: Accuracies of Deep SHAPNETS for images, comparable CNNs and sfate-of-the-art models.
Models
Deep SHAPNET
Comparable CNN
SOTA
Datasets
MNIST FushionMNIST
0.9950
0.9917
0.9984
0.9195
0.9168
0,9691
Cifar-10
0.8206
0.7996
0,9970
Table 3: Average difference from exact Shapley values for different models.
Explanations
Ours
Deep SHAP
Kernel SHAP (77)
Kemel SHAP (def.)
Kernel SHAP (ext.)
Models
Untrained models (100 independent trials each entry)
Deep SHAPNET
1.29c-06
2. 2.89c-05
1.42c+03
2.13e405
5,00c-11 1.05c-03
Shallow SHAPPNET
2.97c-07
0,802
0.0733
4.05c-03
Regression models on syathetic dataset (20 independeut trials each entry)
Deep SHAPNET
4199e-13
4.91e-03
3.05e-04
2.26e-05
Shallow SHAPPET
4. 4.19--08
3.88
4.05e-03
2.39c-04
2.75c-05
Classification models on Yeast dataset (50 independent trials each entry)
Deep SHAPNET 0.0504
0.307
0.01467
0
Shallow SHAPNET
2.20c-07
0,516
0,0132
Classification models Breast Cancer Wisconsin (Diagnostic) (50 independent trials each entry)
Deep SHAPPET
0.0167
0,120
0,0369
4.09c-03
8.05c-04 2.03c-04
Shallow SHAPNET
1.20-04
0.0581
0.0224
1.91e-03
Table 4: Explanation regularization experiments with Deep SHAPNETS (averaged over 50 runs).
Models
Yeast
Breast Cancer Wisconsin
Metrics
for Reg.
6 Reg.
No Reg.
for Reg.
6 Reg.
No Reg.
Coefficient of variation for abs. SHAP
0.768
1.23
1.05
1.28
NaN
2.04
Sparsity of SHAP values
0.003
0.00425
0.00275
0.429
0.841
0,209
Accuracy
0.592
0.592
0.587
0.957
0.960
0.960
numerical precision as proved by Theorem 2. and Deep SHAPNET explanations are comparable to other post-hoc explanations (result in Table 3 and more discussion in [AAppendix G). We will evaluate our nigh-dimensional explanations in the MNIST dataset based on 1) digit Hlipping experiments introduced in [Shrikumar al, (2017). where higher relative log-odds score better, 2) removing the least-k salient features, where lower fraction of change in the output is better, 3) dropping the top-k salient features and measuring the remaining output activation of the original predicted class, where ideally, the top features would drastically reduce the output logits-showing that our explanations do highlight the important features. We compare, with the same baselines for all values, Deep SHAPNET explanations with DeepLIFT (rescale rule in Shrikumar et al. (017)). DeepSHAP (Lundberg & Lee) 2017), Integrated Gradients (Sundararajan et al. 2017). InputxGradient Gradients (Shrikumar et al. 2016
and gradients (saliency as in Bachrens al. (2010); [Simonyan al. (2014)). From Fig. 4 Deep SHAPNET offers high-quality explanations on different benchmarks.
Layer-wise explanations With the layer- explanation structure, we can probe into the Shapley representation at each stage of a Deep SHAPNET as in Fig. 3Jand perform dynamic pruning. Hence we also provide set of pruning experiment with retaining accuracy shown in Fig. 5] visualization shown in Fig. 3] and discussion in in/ppendiix H, where we show how pruning the values in the Shapley representations of different layer changes model performance, and argue that a large portion of the computation can be avoided during inference time and the model can retain almost the same accuracy. This showcases the possibility of dynamic input-specific pruning. We note that though this has been done before, our models is, to the best of our knowledge, the only model capable of learning the importance of different computational components in tandem with the importance of each feature, performing conditional pruning during inference time and explanation regularization.
8

---

Published as a conference paper at ICLR 2021
Inlegrated Gradbents
SHARNET
InpatxGrad
- DeepLiFF
DepSHAP
Saliency
Log-ndik and
of change In output
0,004
Least salient removal
Remaining (ractiion of acstvaatio
1.0
Top salient removal
30
0,003
0.5
20
0,002
20
0.001
0.0
0.000
20
2000 4000 600
Number of pisels removed
Namber of pisels removeed
Figure 4: Our intrinsic Deep SHAPNET explanations perform better than post-hoc explanations in identifying the features that flip the model prediction that contribute most to the prediction as in figures on the left showing the results of flipping digits as introduced in [Shrikumar et al. (2017) and right showing the remaining activation after removing the top k features identified by each explanation method. While Deep SHAPNET explanations did not perform the best in the middle where we show the results after removing least k-salient features as introduced in Srinivas & Fleuret (2019), our model still scores the second. All results are measured on MNIST test set. More results show the same conclusion with statistical significance in Table
for digit flipping, Fig. II
Layer 1
Output
Regular
Output 8oo
1.0
1.0
1.0
Accuracy
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
0
20
40
60
80 100
% of pixels pruned
% of pixels pruned
% pixels pruned
of pixels pruned
Layer 1 2
Layer
Layer
Layer
Layer
Layer?
Figure 5: 1) Pruning values from earlier layers have at least equal or stronger effects on the perfor- mance than from later layers due to Corollary 4) 2) & regularized model performs better especially when the regularization is applied on all the layers instead of just the output; 3) Most of the values can be removed and retain the accuracy. We prune the values in the order from the least salient (in magnitude of & norm) to the most pixel by pixel. This is measured on the MNIST test set.
Model with regularization
6 regularization
Model with negularization
Figure 6: MNIST SHAPNET explanations for different regularizations qualitatively demonstrate the effects of regularization. We notice that & only puts importance on a few key features of the digits while foo spreads out the contribution over more of the image. Red and blue correspond to positive and negative contribution respectively. Details in oossbbecciion I.4
Explanation regularization during training We experiment with & and loo explanation regular- izations during training to see if the regularizations significantly affect the underlying model behavior. For the power-dimensione datasets, we compute the accuracy, the sparsity, and the coefficient of variation (CV, standard deviation divided by mean) for our Deep SHAPNET. We see in Table thtt our Lo regularization spreads the feature importance more evenly over features (i.e., low CV), E regularization increases the sparsity (i.e., higher sparsity), and both either improve or marginally degrade the accuracy. For MNIST data, we visualize the explanations from different regularization models in Fig. 6, and we perform the top-k feature removal experiment with different regularizers in Fig. which shows that our regularizers produce some model robustness towards missing features. Fig. shows & regularized model achieves the best result under pruning across different layers, especially when applied on all layers, further increasing the amount of computation to be pruned.
ACKNOWLEDGMENTS
R.W. and D.I. acknowledge support from Northrop Grumman Corporation through the Northrop Grumman Cybersecurity Research Consortium (NGCRC) and the Army Research Lab through contract number W91 INF 2020-221. W.W. acknowledges support from NSF IIS #1955890.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rishabh Agarwal, Nicholas Frosst, Xuezhou Zhang, Rich Caruana, and Geoffrey E. Hinton. Neural additive models: Interpretable machine learning with neural nets. arXiv preprint MXXx:2004..331222 2020.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 7775- 7784. Curran Associates, Inc., 2018. URL http:://paaaa www.s..iins.ccc..c cc/paper/ wttpssscpp..ccccppc///pppppp
towards soarrdd-rbbbut..m rowurttoootteeeeee pdf
het-:ovarreooooutttt-eerrreeeee- hwttt--ieeerrreeaiiiiin. with self explaiiila neura networks.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of
cradient-basedd attribution methods for deep neural networks. In International Conference on Learn- ing Representations, 2018. URL (https::: / wwwpreweew..weee////. net/ forum Sy
Marco Ancona, Cengiz Öztireli, and Markus Gross. Explaining deep neural networks with polyno- mial time algorithm for shapley value approximation. In Kamalika Chaudhuri and Ruslan Salakhut- dinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 272-281, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL
prw.srrsss.ee.... wwt.:ppppess.s.seeesee m wttp:ss.w..rpr.ss./e.e..
9a html
Marco Ancona, Cengiz Öztireli, and Markus Gross. Shapley Value as Principled Metric for Structured Network Pruning. 2xxv:2006.0177 [cs], June 2020. URL http:/ttt/aaxxi..oorg////// 6206
2006
01795 arXiv: 2006.01795.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1 46, July 2015. doi: 10. 1.1377//ourmal.pone poo..003014..
Bacon. Conditional computation in neural networks using a decision-theoretic approach. 2015.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. J. Mach. Learn. Res., 11: 1803-1831, August 2010. ISSN 1532-4435.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. In Workshop at International Conference Learning Representations, 2016.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 527-536. JoLR.org, 2017.
Xavier Boyen and Daphne Koller. Tractable inference for complex stochastic processes. In Proceed- ings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI'98, pp. 33-42, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
Adam Byerly, Tatiana Kalganova, and Ian Dear. A branching and merging convolutional network with homogeneous filter capsules, 2021.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemie Elhadad. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 1h ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, pp. 1721-1730, New York, NY, USA, 2015. Association for Computing Machinery. ISBN 9781450336642. doi: 1LL144/2773258.2788666
URU/ttts://aai.coom http://000...0011111511115
2783258. 2788613
Hong Chen, Xiaoqian Wang, Cheng Deng, and Heng Huang. Group sparse additive machine. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 197-207, 2017a.
10

---

Published as a conference paper at ICLR 2021
Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. L-shapley and e-shapley: Efficient model interpretation for structured data. In International Conference on Learning Representations, 2019. /ttpss/s////peeeeeeiee www wet.//orrmmmdd-SEeeo00
L. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. /EEE Transactions on Putlern Analysis and Machine Intelligence, 40(4))834-848, 2018.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. CoRR, Abs//106.05587, 2017b. URL http:
7/arxiv. owtp /06. 05587
Guillaume Desjardins, Karen Simonyan, Razvan Pascanu, and koray kavukcuoglu. Natural neural networks. C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in neural information processing systems 28, pp. 2071-2079. Curran Associates, Inc., 2015. URL nww..niiin..ooo 5953 nat ural caairralnnuuraaa neural
networks -eeereeee--------eeeeeee------ pdf
Laurent Dinh, Jascha Sohl-Diccstein,, and Samy Bengio. Density estimation using Real NVP. ICLR,, 2017. htt.Lp/pp.//peeeeeeeeeeeeee..eop
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http:://aachhha
ics. uci...... wwccc.duumm
Gabriel G. Erion, Joseph D. Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Learning
explainable models using attribution priors. CoRR, hbbs//990..166000 2019. URL htRphttp://aaa
org/ hog/abs/1905 10670
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021. URL https: /tttp::://ppeerevic.ce /ttpt///t////eeeeeeeeeeeeeeeeeee wow.eeeeeeoooummeeme 6Tmlddommmmmmmmmmmmmmmmm
Jochen Gast and Stefan Roth. Lightweight probabilistic deep networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Amirata Ghorbani and James Zou. Neuron shapley: Discovering the responsible neu- rons. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 5922-5933 Curran Asso- ciates, Inc., 2020. URL http::tts///ppcccediiiii/// neuri ps TT16P2afeeefraaee2226666666eee22
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti. Local rule-based explanations of black box decision systems. CoRR, abs/1 2018a. ht.lhtttpp///rrxs..rr///bbb
1000/ 10820.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), August 2018b. ISSN 0360-0300. doi: 10.1145/3236009, URL/https://doi http:///ooor///..1111 1145/ 3236009
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS 15, pp. 1135-1143, Cambridge, MA, USA, 2015. MIT Press.
K. He, X. Zhang. S. Ren, and Sun. Deep residual lcarning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, June 2016. doi: 10. 109/1/9R.2000..900
Yani Ioannou, Duncan Robertson, Darko Zikic, Peter Kontschieder, Jamie Shotton, Matthew Brown, and Antonio Criminisi. Decision forests, convolutional networks and the models in-between, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2014. httht // //arxiv.. org/ htt//bb//4422 6980 cite 11..1444..880mmmmnntt Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
11

---

Published as a conference paper at ICLR 2021
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convo- lutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and
R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215- 10224. Curran Associates, Inc., 2018. URL
n wttpp/errrs.iis..peppepeeeeeeeee
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http://yann
lecun. com/ exdb wxwp//nibmmmnn/
Frederick Liu and Besim Avei. Incorporating priors with feature attribution on text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6274 6283, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/t/65//////11111/// 1631. URLhttps:/ wwwp///wwwllccccc..ccccccoocccc. wwwp:/awwwwweeeg..eeeeo..e www.w....g///aabooorr////
Yin Lou, Rich Caruana, and Johannes Gehrke. Intelligible models for classification and regression. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ' 12, pp. 150-158, New York. NY, USA, 2012. Association for Computing Machinery. ISBN 17778145011440 466214666.
4 2339556
doi: 10. 10.11452239953022333566 URL hRLhhtts:://o/ org/
Yin Lou, Rich Caruana, Johannes Gehrke, and Giles Hooker. Accurate intelligible models with pairwise interactions. In Proceedings of the 19th ACM SIGKDD International Conference
Knowledge Discovery and Data Mining, KDD '13, pp. 623-631, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450321747. doi: 74.1445724875555248777799
URL htt.hhtt:///ddo http::/:://ooo.or/11111111111 wwt.oo...../.........
24477777
Scott M Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predic- tions. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnetr (eds.), Advances in Neural Information Processing Systems 30, pp. 4765-4774. Curran Associates, Inc., 2017. URL model 7062 -il-a--mified- hit.aaunnnieee--ppprh--------- to inw.rnieeeeetie. -otereee moceepp-rrrnngg-moell--eereeed
nips. wwwpssccpppppppeeeeeeeee cc/paper/
model
pdf
Scott M. Lundherg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1):56-67, January 2020. ISSN 2522-5839. doi: 10. 1003://42256--11-0133-9999 URL
W. James Murdoch, Peter J. Liu, and Bin Yu. Beyond word importance: Contextual decomposition to extract interactions from LSTMs. In International Conference on Learning Representations, 2018.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'11, 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 978160558.
Adam Noack, Isaac Ahern, Dejing Dou, and Boyang Li. An Empirical Study on the Relation Between Network Interpretability and Adversarial Robustness. SN Computer Science, 2(1):32, January 2021. ISSN 2661-8907. doi: 10. 10.10077/422999-000000000 URL https://doi..cooo
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http:::://apppp..eeeeee http:://ppppsss.eeirssseseseeeee 2015-pytorch-e an www.eattteeestteeeeeeeee steeiiieessyylll high wtwpre/rrmmmmceeeeeeeeeeaaaaaaea org.apaccccaaaaaia.tt.ettteetoer
Library.
pdd
12

---

Published as a conference paper at ICLR 2021
Gregory Plumb, Maruan Al-Shedivat, Ángel Alexander Cabrera, Adam Perer, Eric Xing, and Ameet Talwalkar. Regularizing black-box models for improved interpretability. 33: 10526-10536, 2020. URL BRLEtttp:///poocciiioo www.riinnn.nnennn..o...m.m.. wwtpappe//02/////...cp/2 file/ 770f8e4
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining. KDD 16, pp. 1135-1144, New York, NY, USA, 2016.
Association for Computing Machinery. ISBN 978 9781450342322. doi: 10.1145/299967222999777
URL http:h/pp///bblloogg/.0.111122/2
2939778
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In AAAI Conference on Artificial Intelligence, 2018. URL http:::/tt//t aaai
exg/oca/index
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8116-8126, Virtual, 13-18 Jul 2020. PMLR.
Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Vekee Right for the Right Reasons: Train- ing Differentiable Models by Constraining their Explanations. arXiv: 03717 [cs, stat], May 2017. (tRL:ttp::/// /tttt:://ariv.... org/ abs 0070003.3333 arXiv: 1703.03717.
tex.ids: rossiighniighikeesonS200111
Lloyd S Shapley. A value for n-person games. Contributions to the Theory of Games, 2(28):307-317, 1953.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. CoRR, 2016. http::////axii..oog/////////////
org. 1105.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Doina Precup and Yee whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3145-315515 International Convention Centre, Sydney, Australia, 06-11 11 Aug 2017. PMLR.
.magesss.llr........ wwwppppssss...opp shri kumar 7a html
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In Workshop International Conference on Learning Representations, 2014.
Chandan Singh, W. James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In International Conference on Learning Representations, 2019. URL https 7/openreviee ew net
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. CoRR, abs/1776.00825, 2017. htlttttpp///xii// abs/1706. 03825
Suraj Srinivas and François Fleuret, Full-gradient representation for neural network visualization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 4124-4133 Curran As- sociates. Inc 2019. RRL[/attpps:/prccedding .ww.iiinnnnn.nuuusneeereemee neurips wctppppere21///11eee...eeee/eeee 80537a 94 5c aaa 8Bcc htt.aa/5c....aaacccccccccee16ea.
pddf
Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. J. Mach. Learn. Res., 11:1-18, March 2010. ISSN 1532-4435.
Y. Sun, X. Wang. and X. Tang. Deep convolutional network cascade for facial point detection. In 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3476-3483, 2013. doi: 10.1 W0W/CCVR.2011.444
13

---

Published as a conference paper at ICLR 2021
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9269-9278, Virtual, 13-18 Jul 2020. PMLR.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3319-3328.
International Convention Centre, Sydney, Australia, 06-11 11 Aug 2017. PMLR. URL http1
//proceedings.. ml www.s...r......ee.e www.prrssssv.o paresss/v70/sundrrarss jan 1a.html
Muhammad Suhaib Tanveer, Muhammad Umar Karim Khan, and Chong-Min Kyung. Fine-tuning darts for image classification, 2020.
Ilias Theodorakopoulos, Vasileios Pothos, Dimitris Kastaniotis, and Nikos Fragoulis. Parsimonious inference on convolutional neural networks: Learning and applying on-line kernel activation rules. CoRR, ibs/1701.55221, 2017. URL /ttttp://axxi.....m// org/ wttp/ba//////eoooee/...ee/...be/ /101 05221
Alex M. Tseng, Avanti Shrikumar, and Anshul Kundaje. Fourier- www.tttrannffo-mom based attribution priors improve the interpretability and stability of deep learning models for genomics. 34th Neural Information Processing Systems, June 2020. doi: 10. 00.1101/202000611..44777
URL http:
Bengt von Bahr. On sampling from a finite set of independent random variables. Zeitschrift für Wahhscheiilicchkeiistteen und Verwandte Gebiete, 24(4):279-286, December 1972. ISSN 1432- 2064. doi: 10. 10.100//0000691333...
Xiaoqian Wang, Hong Chen, Jingwen Yan, Kwangsik Nho, Shannon L Risacher, Andrew Saykin, Li Shen, Heng Huang, and for the ADNI. Quantitative trait loci identification for brain en- dophenotypes via new additive model with random networks. Bioinformatics, 34(17)1866-8874, 2018. ISSN 1367-4803, doi: 10. 1093/hiiinoommatcc//ty555.. URL / _eprint: http://ddddemmiccc.cccmmooooomod
Han Xiao, Kashif Rasul, and Roland Vollgraf. Rashion-mnist a novel image dataset for benchmarking machine learning algorithms, 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical Evaluation of Rectified Activations in Convolutional Network. M 00853 [cs, stat), November 2015. URL htth:://pxx/ii arXiv: 1505.00853.
exq/abs/1503 00853
Chih- Yeh, Cheng Yu Hsich, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in)fidelity and sensitivity of explanations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 10967-10978. Curran Associates, Inc., 2019. URL URL http:///pppppssssss..cccppeeeeee ineeinnfdee
and of
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR, 2016.
X. Zeng, W. Ouyang, and X. Wang. Multi-stage contextual deep learning for pedestrian detection. In 2013 IEEE International Conference on Computer Vision, pp. 121-128, 2013. doi: 10.1 109/CCV. 2013.22.
14

---

Published as a conference paper at ICLR 2021
A PROOF OF THEOREM 2 & LEMMA 1
Proof of Lemma 1 For each row of W - AZ, we can show that it is the Shapley value for f(() using the linearity of Shapley values (the original motivating axiom of Shapley values, see Shapley 1953 Axiom 4 & Axiom 5)):
wk Σ Eakolff
2 ok. f $(f(h),
Combining this result for all rows with Def. 3 we arrive at the result.
Now we are ready to present the proof for Theorem
Proof. For a general f. we have that () f(x) because [f x)) produces the Shapley values of by the definition of Shapley transform f2 (Def. 3) and Shapley values maintain the local accuracy property, i.e., Σier, dx(J, a) = [xx). Thus, g(;; f) = Sx f) = Qxx, Gy), which is equivalent the Shapley values of Gy. (Our proposed sparse SHAPNET architecture, denoted by f. based on Shapley Modules and simple linear transformation is merely special case of the underlying function f.)
B PROOF OF THEOREM
Proof. The proof local accuracy is trivial by construction since H(xx) _ sum)P)(h(z))) from Def. The proof of missingness is straightforward via induction on the number of layers, where the claim is that if x; = r; (i.e., the imput for one feature equals the reference value), then AE, (x((()) the pre-summation outputs of the Shapley modules in all layers corresponding to feature i will all be zero). For the base case of the first layer, we know that if x; rú then we know that z by the construction of Shapley modules. For the future layers, the inductive hypothesis is then that if previous layer representation z((-1) of feature is 0, then the representation from the current layer for this feature remains the same, i.c., z 0. Because the reference values for all layers except the first layer are zero, then we can again apply the property of our Shapley modules and thus the inductive hypothesis will hold for >>> 1. Thus, the the claim holds for all layers f, and the end representation h(x)) will also be zero zero-which proves missingness for Deep SHAPNETS.
(i.e..
C PROOF FOR COROLLARY 4
Proof. Given the existence a feature index Ic and a layer index & where z 0, where note that 0 is the reference values we set for inter-layer representations (see subsection 2.3) we can truncate the model to create a sub-model that starts at layer l, which, by the definition of missingness, will ensure, by the same proof as in Appendixx B the corresponding place in the representation to be zero (i.e., Vm > e, (m) 0).
D DEDICATED LITERATURE REVIEW
D.1 EXPLANATION METHODS
Our methodology sits in between post-hoc and intrinsic approaches by constructing an explainable model but using SHAP post-hoc explanation concepts and allowing more complex feature interactions.
Post-hoc feature attribution Post-hoc feature-attribution explanation methods gained traction with the introduction of LIME explanations (Ribeiro et al. 2016) and its variants (Ribeiro et al. 2018 Guidotti et al. 20 b). which formulate an explanation as a local linear approximation to the model near the target instance. LIME attempts to highlight which features Saliency map explanations for image predictions based on gradients also assign importance to each feature (or pixel) (Simonyan et al. 2014: Shrikumar al. 2017). SHAP explanations attempt
important for the prediction.
15

---

Published as a conference paper at ICLR 2021
to unify many previous feature attribution methods under a common additive feature attribution framework (Lundberg & Lee 2017 Lundberg et al. 2020). which is the main motivation and foundation for our work.
Layer-wise Relevance Propagation (LPR) (Bach et al. 2015) provides a way to decompose the prediction into layered-explanationss, similar our layer-wist explanation with the caveat that our inter-layer explanation corresponds to concrete input features.
From a neighbouring route, gradient- saliency map is another family of methods to highlight important components of each input (Ancona et al. (018). Input gradients (saliency as in [Baehrens) et al. (2000); Simonyan al. (2014) tries to identify the important features for prediction value by taking the gradient from the output w.r.t the input. Input Gradients (Shrikumar et al., (016) uses
the saliency map acquired by the input gradient method and performed element- wiise multiplication with the input instance, which tried to sharpen the saliency map. Integrated Gradients (Sundararajan jet al., (017), on the other hand, sets up a reference values from which the gradient is integrated the current input values, and integrate the input gradient taken along chosen path from the reference values to the current input instance.
Intrinsically interpretable models Intrinsically interpretable models are an alternative to post-hoc explanation methods with a compromise on expressiveness of the models. In particular, GAM (Lou it al.
2012) and its extension (Lou et al., 2013] Caruana et al., 2015 Chen et al. 2017a; Wang
al. 2018 Agarwal al. 2020) construct models that can be displayed as series line graphs or heatmaps. Because the entire model can be displayed and showed to users, it is possible for user to directly edit the model based on their domain knowledge. In addition, Alvarez Melis & Jaakkola (2018) proposed an architecture that are complex and explicit if trained with regularizer tailored to that architecture.
D.2 APPROXIMATING SHAPLEY VALUES
To alleviate the computational issue, several methods have been proposed to approximate Shapley values via sampling (Strumbelj & Kononenko| 2010) and weighted regression (Lundberg & Lee
2017. Kernel SHAP); modified backpropagation step (Lundberg & Lee 2017 Deep SHAP), utilization of the expectation of summations (Ancona et al. 2019 DASP), or making assumptions on underlying data structures (Chen al., 2019 L-Shapley and C-Shapley). In terms of computational complexity, Sampling based methods (Strumbelj & Kononenko, 2010; Lundberg & Lee| 2017) are
asymptotically exact in exponential complexity in the number of imput features. Deep Approximate Shapley Propagation (DASP) approximates in polynomial time with the help of distributions from weighted summation of independent random variables (von Bahr 1972) and assumed density filtering (Boyen& & Koller| 1998 Gast& & Roth 2018)) L-Shapley and C-Shapley do in linear with the Markov assumption on the underlying data structure, Deep SHAP approximates with an additional backward pass after the forward pass, and our method performs approximation with a constant complexity with a single forward pass, which additionally allows regularizing the explanations according to human priors and hence modifying the underlying model during training.
D.3 REGULARIZING ON THE EXPLANATIONS
Our SHAPNETS allow the users to modify the underlying function based on the explanations. This falls in line with previous works (Ross et al. 2017: Liu & Avei 2019 Brion et al., 2019 Rieger et al. 2020 T'sng et al. 2020), each of which adds an extra term in the loss function measuring the disagreement between the explanations and human priors. Ross al. (2017) proposed regularize the input gradients as explanations, which creates the need for -econd-order gradient. Liu & Avci (2019) uses Integrated Gradients (Sundararajan et al,, 2017 IG), which an extension to Shapley values (Sundararajan & Najmil 2020). Due to the use of IG, several samples (for one explanation) are additionally warranted. [Erion et al. (211)) proposed Expected Gradient to rid of reference values in IG and suggested the usage of a single sample. [Tenn et al. (2020) proposed to force the model to focus on wow-frequency component of the input data by applying Fourier Transform on input gradients. |Rieger et al.] (2020) adds to this line of work with Contextual Decomposition (Murdoch et al.) 20188) /iingh et al. (019) which avoids the need of computation of second-orde gradients and extra samples, and addition enable regularization interactions. Our work differs with
16

---

Published as a conference paper at ICLR 2021
these five in that do not need second order gradient and that are focusing the more mneortiially-ggounddee Shapley values.
The iaplanation-relatee regularizations in Alvarez Mclis & Jaakkola (2018): : Plumb et al. 2020);; Noack et al. (202T) differ with ours that the regularizers are used to promote interpretability and uses regularization to modify the underlying model to conform to prior belief. (Noack al. (2021) aims at connecting interpretability with adversarial robustness. Continuing that line of work, Plumb al. 2020) proposed EXPO, regularizer to help improve the interpretability of black boxes. EXPO was designed not for a specific structure but for all neural network models. We also draw close similarity to works that that attempt to smooth saliency map explanations (Smilkov et al.) 2017 Sundararajan et al.) 2017;; Yeh et al., (019) which acts on explanation methods to induce certain properties explanations but not the actual models.
D.4 CONDITIONAL PRUNING
In contrast to prior work where people perform static structure pruning based on Shapley values (An cona al. 2020), we compute the importance values on the fly during inference time.
Conditional Computation in Deep Neural Networks Conditional computation refers certain fashion where the neural network only activates a certain subset of all computations depending on the
input data (Bengio et al. 2016). This is desirable in low-power settings where the device can run only fraction of the entire model to save memory. Works investigating this computation style include Sun al. (2013); Zeng et al. (2013): Han al. (2015); Bacon [2015) et al. (2016); [engiio
et al.] (2016): hheooorkoopouoos et al. [2017)
Bolukbasi et al. 2017
E LEARNED FEATURE INTERACTIONS
With both & regularization and missingness in Deep SHAPNET discussed in subsection 2.5 can discuss how it is possible to learn the specific choices for Shapley modules each Shallow SHAPNET in Deep SHAPNET (how features interact in Deep SHAPNET). Note that have already discussed how we can perform instance-based pruning during inference.
To learn the interactions, we first create redundancies in each Shapley transform inside Deep SHAPNET. During training, we automatically gain the importance values for each features by construction, from which we will determine the modules that are worth computing by comparing input representation of the features in their active sets to Os. If the input representation is close enough to Os, we can simply ignore that module. Of course, the underlying function can always change during training, and we argue that we can slowly strip away modules from the latter layers to the first layers as the training progresses to allow for more expressiveness and more room to correct early mistakes by the training procedure.
F WALL-CLOCK TIME FOR EXPLANATION EXPERIMENT
To validate the efficiency of computing an explanation, we compare the wall-clock time of our SHAPNET explanations to other approximation methods related to Shapley-values including Deep SHAP and Kernel SHAP with different numbers of samples -00 77. and default (2 x d 2).. The wall-clock computation time in seconds can be seen in Table 5) We note importantly that this comparison is a bit odd because other Shapley-value explanation methods are post-hoe explanations, whereas SHAPNET is both model and an intrinsic explanation method. Thus, the most fair comparison in terms of time is to have the post-hoe explanation methods explain our SHAPPET model as seen in the first column of Table 5 Nevertheless,
also show the wall-clock times for note that that our
the host-hoc methods on our comparable DNNs columns two and three of Table 5 5)-note SHAPNET cannot explain other DNNs hence the N/A's. From the first column, method is inherently faster than the others when applied to our model. However, smaller model (i.c., DNN equivalent parameters) may come with faster explanations.
17

---

Published as a conference paper at ICLR 2021
Table 5:
Time for computing explanations (averaged over 1000 runs)
Models
Deep SHAPNET
DNN (eq. comp.)
DNN (eq. param.)
Explanations
Ours
20.47
N/A
N/A
Deep SHAP
46.10
83.38
8.56
Kemel SHAP (40)
60.78
130.04
11.52
Kernel SHAP (77)
72.61
201.88
13.53
Kernel SHAP (def.)
598.79
789.08
142.85
G LOWER NOWER-DIMENSIONAL COMPARISON BETWEEN SHAPNET EXPLANATIONS AND POST-HOC EXPLANATIONS
For lower-dimensional datasets, seek validation for the explanations k(()) provided by Deep SHAPNETS by comparing to the true Shapley values of our models (which can be computed exactly or aproximated well in low dimensions). Our metric is the normalized & norm between the true Shapley values denoted o and the explanation denoted y defined as:
| - 7/11
llé, 7) = d d.
where the number of dimension d is used as an effort to make the distance Note that we measure the average difference from vector-output classifiers by means discussed in subsection G. The experimental setup and results are presented in subsecttionn G.2
G.1 COMPARING SHAP APPROXIMATION ERRORS WITH VECTOR OUTPUT MODELS
To compare the approximation errors for every output of the model, we use a weighted summation of the explanations for different outputs, where the weights the models' output after passing into the soft-max function. Thus, we up weight the explanations that have a higher probability rather than taking simple average. Thus, in the extremes, if the predicted probability for label is zero, then the difference in explanation is ignored, while if the predicted probability for a label is one, then we put all the weight on that single explanation.
Concretely, for classification model C : I2 + R", where d is the number of input features and n the number of classes, we have the output vector of such model with an input instance Rd as C' (xx ) E R". For each output scalar value [C (z)ly, we compute the approximation of the SHAP values w.r.t. that particular scalar for feature indexed i, denoted by 7/ and hence its corresponding errors as measured in normalized & distance as discussed in Eqn. 8 ((),, The final error measure that we compare between classifiers & is simply weighted sum version of the normalized f;:
|softmax (C (C(x)) 40)
=
where the softmax is taken on the non-normalized output of the classifiers. Note that & is also taken on the raw output.
G.2 SETUP AND RESULTS
Because the explanation difference from SHAP is independent of the training procedure but solely depends on the methods, we compare the performance of the explanation methods on untrained models that have randomly initialized parameters (i.e., they have the model structure but the parameters are random)-thh is valid model it is just not a model trained on a dataset. Additionally, we consider the metric after training the models on the synthetic, yeast, and breast cancer datasets. Note that for even modest number of dimensions, computing the exact Shapley values is infeasible use a huge number of samples (2 with the kernel SHAP method to approximate the true Shapley values as the ground truth for all but the Yeast dataset, which only requires 2 samples for exact computation. Note that the extended Shapley value (ext.) requires 24 times of model evaluation for single explanation which is already extremely expensive, and thus included merely for comparison.
18

---

Published as a conference paper at ICLR 2021
Table 6: Standard deviation of Shapley approximation errors on different datasets
Explanations
Ours
DecpSHAP
Kemel SHAP (77)
Kernel SHAP (def.)
Kernel SHAP (ext.)
Models
Untrained models
Deep SHAPNET
0.734
5.23
7.67c+08
4.25c+09 8.02e-03
9.20c-03
Shallow SHAPNET
7.55e-07
1.57
0.124
2.98c-03
Regression models synthetic dataset
Deep SHAPNET
3.28c-03
5.15
3.13e-03
8.79c-05 1.62e-04
2.23e-05
Shallow SHAPNET
3.09c-04
1.28
4.75c-03
6.33e-06
Classification models on Yeast dataset
Deep SHAPNET
0.105
0.358
0.0288
Shallow SHAPNET
5.35c-07
0.894
0,0304
Classification models on Breast Cancer Wisconsin (Diagnoistic)
Deep SHAPNET
0.0104
0.0359
0.0987
3.61e-03 1.93e-03
1.72c-03
Shallow SHAPNET
0.0109
0.246
0.0141
2.91c-04
Results of the average difference can be seen in Table 3 From the results in Table 3, we can see that indeed our Shallow SHAPNET model produces explanations that are the exact Shapley values (error by floating point computation) as [heoremm states. Additionally, we can see that even though our Deep SHAPNET model does compute exact Shapley values, explanations are actually similar to the true Shapley values.
We also provide in Table the standard deviation of the approximation errors in Table These the same runs as those in Table
H PRUNING EXPERIMENT
We also conduct set pruning (set to reference values) experiment. This is, again, enabled by the layer-wise explainable property of Deep SHAPNET. Recent work has investigated the usage of Shapley values for pruning (Ancona et al.) 2020, Ghorbani & Zou 2020). but our work means to showcase the ability even further. Results are shown in Fig.
perform the pruning experiment with four MNIST-traiend models as discussed subsectiinn 2.3 and (ubsection 2.5) with no, output é1, layer é1, and fxx regularizer, respectively. The values are removed (set to reference) by the order of their magnitude (from smallest to largest). We can see that most of the values might be dropped and the model performance stays unhinged. Moreover, 4 seems to have the best robustness against pruming, which makes sense as it induces sparsity in explanation values. Future works to be done on the computational cost reduction under such pruning technique.
I MORE EXPERIMENTAL DETAILS
For all the experiments described below, the optimizers Adam (Kingma & Ba 2014) with default parameters in PyTorch (Paszke et al. 2019).
1.1 TIMING EXPERIMENTS
the timing is for the untrained (randomly initialized) models with 16 features. The numbers after Kernel SHAP indicates the number of samples allowed for Kernel SHAP's sampling method. Time is measured in seconds CPU with 1000 independent trials per cell as Kernel SHAP supports solely CPU from the implementation of the authors of Lundberg Lee (2017). The model of the CPU used for testing Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz.
For Table 5
We perform explanation on randomly initialized 16-feature Deep SHAPNET model for 1000 rounds. The 16-feature setup means will have (16/2) log, 16 32 Shapley modules inside the entire
19

---

Published as a conference paper at ICLR 2021
model spanned out in 4 stages according the Butterfly mechanism. The inner function f for every module in this setting is set as a fully-connected neural network of hidden layer with 100 hidden units. For the two reference models, the model equivalent in computation time has 8 layers of 1800 hidden units in each layer, while the model equivalent in parameters has 11 layers of 123 hidden units in each layer. The output of the models are all scalars.
1.2 YEAST AND BREAST CANCER DATASETS EXPERIMENT SETUPS
This setup applies for the experiments involving both datasets, including those experiments for model performance and those that showcase the ability for explanation regularization. The experiments on the two datasets were run for 10 rounds each with different random initialization and the end results were obtained by averaging. For each round, we perform 5-fold cross validation on the training set, and arrive at 5 different models, giving us 50 different models to explain for each datasets. For the two datasets, we do training epochs with a batch size of 32.
For the preprocessing procedures, we first randomly split the data into training and testing sets, with 75% and 25% of the data respectively. Then we normalize the training set by setting the mean of each feature to O and the standard deviation to 1. We use the training sets' mean and standard deviation to normalize the testing set, as is standard practice.
The inner functions of the Shapley modules are fully connected multilayer neural networks. The first and second stages of two models on the two datasets are identical in structure (not in parameter values). The first stage modules have 2 inputs and 25 outputs, with single hidden layer of size 50. The modules in the second stage have 100 input units and 50 output units with two hidden layer of size 100. The non-linearity is introduced by ReLU (Nair & Hinton| 2010) in the inner functions. The rest of the specifics about the inner function are discussed below. Note that the dimensionality of the output from the last stage differs with that of the input to the next stage, as is expected by Shapley module where the input has twice the dimensionality of that of the output.
Model for Yeast dataset The Yeast dataset has 8 input features and hence 4 Shapley modules at each of the log, 8 = 3 stages. Within each stage, the structure of the inner function is identical except for the parameters. For the Yeast dataset, the third (last) stage comprises four-layer fully-connected model with inputs from 50 x 2 = 100 units to the number of classes, which is 10, with two hidden layers of 150 units.
Model for Breast Cancer Wisconsin (Diagnostic) This dataset has 30 input features, which is not a exponent of 2, but we can still construct a Deep SHAPNET by setting two extra variables to zero. In fact, in doing so can simplify the Shapley modules since know which of the modules in the network always have one or both of its input being 0 at all times. To construct the model, we realize that the number of stages is 5 and the number of modules at each stage is 16. For the Breast Cancer Wisconsin (Diagnostic) dataset, in the the third stage of the model, we use model with 100 inputs units and 75 output units with two hidden layers of 150 units. The fourth stage has input units of 75 x 2 = 150 and output of 100 units with two hidden layers of size 200. The fifth (last) stage has a input dimension of 200, two hidden layers of 250 units and output of size 10.
I.3 SPECIFICS FOR VISION TASKS
I.3.1 STRUCTURE USED FOR VISION EXPERIMENTS
For all of the vision tasks, we use the same structure with the exception of input channel counts. We use a simple Linear-Lea (Xu et al. 2015) inside each Shapley Module, and the following are the output dimension of the linear layer: 1, 128, 128, 128, 128, 128, 128, 10, with being the input channel of images in the case of MNIST & FashionMNIST or 3 with 10.
À A-trous convolution A-trous convolution is used for learning hierarchical representations of im- ages. In model, the dilations are set as follows: 3, 5, 7, 9, 11, 13.
20

---

Published as a conference paper at ICLR 2021
Table 7: The boost in performance is statistically significant as, shown below, the t-scores for SHAPNETS vs. different methods
Explanations
Degrees of Freedom
Integrated Gradients
Gradeenss
DeepLIFT
DeepSHAP
Saliency
Digits
9
1008
28.0404
40.2129
28.7084
118.6641
1015194
4 1
981
54,4659
68.6895
63.1221
139,4350
122.9292
8 3++33
973
56.8543
67.6736
56.8097
128.9730
94.6011
8 6
973
41.5245
57.2258
49.9137
97,5942
90.1728
I.3.2 TRAINING RECIPE
Learning process As discussed before, all the models are trained with Adam in PyTorcl with the default parameters. The batch size is set 64 for all the vision tasks. No warm-up or any other scheduling techniques are applied. The MNShionMNIST & MNIST models in Table I are trained for 10 epochs while the MNIST models with which we investigate the new capabilities (layer-wist wise explanations & Shapley explanation regularizations) in Section 3aare trained with just 5 epochs. The Cifar-10 models are trained for 60 epochs.
Preprocessing For both MNIST dataset and FashionMNIST dataset, normalize the training and testing data with mean of 0. 1307 and standard deviation of 0.3081, I this gives us non-zero background and we would therefore from the background. Both MNST and MNIST share the same preprocessing pipeline: I) for training dataset, we extend the resolution from 28 28 to 36 36 and then randomly crop to 32 x 32, and then normalize. 2) for testing data, we simply pad the image to 32 x 32 and normalize them.
the reference values to O's, allowing negative or positive contribution
For Cifar- 10 dataset, during training, we first pad 4 pixels on four sides of the image and then randomly crop back to 32 32, perform horizontal flip with probability 0.5, and normalize the data with means 0.4914, 0.4822, 0.4465 and standard deviaiton of 0.2023, 0.1994, 0.2010 for each channel, respectively. For testing, we simply normalize the data with the same mean and standard deviation.
1.4 ILLUSTRATIONS OF EXPLANATION REGULARIZATIO EXPERIMENTS
We present visual representations dataset in Fig. 8 Fig. 9 and Fig. 10 In all three figures, the gray images from left to right are explanations laid out for all 10 classes from O to 9, where the red pixels indicate positive contribution and blue pixels indicate negative contribution, with magnitudes presented in the color bar below.
our models' explanation on MNIST (eeCun & Cortes 2010)
From an intuitive visual interpretation, we argue that our explanation regularizations are indeed working in the sense that Loo regularization is indeed smoothing the explanations (to put more weight on a large number of pixels) and that l regularization limits the number of pixels contributing to the final classification.
One can notice that, by comparing the color bars below the figures, the contribution magnitudes for Ex and & regularized models are lower than those the unregularized counterpart. We have expected this to happen as both regularization techniques require to make one or more of the attribution values to be smaller. However, we note that this change in scale of the attribution values should not impact the accuracy of the prediction as the softmax operation at the end will normalize across the outputs for all the classes.
More interesting discussions are provided in the caption of encourage readers to have a read.
ig. and Fg 10 and we strongly
I.5 MORE RESULTS ON DIGIT FLIPPING
More results on digit flipping are in Fig. 11 We also provide the t-scores and the degrees of freedom (instead of p-values because it is so statistically significant that p-values are extremely close to O's.) of the paired t-tests show that the explanation methods of ours provide boost in performance that
statistically significant in Table 7
21

---

Published as a conference paper at ICLR 2021
regular w. top-down order
regular w. reversed order
egular w. random order
regularized w. top-down
é regularized w. reversed
é regularized random
regularized top-down
Eu regularized w.
regularized random
of original accuracy
% of original accuracy
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
200
400
600
800
50
100
150
200
of features removed
# of features removed
Figure 7: Both f and foo regularizations often introduce some robustness to the model against feature removal (the dotted and dashed lines are often above the solid lines). Three different feature orders used: 1) most positive most negative (top-down), 2) most negative most positive (reversed), and 3) randomly chosen as baseline. l regularization puts most importance on the first 100 features with relatively low importance to other features (seen in both top-down and reversed). too regularization improves robustness when removing the top 50 or so features (as seen in the right figure which zoomed in the top 200 features).
22

---

Published as a conference paper at ICLR 2021
Image
Class 0 Class 1
Class
Class
Class 4 Class 5
Class
Class
Class
Class 9
7
7
7
7
7
2
2
2
2
2
2
2
/
/
/
/
1
/
4
/
4
9
5
9
0
6
9
0
1
5
9
7
3
4
0.6
0.2
0.2
0.4
Deep SHAPNET Explanations (unregularized)
Figure& 8: The explanations produced by our model trained without explanation regularization for all 10 classes. From left to right are class index (also corresponding digits) from 0 to 9.
23

---

Published as a conference paper at ICLR 2021
Image
Class
Class
Class
Class
Class 4 Class
Class
Class 7 Class
Class
7
7
7
7
7
7
7
7
7
7
2
2
2
2
2
2
2
2
2
2
/
/
1
/
/
/
/
6
O
O
O
O
0
0
4
4
4
4
4
4
4
4
4
4
/
/
7
7
7
7
7
7
7
7
7
7
6
6
9
7
O
0
0
6
6
6
6
6
6
6
6
9
9
9
9
9
9
9
9
9
9
0
0
0
0
0
0
0
0
1
J
J
5
5
5
5
5
5
5
5
5
5
9
9
9
9
9
9
9
9
9
9
7
7
7
7
7
7
7
7
7
3
3
3
3
3
6
3
3
8
8
4
4
4
4
4
4
4
4
4
4
0:15
-0100
0.05
0.10
0.15
Deep SHAPNET Explanations ( regularized)
Figure 9: E regularization promotes sparsity in feature importance, which, by comparison with Fig. and Fig. 10. one can easily argue has been successfully achieved. This shows that & regularization, combined with cross entropy loss, does (almost) remove negative contribution and only focuses on each digit's distinguishing features.
24

---

Published as a conference paper at ICLR 2021
Image
Class
0
Class 1
Class 2 Class 3
Class 4
Class 5 Class 6 Class 7
Class 8 Class 9
7
7
7
7
7
7
7
2
2
2
2
2
2
2
2
2
/ 0 4
/
/
/
1
0
0
0
O
6
0
6
4
4
4
4
4
1
6
6
9
0
6
9
9
9
0
J
1
1
5
5
9
9
9
9
9
7 3
7
7
7
7
4
-0.15
-0.10
-0,15
0.00
0,05
0.10
0.15
Deep SHAPNET Explanations (6s, regularized)
Figure 10: With Eoo regularization, the contribution among pixels are more laid out and less concen- trated than they would be other wise as shown in Fig. 8 The explanations produced by our model trained with Eo explanation regularization for all 10 classes. However, it is still easy to notice which part of the images contribute positively or negatively for a particular class.
25

---

Published as a conference paper at ICLR 2021
4 >1
40
Log-odäs between. and
30
20
0000
00
SHAPNET Integrated Inpd DxepLiFT DeepSHAP
Saliency
Gradients
8-33
40
Log-odia betweent and
30
10
SHAPNET Integrated Grad DeepLIFT DeepSHAP
Saliency
8-+6
Logodds beteeen and
SHAPNET Integrated InputxGrad DeepLIFT DeepSHIAP
Saliency
Grodients
Figure 11: Digit flipping results: Deep SHAPNET explanations perform well against post-hoc methods.
26


Reviews:
[{"review_id": "yhM-rc_h3yV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new model class designed to make SHAP value calculations more efficient. The proposed method exploits sparsity and additivity among intermediate values to provide fast exact SHAP values for shallow ShapNets, and fast approximate SHAP values for Deep ShapNets. This approach enables SHAP-based regularization during training, layer-wise explanations, and faster SHAP-based explanations with minimal loss in quality.\n\nThe approach seems promising, but I have several questions and concerns about the method and presentation. I'll start with questions about the method.\n\n- The authors write that SHAP values provide the unique explanation that satisfies a number of desirable properties; however, as pointed out by many works at this point [1, 2, 3], there are several proposed approaches for handling the held out features. Why do the authors choose to use single reference values rather than the interventional or conditional distribution? To what extent is the ShapNet approach amenable to other definitions of missingness?\n- A couple questions about the shallow ShapNet. (i) When performing a forward pass, is it actually necessary to calculate the SHAP values (the Shapley representation)? The sum of the SHAP values $\\phi_i(f^{(j)}, x)$ for $i = 1, \\ldots, d$ is equal to $f^{(j)}(x)$, so it seems that SHAP value computation (represented by $g$ or $\\Omega$) is not required unless one requires the explanation (e.g., for regularization). If that's the case, then a follow-up question is (ii) could any function whose output is the linear combination of intermediate values $f^{(j)}(x)$ be called a shallow ShapNet? And does your version of a shallow ShapNet yield faster explanations only because of the sparsity in the intermediate value functions?\n- Is the ShapNet method unable to work on models that are not the linear combination of intermediate values (e.g., the classification probability after softmax activation rather than logit)? If so, that would be worth mentioning briefly.\n- It seems that the recommendation is to determine the sparsity pattern in the value functions $f^{(j)}$ using the butterfly trick inspired by FFT; are there any other approaches worth considering that could improve either performance or compute time? Do the authors think that it would be worth learning the pairs that achieve the best performance?\n- Perhaps the most important question about Deep ShapNet explanations is how similar they are to SHAP explanations (e.g., from KernelSHAP). If possible, the metrics from Table 4 would ideally be shown in the main text, and perhaps extended to include results from the vision models. These metrics, along with Table 3, seem very promising.\n\nSome concerns regarding the presentation and experiments:\n\n- Strumbelj and Kononenko [4] proposed a sampling algorithm for SHAP values that should be cited in the second paragraph of the introduction, along with the other estimation algorithms.\n- Figure 2 is helpful but a bit complicated due to all the lines and bounding boxes, and the legend does not exactly correspond to the colors used. This figure may be worth revising and clarifying.\n- There's a lot of verbosity used to describe some straightforward ideas involving SHAP values. The \"Shapley transform,\" the \"Shapley representation\" and the \"Shapley module\" are all based on the simple idea of concatenating SHAP values into a matrix/tensor. The authors might consider presenting these ideas in a more straightforward manner.\n- The sum operator $\\text{sum}^{[\\alpha_i]}$ uses ambiguous notation. Shouldn't the superscript indicate the dimension to sum across rather than the number of entries along that dimension? It would be unclear how to apply the operator $\\text{sum}^{[\\alpha_i]}$ if there was ever $\\alpha_i = \\alpha_j$ for some $j$.\n- The generalized Shapley transform $\\mathcal{S}$ is not clearly defined in Definition 13. Perhaps the authors could provide a definition in terms of $\\Omega$? The use of this notation in the ShapNet composition (Eq. 6) seems to mask an underlying dependence on functions like $f^{(1)}, \\ldots, f^{(c)}$ at each layer. Would it be possible to use notation that shows these functions?\n- In Table 1, perhaps baseline model performance should be provided for the image datasets as a reference.\n- The text and caption for Figure 3 do not seem to indicate which dataset is used. The results look promising; could KernelSHAP be provided as an additional comparison method?\n- The authors point out the ShapNets enable (i) explanation regularizations and (ii) layer-wise explanations. Table 2, Figure 4 and Figure 6 show that the regularization works as intended, but this is a lot of space to prove this point; other results might be more important to include in the main text (mentioned earlier). Figure 5 shows that layer-wise explanations are possible, but the text does not explain how this is useful or when users would want this.\n\nDespite some concerns, I think this paper contributes some novel ideas that move us towards tractable + principled explanations for expressive models. \n\n[1] Aas et al., \"Explaining individual predictions when features are dependent: More accurate approximations to Shapley values\" (2019)\n\n[2] Kumar et al., \"Problems with Shapley-value-based explanations as feature importance measures\" (2020)\n\n[3] Merrick and Taly, \"The Explanation Game: Explaining Machine Learning Models Using Shapley Values\" (2019)\n\n[4] Strumbelj and Kononenko, \"An Efficient Explanation of Individual Classifications using Game Theory\" (2010)\n\n##########\nUpdate\n##########\n\nI'm pleased with many of the changes the authors have made in response to the reviewers' concerns. However, I'll mention a couple lingering concerns.\n\n- The authors have updated their paper regarding use-cases for layer-wise explanations, but it remains unclear whether this feature is impactful. While it is possible to perform instance-level pruning with ShapNets, is there any reason to do so? Finding features with low SHAP values requires actually calculating the SHAP values, which takes longer than just evaluating the function. Therefore, it seems that this technique would not lead to saving either time or memory. If the authors disagree, they may consider improving this aspect of the paper.\n- The authors have clarified that they use the \"reference values\" approach to holding features primarily because it is convenient. To my knowledge, no existing research actually advocates for this approach. Most research advocates for either the \"interventional\" [2] or \"observational conditional\" approach [1]. The authors cite a paper that mentions the \"reference values\" idea (Baseline Shapley) [3], but not even this work truly advocates for this approach. In my view, this is a rather severe limitation of the proposed approach, and the authors did not suggest that they see a way to overcome it. To call these \"SHAP values\" is almost misleading because it silently changes one of the core aspects of SHAP. As an example of the consequences of this limitation, any feature that is equal to its reference value will have a SHAP value equal to zero, but it is easy to image how such features can be informative (e.g., black regions in an MNIST digit, such as missing arcs on the left-hand side of a \"3\" that distinguish it from an \"8\").\n\nWhile it is very helpful to calculate SHAP values faster, this is a flawed version of SHAP that is not supported by existing research that considers the question of how to model missing features. Unfortunately, the proposed approach apparently lacks the flexibility to work with different notions of missingness (e.g., the interventional or observational approach). For that reason, I'm lowering my score by one point (6).\n\n[1] Frye et al., \"Shapley-based explainability on the data manifold\" (2020)\n\n[2] Janzing et al., \"Feature relevance quantification in explainable AI: A causal problem\" (2019)\n\n[3] Sundararajan and Najmi, \"The many Shapley values for model explanation\" (2019)", "extra_scores": {}, "extra_text": {}}, {"review_id": "bXI-iA8fERc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(I am raising my score by 2 points after the author response)\n\n--\n\nSUMMARY\n\nThe authors present a new type of network architecture where the each intermediate layer outputs the shapely contributions of the input $\\boldsymbol{x} \\in \\mathbb{R}^d$ (with respect to some reference $\\boldsymbol{r} \\in \\mathbb{R}^d$)  to $\\sum_{j=1}^c f^{(j)}(\\boldsymbol{x})$, where $f^{(j)}$ are functions that are parameterized by a neural network.  The functions $f^{(j)}$ can have vector-valued outputs (called \"meta-channels\" and denoted by the letter $n$ in the text), in which case the layer would have output dimensions $\\mathbb{R}^{n \\times d}$ (one entry for each meta-channel and input feature). Typically, computing shapely contributions to a function requires a number of operations that increases exponentially with the input size; the core trick that the authors use to make their networks scalable is to aggressively limit the subset of features (which they call the \"active set\") that are provided to each individual function $f^{(j)}$ - specifically, if only two inputs $(x_1, x_2)$, with reference values $(r_1, r_2)$, are provided to $f^{(0)}$, then the shapely contributions $\\phi(f^{(0)}, x_1)$ and  $\\phi(f^{(0)}, x_2)$ can be computed exactly from $f^{(0)}(x_1, x_2), f^{(0)}(x_1, r_2), f^{(0)}(r_1, x_2)$ and $f^{(0)}(r_1, r_2)$. In the case of deeper networks, the \"active sets\" (i.e. inputs) of the functions $f^{(j)}$ are disjoint pairs of size 2 that cover the full set of inputs, and from one layer to the next the pairs are permuted, which can allow the network to learn complex dependencies. When computing shapely contributions to subsequent layers, the \"meta-channels\" for a given input feature are grouped together - thus, every layer outputs estimated Shapely contributions for the original $d$ input features. To obtain the network output, the shapely contributions after the final network layer are simply summed over the $d$ input features. The authors also extend this setup to work on computer vision tasks, where parameters are shared between different input patches.\n\nThe authors benchmark their networks on two real-world tabular datasets and several computer vision tasks, and claim that their proposed Shapely Explanation Networks, or ShapNets, can achieve competitive performance while retaining explainability. On the tabular datasets, the authors demonstrate that their Shapely Explanation Networks provide good estimates of the true Shapely contributions for a fraction of the computational cost compared to methods like KernelSHAP. The authors also demonstrate how the explanations provided by ShapNets can be used for attribution regularization.\n\nSTRENGHTS\n\n- I think the idea of these ShapNets is very interesting, and the authors have clearly put considerable thought into how to design the ShapNets to make them efficient and scalable. The authors do well to draw a comparison to generalized additive models, which are easy to interpret but do not learn interactions; these ShapNets could potentially enable good explainability while also allowing the network to learn interactions\n- Estimating Shapely values through ShapNets is far more computationally efficient that KernelSHAP (table 3 of the supplement) and more accurate (with respect to the true Shapely values) than DeepSHAP, as shown in Table 4 of the supplement (however, agreement with the true Shapely values is not synonymous with the explanation being the most useful in a particular practical situation, as I discuss below)\n- Because the Shapely values are output by the network, they can also be regularized during model training\n\nWEAKNESSES\n\n(1)  The authors mention that they use an FFT-like butterfly permutation in order to allow the network to learn complex interactions between pairs of inputs, and they set the number of layers to be log_2 [number of features]. As diagrammed on the right of Figure 2, it does not look like this permutation is sufficient to allow every input to interact meaningfully with every other input - in particular, $x_1$ does not seem to be interacting (strongly) with $x_4$, and $x_2$ does not seem to be interacting (strongly) with $x_3$. The reason I put \"strongly\" in parentheses is because the only way $x_1$ can interact with $x_4$ is by modifying the Shapely contribution $g_2^{(1)}(x)$ of the input $x_2$ - but in the case where $x_2$ does not vary much (e.g. imagine it is always set to its reference value), that would altogether prevent the network from learning an interaction between $x_1$ and $x_4$. Also, it is counter-intuitive for a network to learn an interaction between two features by modifying the Shapely contribution of a third feature. It seems like the core issue is the absence of a hierarchical structure? The use of log_2 [number of features] layers implies a binary-tree-like hierarchical structure that seems to be missing in this setup.\n\n(2) One thing that can limit the applicability of ShapNet is that, for tabular data, TreeSHAP already provides a way to learn a powerful predictive model while also providing an efficient way to compute the SHAP values. Thus, in order to make a case for ShapNet on tabular inputs, the authors would need to show that ShapNet outperforms tree-based models. Table 1 shows the results of benchmarking on the Yeast and Breast Cancer datasets, but it looks like Generalized Additive Models achieve the best accuracy on these datasets (outperforming the DNNs as well as ShapNet)...thus, it seems that these datasets don't have complex interactions in them, and thus there wouldn't be a reason to prefer ShapNets over GAMs on these datasets (let alone prefer ShapNets over tree-based models). \n\n(3) Of course, tree-based models are not suitable for computer vision data, and ShapNets definitely have an edge-here - however, the benchmarking of explanation quality for the computer vision data seems to be limited to Figure 3, which used the \"drop in accuracy as features are removed\" as measure of explanation quality. This particular metric has issues in that the modified image could be out-of-distribution (see the ROAR paper by Hooker et al. - https://arxiv.org/abs/1806.10758). I would recommend that the authors instead either explore: (a)  the \"remove least k salient pixels\" method from the FullGrad paper: https://arxiv.org/pdf/1905.00780.pdf (b)  the \"digit flipping\" experiments used in the DeepSHAP/DeepLIFT papers, or (c) (best) train their models on the BAM dataset and quantify interpretation quality accordingly: https://github.com/google-research-datasets/bam. Also, I would be interested in seeing the timing benchmarks for these computer vision models (it seems that Table 3 in the supplement is for an untrained randomly initialized model with 16 features). I would also be curious to see how the contributions estimated by ShapNets compare to the true Shapely values for the computer vision datasets (Table 4 in the supplement seems to report this for tabular data only). However, I understand if it is too computationally expensive to accurately estimate the true Shapely values on vision data.\n\n(4) A note regarding Table 4 in the supplement: while it is good that ShapNets produce explanations that are similar to the true Shapely values (as this is their stated goal), we should not mistake \"agreement with true Shapely values\" as meaning the explanation is useful in a particular practical application - for that, different benchmarks would be required (such as the ones mentioned in the previous point).\n\n(5) The authors present \"Layer-wise explanations\" as a \"new capability\", but this seems a bit misleading; any post-hoc explanation method can be applied to give explanations for intermediate layers by simply treating the intermediate layer as the first layer of a truncated model, and using the activations of the intermediate layer under the reference input as the reference value of the intermediate layer (e.g. see https://captum.ai/api/neuron.html).\n\n(6) In a similar vein, the authors present \"Explanation regularization during training\" as a \"New capability\" but this is a bit misleading; explanation regularization during training has been performed using post-hoc attribution methods (as the authors acknowledge under \"Extended Literature Review\" in Appendix C). The authors don't seem to have compared their proposed explanation regularization during training to these earlier works.\n\n(7) The paper was very hard to read - it was only thanks to staring at Figure 2 that I managed to form a sense of what was going on, and I think most readers would similarly struggle. One point of confusion is that there are two kinds of outputs of each layer - the function output, denoted by the capital letter $G$ in Definition 11, as well as the shapely contributions, denoted by the lowercase $g_i$. The authors don't always clearly distinguish which \"output\" is being considered (e.g. the authors write \"we construct deep ShapNets by cascading the output of Shallow ShapNets from Def. 11\", but Def. 11 is for the function output denoted by the capital letter $G(x)$, whereas from Figure 2 it is clear that what is provided to subsequent layers is not $G(x)$, but rather the Shapely contributions $g_i(x)$.  Similarly, the discussion of \"meta channels\" is very confusing; when\"meta channels\" are first introduced in Remark 9, the authors describe meta-channels as the result of applying a Shapely Transform to a tuple of vector-valued functions - i.e. it is implied that meta-channels arise as part of the outputs of the ShapNet layers; however, the example of meta-channels that is given is the RGB channel of pixels, which is in the input. I eventually gathered that meta-channels are analogous to what one would normally think of as the \"channels/neurons\" in a neural network, because the actual \"channel dimension\" as it is described in this paper disappears due to the summation over the channels $c$ in Definition 11. I think the flow of the paper would be greatly improved if the authors stated their big-picture strategy up-front: that they can compute Shapely values efficiently because every parameterized function only operates on two features, and that they sum over many such functions in order to obtain the layer output (the analogy to a generalized additive model was helpful).\n\nMINOR\n\n- The caption of Figure 3 should specify which model/task the lines are being plotted for\n- The authors do not appear to have trained their models using early stopping; based on the methods description in the supplement (section G), it seems that the models were trained for fixed numbers of epochs. This does not, as far as I can tell, change the core results of the paper, but it is important context (e.g. helpful to know when evaluating  the change in accuracy due to regularization in Table 2).\n- In table 4, the authors wrote 1.42e+03 for KernelShap & Deep SHAPNet for the untrained models - I'm guess that was meant to be 1.42e-03?\n- The authors write \"DeepLIFT (scaled to get DeepSHAP)\" - can they clarify what they mean by this scaling? Are they referring to the Rescale rule of DeepLIFT? The Rescale rule doesn't refer to a scaling of the explanations; it's a particular type of modified backpropagation rule that was described in the DeepLIFT paper.\n- A suggestion regarding explanations that are computed with respect to the softmax logits, in case the authors are not doing this already: for each feature, it may be helpful to subtract the average contribution score across all classes from the contribution score to each individual class; that way, you can account for the softmax normalization, in that if a feature contributes equally to the logits of all classes then it effectively contributes to none of the classes (due to cancellation in the softmax). This point is discussed in the DeepLIFT paper in the section \"Adjustments for Softmax Layers\".\n\nEXPLANATION OF RATING\n\nWhile I think the core idea of ShapNets is very interesting and promising, I think the empirical results are currently somewhat lacking - in particular, I hope to see a clear use-case where one would prefer ShapNets over alternative methods. For this reason, I'm rating the paper at marginally below the acceptance threshold.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5S6TPwJeDVd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper proposes to incorporate Shapley values as latent representations in deep models. Specifically, the paper constructs Shallow SHAPNETs that computes the exact Shapley values. The paper also constructs Deep SHAPNETs that maintain the missingness and accuracy properties of Shapley values. The effectiveness of the proposed SHAPNETs is demonstrated through experiments on synthetic and real-world data. \n\nOverall, it seems to be a good idea to incorporate Shapley values into deep models and the proposed method seems to be reasonable. The empirical results have demonstrated the usefulness of the proposed method. The paper is also well-written and technically sound. I have some comments as detailed below.\n\n##########################################################################\n\nComments: \n\n- The main challenge for Shapley values is its computational complexity. The paper overcomes this challenge by forcing the active set of all Shapley modules to be size=2. While mitigating the computational challenge, this would limit the representation power of the model. The authors showed that this is not a big issue by providing a comparison in Table 1 on three datasets (Synthetic, Yeast, Breast Cancer). However, all three datasets are low dimensional and do not require a high representation power of the model. Therefore, I am not quite convinced that the proposed SHAPNETs have satisfactory representation power. A comparison of Deep SHAPNETs and the DNN models on the three high-dimensional image datasets (MNIST, FashionMNIST, Cifar-10) would better answer this question. \n\n- In principle, we can tradeoff between the representation power and the computational efficiency by varying the size of the active sets of the Shapley modules. Have the authors considered a comparison of SHAPNETs with different active set sizes? \n\n- In Table 1, Shallow SHAPNET has a better performance than the Deep SHAPNET. Why is that? ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Shapley values are an important approach in extracting meaning from trained deep neural networks, and the paper proposes an innovative approach to address inefficiencies in post-processing to compute Shapley values, by instead incorporating their computation into training.  There was a robust discussion of this paper, and the authors' comments and changes substantially strengthened the paper and the reviewers' view of it, to the point that all reviewers now recommend acceptance.  Some lingering concerns remain that the authors should continue to work to address.  Is the method of computing Shapley values used as the baseline in the paper really state-of-the-art, or artificially weak?  The empirical results were methodologically sound but not as strong as one might expect or hope.  These concerns detract somewhat from enthusiasm, but nevertheless the paper yields an innovation to a widely-used approach to one of the most pressing current research problems.  The reviewers had a number of smaller suggestions that should also be incorporated including more significance testing and reporting of resulting p-values.
Decision: Accept (Poster)