A Rigorous Study Of The Deep Taylor Decomposition
Authors: Leon Sixt
Tim Landgraf
Venue: Accepted by TMLR ()

Abstract: Saliency methods attempt to explain deep neural networks by highlighting the most salient features of a sample. Some widely used methods are based on a theoretical framework called Deep Taylor Decomposition (DTD), which formalizes the recursive application of the Taylor Theorem to the network's layers. However, recent work has found these methods to be independent of the network's deeper layers and appear to respond only to lower-level image structure. Here, we investigate DTD theory to better understand this perplexing behavior and found that the Deep Taylor Decomposition is equivalent to the basic gradient$\times$input method when the Taylor root points (an important parameter of the algorithm chosen by the user) are locally constant. If the root points are locally input-dependent, then one can justify any explanation. In this case, the theory is under-constrained. In an empirical evaluation, we find that DTD roots do not lie the same linear regions as the input -- contrary to a fundamental assumption of the Taylor Theorem. The theoretical foundations of DTD were cited as a source of reliability for the explanations. However, our findings urge caution in making such claims.

Published in Transactions on Machine Learning Research (11/2022)
A Rigorous Study Of The Deep Taylor Decomposition
Lcon Sixt
Department of Computer Science Prese Universität Berlin
Tim Landgraf
tim.
Deportment Computer Science
Preie Universität Berlin
Reviewed on OpenReview: https: // openreviee. net/forum? id- Y4mgmu90gV
Code Repository: https: //
Abstract
Saliency methods attempt. to explain deep neural networks by highlighting the most salient features of sample. Some widely used methods are based on a theoretical framework called Deep Taylor Decomposition (DTD), which formalizes the recursive application of the Tay- lor Theorem the network's layers. However, recent work has found these methods be independent of the network's deeper layers and appear to respond only to lower-level image structure. Here, investigate the DTD theory to better understand this perplexing behav- ior and found that the Deep Taylor Decomposition is equivalent to the basic gradient x inppu method when the Taylor root points (an important parameter of the algorithm chosen by the user) are locally constant. If the root points are locally input-dependent, then one can justify any explanation.
this case, the theory is under-constraineed In an empirical eval- uation, we find that DTD roots do not lie in the same linear regions as the input contrary to fundamental assumption of the Taylor theorem. The theoretical foundations of DTD were cited as source of reliability for the explanations. However, our findings urge caution in making such claims.
1 Introduction
Post-hoc explanations are popular for explaining Machine Learning models as they do not require changing the model's architecture training procedure. In particular, feature attribution methods are widely used. They assign saliency score to each input dimension, reflecting their relevance for the model's output. For images, the saliency scores can be visualized as heatmaps (see Figure 2).
Evaluating post-hoe explanations is challenging because it is inherently circular: As we do not understand the internal workings of the model, which we are trying to explain, we cannot judge the quality of the explanation. The situation is further complicated as many methods simplify the model's complexity to render explanations accessible the human eye. For example, most methods focus on the local neighborhood of an input sample, and rely on assumptions such as linearity (e.g. gradient-based methods) independence of the input features (e.g. approximation of Shapley values (Strumbelj & Kononenko, 2011; 2014; Lundberg & Lee, 2017; Kumar et al., 2020)).
These factors and the complexity of deep neural networks make it difficult to assess whether an explanation is correct or not. We can not disentangle failures the explanation method and unexpected behavior of the model. While it acceptable for methods introduce simplifications existence, purpose, and violation should be made transparent. In the best case, method would be based solid theoretical foundation providing guarantees regarding an explanation's correctness.
rely on assumptions, their

---

Published in Transactions on Machine Learning Research (11/2022)
Such theoretical foundation is the Deep Taylor Decomposition (DTD, Montavon et al. (2017)). DTD recursively applies the Taylor Theorem to the network's 's layers, and backpropagates modified gradients to the imput, thereby computing the input's relevance. It was used as theoretical foundation of LRP (Bach et al., 2015), a poplar method to explain image models. LRP was repeatedly advertised as a sound and reliable explanation technique (Montavon et al., 2019; Samek et al., 2021b; Holzinger et al., 2022). For example, Holzinger et al. (2022) stated: "The main advantages of LRP are its high computational efficiency [.-]], its theoretical underpinning making it a trustworthy and robust explanation method [..]] and its long tradition and high popularity [..]]."
However, Sixt et al. (2020) has shown that certain LRP and DTD backpropagation rules create explanations partially independent of the model's parameters: the explanation will remain the same even if the last layer's parameters are randomized. The theoretical analysis in (Sixt et al., 2020) revealed that the propagation matrices, which correspond to the layers' Jacobian matrices, are all positive and their product converges to a rank-1 matrix quickly. To obtain the saliency map, the result is usually normalized, and thereby even the last single degree of freedom is lost. Thus, the explanation does not change when explaining a different class or the parameters of the deeper layers is changed.
This perplexing behavior questions the consistency of DTD directly. While Sixt et al. (2020) described the convergence to the rank-1 matrix in detail, the failure was not related to DTD's theory such as the choice of root points' and the recursive application of the Taylor Theorem. Here, we fill this gap: Can we identify flaws in DTD's theory that would explain the perplexing behavior of ignoring the network's 's parameters? Does DTD provide trunsparency regarding its assumptions and guarantees about the explanation's correctness?
Before we approach these questions, we summarize the relevant background of the Deep Taylor Decomposition in Section 3. For completeness, we start with the well-known Taylor theorem and then discuss how the theorem connects to DTD's relevances. We then continue with stating the recursive application of the Taylor Theorem formally and recapitulate the so-called train-free DTD approximation, which allows to compute layer-wise relevances efficiently.
In section 4, we present our theoretical analysis of DTD. In particular, we contribute: (C1) proof that the root points must be contained in the same lincar region as the input; (C2) we generalize a previous observation about LRP, (Shrikumar et al., 2016): if the layers' root points are chosen locally constant w.r.t the layers' input, then DTD's relevances take a similar form as input x gradient;; (C3) DTD is under- constrained: if the root points depend on the layers' input, then the Deep Taylor Decomposition can be used to create any arbitrary explanation: (C4) we also find that DTD cannot be extended easily to analytic activation functions (e.g. Softplus), without introducing complex higher-order derivatives with the same order as the number of network layers.
In an empirical evaluation (Section 5), we applied the theoretical insights from the previous section and studied the train-free DTD approximation in several experiments: (C5) The train-free DTD does not enforce the root points to be located in the valid local lincar region of the network; (C6) We also validated this empirically using a small multilayered perceptron, where we found a substantial number of samples having roots located outside the valid local linear region; (C7) Additionally, we include a reproducibility study of (Arras et al., 2022) that claimed that DTD's explanations would not suffer from the problems reported in Sixt et al. (2020). This reproducibility study also highlights DTD's black-box character and how difficult it is to evaluate explanation quality empirically.
Given the theoretical and empirical evidence, conclude that DTD obscures its simplifications and violates its own assumptions. DTD is underconstrained and even allows justifying virtually any explanation.
2 Related Work
The thcoretical analysis of explanation methods is small research area. PatternAttribation (Kindermans et al., 2018) investigated the insensiliveness of DTD rules to input noise and then proposed way to learn
I Following Montavon et al. (2017), we name the points used for the Taylor Theorem root points. For example, for function f R, the first-order Taylor approximation f(x) f()) (*) 2), where & the root point.
2

---

Published in Transactions on Machine Learning Research (11/2022)
the root points from data. Other lines of work are the manipulation of saliency maps (Dombrowski et al., 2019; Viering et al., 2019; Wang et al., 2020), the runt comileecomlleitty of explanation methods (Waeldchen et al., 2021), or the explanations methods with provable guarantees (Chen et al., 2019).
Previous works have analyzed the theoretical properties various saliency methods. For example, the insen- stivity of Guided-Backproo (Springenberg al., 2014) was analyzed in (Nie et al., 2018), and (Lundstrom et al., 2022) found flaws in the theoretical motivation of integrated gradients Sundararajan et al. (2017). (Kumar et al., 2020) discussed the issues from independence assumption between imput variables, often introduced in sampling algorithms (Strumbelj & Kononenko, 2011; 2014; Lundberg & Lee, 2017). In (Shah et al., 2021), it was empirically analyzed and proven for specific dataset that the Gradient's s magnitude will not correspond to relevant features. Our work also analyzes the theoretical properties of saliency methods but differs from previous works as it focuses on the DTD and LRP methods.
Although different review articles (Montavon et al., 2018; 2019; Samek al., 2021b;a) and extensions of LRP and Deep Taylor (Binder al., 2016; Kohlbrenner et al., 2020; Hui & Binder, 2019; Ali et al., 2022) have been published, none discussed the theoretical issues brought forward in our manuscript.
3 Background
In this section, we provide the necessary background Deep Taylor Decomposition to understand the theoretical analysis in Section 4. We mainly reproduce the derivations given in (Montavon et al., 2017; 2018). If we comment on the derivations, we do this in the Remark sections.
3.1 Taylor Theorem for multivariate functions
Taylor Theorem for multivariate functions can be concisely stated using multi-index notation. A multi-index
a E N6 is a vector of non-negative integers (a = [ai,...,,aa axl). The following operations are defined as: ad aglaz!... ... agl, |o| =E 04 x" 21 22 ... ...2% and j= f f/ 0 0=22002222000000000000 .0% ....*))), zk), where
x R and f R$ - R. The following theorem is adapted from Folland (2002, Theorem 2.68):
Thcorem (Multivariate Taylor Thcorem). Suppose Rd IR is of differentiabilityy class Ch on an open convex set S. If e S and a S, then:
8 f(i) 5)a² gh (ar
((x)
(1)
ad!
lakk
where the remainder is given by:
9h (x,
x)
k
( 3)
(1 1)^-1
(or /on s(x (1 1) ())) 0" [(x)) Jdl
(2)
ad!
ja
k
As the Deep Taylor Decomposition focuses on neural networks with ReLU activations, we will mainly look at the first-order Taylor Theorem:
af(x) δx
f(x) f(x) +
(x a
-=
where ã is the root point, and la- denotes the gradient evaluated at the root point a. The higher order terms are zero due to the local linearity of ReLU networks. As the Taylor Theorem requires E C (i.e., all partial derivatives (z))//x must be continuous in the local neighborhood S), the root point must be within the same linear region as the input.
Definition (Linear Region). A linear region of function f Rdd - R is the set N2( of all points x' (x) that (1) have the same gradient at æ: fz(x) V f('), and (2) can be reached from x without passing through a point with different gradient, i.c., (f(a) * Vf(x).
In case of a ReLU network, the approximation error cannot be bounded when selecting a root point ă ş Ny (xx) as that linear region's gradient might differ substantially.
3

---

Published in Transactions on Machine Learning Research (11/2022)
3.2 Taylor Theorem and Relevances
In the previous section, we have recapitulated the Taylor Theorem. We now discuss how the Taylor Theorem can be used to compute imput relevances. A common approach explaining deep neural network is to contrast the network's output with similar point predicted differently. A user could then study pairs (x, f(x)) and (ø, [()) and relate the input differences to the output. differences. To guide the user's allention, it would be desirable to highlight which changes between æ and E were responsible for the difference the output. The first observation in (Montavon et al., 2017) is that the network's output differences can be redistributed the imput by using the Taylor Theorem. If the point I in the local neighborhood Ny(x)), we can use the first-order Taylor Theorem (equation 3) to write the difference [(x) ((t)
f(x) - [(Ø)- = 2f(x))
Ō::: x== (xc - x)
(4)
The relevance of the input R: Rdd - RRd is then defined to be the point-wise product of the partial derivatives with the input differences:
0f(x))
R(x))
x -i)
(5)
Ōx
While this would be a simple way to compute the relevances, the following reasons are given in (Montavon et al., 2017; 2019), to not directly use the Taylor Theorem on the network output:
1. Adversarial perturbations (Szegedy et al., 2013): Small input perturbations can lead to a large change in the output. Therefore, the difference in the output might be enormous but xx - ø| tiny and uninterpretable.
2. Finding a root point might be difficult: "TI is also not necessarily solvable due to the possible non-convezity of the minimization problem" (Montavon et al., 2017).
3. Shattered gradients (Balduzzi al. (2017)): "While the function value f(x) is generally accurate, the gradient of the function is noisy" " (Montavon et al., 2019).
Remark 1. We want to point out that the more general problem seems to be that the local lincar regions are tiny, or rather the number of linear regions grows exponentially with the depth of the network in the worst case (Arora et al., 2018; Xiong et al., 2020; Montufar et al., 2014). This restricts the valid region for the root point to small neighborhood around the input.
3.3 Deep Taylor: Recursive Application of Taylor Theorem
The main idea of (Montavon et al., 2017) is to recursively apply the Taylor Theorem to each network layer. Before present this in detail, we first we need to clarify the notation of an n-layered ReLU network shortly:
Definition (ReLU network). An n-layered ReLU network f Rdi - B is the composition of n functions f = fn o... fI, where each function fi IRd Rddy has the form fi(a;) = [Wiad]* and where [] is the ReLU activation.
Instead of directly calculating the relevance of the input as done in the previous section, we can apply Taylor Theorem to the final network layer and then apply the Taylor Thcorem again to the resulting relevance. By recursively applying the Taylor Theorem per individual layer, we can calculate the relevance the input. As the base case of the recursive application, the relevance of the network output set to the value of the explained logit (z)) 'ntigi
R" Rn+1(an+1) antlyg
(6)
where Ra+1 denotes the relevance of the n + 1-th network activation. We decided to use superscripts for the relevance functions their individual dimensions often index in Rn+1, Suppose that we already
4

---

Published in Transactions on Machine Learning Research (11/2022)
know the relevance function R Rd++ for the layer 1+1. We can then calculate the relevance of a; (the input to layer 1) to the j-th coordinate of (a++1)
aR (fi(ai))
B (ai+1) = Rb (fi(ā.)) +
(al ā=)(a)))
(7)
da:
a;=d:(a;)
where we used a1+1 fi(a:). The root point is selected dependency of the layers' input at i..,, it is a function ày Rd Rd. The total relevance of the input to layer / is given by the sum over all dr41 hidden neurons.
Definition 3 (Recursive Taylor). Given function ʃ RRd Rd which can be written as composition of functions f f1 o.... fn with fi : Rd Rd-,,, the imput to each function are denoted by a and an+1 specifics f's output. Additionally, root point function ā; : R4 - R is defined for each layer, which must only return admissible values (ا()) NR. (x) and ā;(a;) # ag Then, the base case is given by Rm+1(aan1) = autly and the relevance function R : RR - RR: of layer I of n is recursively defined by:
di+1
@ (fi(ai)
(a(a))
Σ
+ a 0(a
a a("(ai)
(8)
da:
a (0 (as)
The above definition corresponds to equation 6 in (Montavon et al., 2017). Except for the root point selection, which will be discussed in the following sections, definition 3 contains all information to implement the recursive decomposition using an automatic differentiation library. An exemplary pscudo-code can be found Algorithm 1. Before continuing with the approximations of the Deep Taylor Decomposition, we want to make a few remarks:
Remark (No Axiomatic Motivation). Only some vague arguments are provided to motivate the recursive application of the Taylor Theorem:
The deep Taylor decomposition method inspired by the coride-ond-conueee paradigm, and erploits the property that the function learned by deep network is decomposed into set of simpler subfunctions, either enforced structurally by the neural network connectivily, or occurring
resull of training. Montavon et al. (2017, Section 3)
In contrast, Shapely values (Shapley, 1951) are motivated by four axiomatic properties, which are uniquely fulfilled by the Shapely values. A comparable set of axioms with uniqueness result does not exist for the Deep Taylor Decomposition.
3.4 Deep Taylor Decomposition For A One-Layer Network and DTD's rules
In the previous section, we introduced the recursive application of the Taylor Theorem. For a concrete exam- plc, we will now discuss how DTD is applied to a onc-laycred network. will also explain the propagation rules of DTD. This subsection corresponds to Section in Montavon et al. (2017), and we will refer to the corresponding equations with the notation (DTD eq. 11).
The one-layered network consists of linear layer with a ReLU activation followed by a sum-pooling layer, ie. f: -dd R, f())= E, [Wa wW 61, where [1] is the ReLU activation. We will denote the output of the ReLU layer as h(x)) [Wx b]+, and the sum-pooling layer as v(h) = E, hy- For this subsoction, we will denote the relevance function with R" R², and Rd for the input, hidden, and output layer, respectively.
The relevance of the final layer simply given by the network's output (cquation 6; DTD eq. 8):
(h(x)) = v(h(z)).
(9)
DTD suggests to select a root point h such that y(h) = The advantage of y(h) = 0 is that the network's output y(h) is absorbed to the first-order term (i.e., f()) O equation 3). We then have y(h) (k) 2h th h) such that the network output is fully redistributed to the hidden layer's relevance. Additionally,
5

---

Published in Transactions on Machine Learning Research (11/2022)
the root point should be valid input to the layer. As y(h)'s imput comes from the ReLU layer h(x), it is positive and only h 0 solves Σ, hy O. The derivative ( (h)/ah = @y(h)/B I and therefore, we can use equation 8 to write the relevance of the ReLU layer's output (DTD eq. 10):
(R)
Rh
3 h
(10)
h-0
As h is the ReLU output, we can write Rh (x) also as (DTD eq. 11):
(x) = [Wx x + b]+
(11)
The next step is to connect the imput relevance Rd (x) with the ReLU neurons' relevances Rh. We will use equation 8 and apply the Taylor theorem to the relevance of each hidden neuron hos
d
(x)
(a)) (xxx)
R
Σ
+ (  -
(w, (000==#00) +())
(12)
dx
where we used that the derivative of the hidden neuron in w.r.t the input is the weight vector wy W Ba-
Relevance Propagation Rules For the root point ã, we could, in theory, select any point in the half- space wiã + bj > O, as they are all valid according to the Taylor Theorem. However, as it is beneficial to fully redistribute the relevance, DTD proposed selecting point that sets the j-th neuron relevance to zero, i.e. any point on the hyperplane w ã + b, = O. The non-differentiabblit at the ReLU hinge is resolved by picking the gradient from the case w,z + bj > O.
As there is no unique solution, DTD derives different points by starting at the input æ and moving along a direction vy such that h, = x tv, with e R. The root point Φ, is then the intersection of the line æ + tu with the hyperplane w x bj- O. Combining these equations yields t w w w o
w wd and therefore the root w
point x(6) = x
Uj. Substituting this into equation 12 yields:
eq. 6, 7)
w x w
wft+b U,
by 1,
d Σ
(x) = Σ
w
R (2)
(13)
wp Uj
2-1
While almost all choices of yield root point with R exists:
(excepl v 1 w)) a few special directions
The rule chooses the closest root point in L2 metric: By wy. This will yield the root point x i w. RM (x) and the following relevance propagation rule: R² Σー where wi = w Wj.
R
(x),
wo w
The 2 rule uses a direction that always yields a positive root: vy = Il w0 which is preferred for wi wlw,aaa R (x)
positive inputs (e.g. from ReLU activations). The resulting root point is a = x w [1w,
and the following relevance propagation rule: (² = Σn ب
- Rh (x), where 2 = L 20
The gamma rule proposed in Montavon et al. (2019) uses the search direction vy = 1 + 7w,, 20 Dx, where y RR.. The corresponding relevance propagation rule then: (x)
mytxx otly (x). In the limit 7 00, the gamma rule becomes the trule. 2 w3(+h257
special case is the LRPo rule which does not use any vector to find a root point but chooses 0. Although zero not valid root point in general, it was shown that LRP, corresponds to gradient x input Shrikumar et al. (2016); Ancona al. (2018); Kindermans al. (2016). The LRP. rule is an extension of LRP, that adds small E to increase numeric stability.
6

---

Published in Transactions on Machine Learning Research (11/2022)
3.4.1 Which rule should be chosen?
20
20
The computed relevance values depend substan- tially on the rule. For example, the LRP, rule can compute negative relevance values, whereas the 2 rule will always return positive relovance values. In Montavon et al. (2017, Sec. 4.1, 4.2, 4.3), the input domain was used as the primary selection criterion. For example, it was suggested to pick 2* rule for Ri and the w² rule for the do- main R. The input domain is not a sufficient se- lection for the root points, it does not provide unique solution. In the later work Montavon et al. (2019), other selection criterion were proposed for deep neural networks, which we will analyze in Section 3.5.1. For now, we can conclude that no principled way to pick the roots and rules exists.
10
10
100
-10
20
-2
-20
0
20
-20
20
(a) Non- Non-positive biases
(b) Unrestricted biases
Figure 1: Local linear regions of an randomly initialized neural network (3 layers, ReLU, inputs, 10 hidden neu- rons). The biases are initialized (a) non-positive and (b) unrestricted. The gradient are visualized as arrows for random selection of points.
3.4.2 Non-positive biases
In Montavon et al. (2017), it was proposed to constrain the biases of the lincar layer to be non-positive, i.c. by < 0. The main motivation was to guarantee that the origin is a root point of the function f. However, this not the case, as the following simple counter-example will show. Suppose the bias b -T. Then the function f(0) 0, as [Wō- I]] 0, but the origin 0 is not a valid root point as the gradient is zero there. However, any input with f(x) > 0 will have non-zero gradient and will therefore be in different local region. In Figure 1, we visualized the local regions of small 3-layered network for non-positive and unrestricted bias.
3.5 DTD for Deep Neural Neworks: The Training-
FFree Free Relevance Model
Applying the recursive Taylor decomposition to a one-layered network yielded a set of easily applied relevance propagation rules, which allowed to skip computing the root points explicitly. Of course, it would be desirable to skip the computation of roots for deep neural networks loo. As solution, Montavon el al. (2017) proposed so-called training-fres free relevance model. We follow the derivation from the review article (Montavon et al., 2018).
Let (at) be the relevance computed for an upper layer. Montavon et al. (2017) then makes the following assumption:
Assumption 1 (Positive Linear Relevance). The relevance of the upper layer R (a:+1) 1) can be written 1 a++1 O C++1, where c++1 g+ should be constant and positive vector.
As (at)(ar) = c @ [Way + bi]t, we can construct a so-called relevance neuron: (( = [W++1041 b
(14)
where we pulled into the layer's parameters: Wen1 W4+1 O C++1 and where C++1 [c c++]] is repeated version of 4411 and b bi+1 C CH+1-
This formulation is similar to the relevance of the hidden layer of the one-layer net in equation 11. The difference is that the root point and search direction will be based on the modified weights W441 and 6 Using w = WH = C+14 W1+1pap we can write the general relevance propagation rule of equation 13 as:
(((a)) = Σ j=1
d
w O ((((a)))))))
Σ
d.
wi Cuj f++1 (i((aa)) (
(15)
1n 03
uo! n,
where the C++,, canceled out. The corresponding root point would be: = - wTetby Uj. Interestingly, this deviation recovers the one-layer from equation 13. Thus, Montavon et al. (2018) argues that all
w
7

---

Published in Transactions on Machine Learning Research (11/2022)
the rules from the linear case (Section 3.4) can be applied to deep neural network too. This result can be easily extended to sum-pooling layers, as they are equivalent to a linear layer with the weights of value 1. Remark 3 (Is it correct that C₁ is constant?). A global constant Cg cannol exist, as changing the input vector can result a in totally different output, which would change the relevance magnitude. A local approximation of Ca could be correct if root points stays within the same local linear region where the function's gradient ff is locally constant.
Remark 4 (C5 no mechanism to enforce that the root point a0 E Ne (ai)?
). The corresponding root wf vy Vj- Will this root point be in the local region Ny(a:)
point to equation 15 would be: a.
w =b
of ag? Probably not, as there is no mechanism enforcing this. We test this in more detail in the empirical evaluation.
3.5.1 Which Root Points To Choose For A Deep Neural Network?
In Section 3.4.1, we already discussed that there is no principled way to select the root points or corresponding rules. For deep neural networks, DTD Montavon et al. (2017) originally proposed to pick the zt-rule for all layers except the first one. In a more recent work Montavon et al. (2019), it was argued to use a combination of LRPo, LRP-E and the y-rule. This was motivated by rather vague properties such as the "activations" entanglement", "spurious variations", and "spreading of relevance". It is concluded that: "Overall, in order to apply LRP successfully on a new task, it is important to carefully inspect the properties of the neural network layers, and to ask the human what kind of explanation is most understondable for him." Montavon et al. (2019, Sec. 10.3). Thus, the choice of the rules lies in the hands of the user who might choose any rule root point.
Kohlbrenner et al. (2020) introdueed a similar combination of rules as CRP-Cooposile. For the convolution layers, they used the -rule (or the LRPa110) and for the fully-connected layers the LRPo- An improve- ment over using the rule for all layers, they found that this combination of rules did not suffer from class-insensitivily; i.e., the saliency map do change when the explained output class is changed. However, it must be noted that this combination relies on the particular properties of the convolutional neural network. Specifically, there is little information mixing between more distant locations. Furthermore, the explana- tions are still insensitive to the later convolutional layers: the 2* rule creates a fixed saliency map for the convolutional layers, which, however, can be scaled by the output of the LRPo-rule. For example, if the final convolutional output has shape (8, 8) than the saliency map can be scaled in an 8x8 grid.
4 Analysis of the Recursive Application of the Taylor Theorem
In the previous sections, we recapitulated how DTD applics the Taylor Theorem recursively to lnc-layer and a deep neural network, and explained how the different propagation rules were derived. In this section, we provide theoretical analysis of the recursive application of the Taylor Theorem. In particular, the definition 3 from Section 3.3. As this definition is the most general formulation of the DTD theory, we ensure that the results of our analysis are applicable to all the propagation rules and are also not caused by one specific approximation but are rather inherent to the recursive application of the Taylor theorem.
study
The following propositions are proven in the Appendix A. The main idea of the proof is to apply the product rule to equation 8 and then analyze the individual terms.
4.1 Size of admissable regions for the root points cannot be increased
Proposition 1 (C1: Recursively applying the Taylor Theorem cannot increase the size of admissible re- gions). Given ReLU network f Rdc - B recursive relevance functions (a((a)) with I E {l,..., n. according to definition 3, and let & index the explained logit, then it holds for the admissible region N (a:) for the root points a{") of the relevance function R! that N (ai) S f...o...off (ai). (ai).
As the valid region for root points is restricted by the network f, we then we cannot evade the local region. This motivates a simple empirical test in Section 5.1: for each root point, we can check whether it is contained
8

---

Published in Transactions on Machine Learning Research (11/2022)
in the correct admissible region. This result questions the motivation that the distance |æ ž might be small T from 3.2, as this distance remains bounded by the local linear region of the network.
4.2 Locally Constant Roots Imply Equivalence of Recursive Taylor and Gradient x XIpput
It is well known that LRPo is equivalent to gradientxinppe for ReLU networks. This was first noted in (Shrikumar et al., 2016) and later also in (Kindermans et al., 2016; Ancona et al., 2018). We proof the following generalization for the recursive application of the Taylor Theorem in Appendix A.l.
Proposition 2 (C2). Let f R4 - B be ReLU nelwork, & be the index of the explained logil, and (ai) (with I 1..... n + 1) are recursive relevance functions according to definition 3. If the root points ār(a:) are locally constant w.r.t. the layer's s input (vlel... n. 0āz/0at== 0), then:
R(x) R(Ø) + (a) (x B)
(16)
where x 01 is the input vector and R(x) = (x).
The similarity with gradient x input can be seen when choosing a root point ã = 0 such that R(0)=0. =0. Then, the resulting relevance would be (aa O æ.
A fixed root point for each linear region would be valid and even desirable choice. For example, from an efficiency perspective, it would be preferable to scarch for valid root point in cach linear region only once. Or one might want lo select the one root point corresponding to the lowest network output. We also want to emphasize that no continuous constraint for selecting the root points exists. Jumps at the boundaries between the linear region are allowed. This result contradicts DTD's motivation described in Section 3.2, as it explicitly aimed to find something more "stable" than the gradient.
4.3 Locally dependent root points
As a next case, we will look at the more general case of root points depending locally on the layer's input: Proposition 3 (C3). For ReLU network S R& R with layers, and layer aclivalions fi-1(ar1), the relevance functions R (a--) of the recursive applications of the Taylor Theorem as given in equation 8 can be written as:
di di
@f((ar) 2016)
(am)
af((ar) ea: )
a (Ji(ai)) ōfi(a:) + (al-1-afh (17)
R' (a-1))
Σ
Σ
2
Ōas
The relevance function Rd-1 is determined by the next layer's Jacobian (fi(aa)/aa and also a term including rool point Jacobian (a /Cay- Although some directions are recommended, the choice of root point is not restricted per se. It is merely recommended to choose it within the layer's input domain" and it should minimize the explained relevance. Any root point could be chosen, as long as it is from the linear region (()) (at). However, this also means that (a-1() can be influenced arbitrarily by the root point's Jacobian. Therefore, any explanation could be justified. A theory under which anything can be justified is clearly insufficient.
4.4
Why Not Use Analytic Activation Functions (Softplus)?
For ReLu networks, the Deep Taylor Decomposition suffers from the problem that the rool point must be from the local linear region around the layer input a A possible solution would be to
analytic activation function, c.g. the Softplus activation. This would allow to choose any root point in R& although a sufficiently good approximation might require an unreasonable amount of higher-order terms. The main obstacle would be that with each decomposition, higher-order derivatives are accumulated:
PIn Montavon et al. (2017, Scction 4.1.), the different rules were selected based on the input domain. However, the 7 rule, introduced in more recent work Montavon al. (2019), can lead root points outside the ReLU's input domain RT, e.g. 7 the root point given by = RC (ar) which can become negative for large relevance values R' (æ).
9

---

Published in Transactions on Machine Learning Research (11/2022)
Table I: Empirical results of different DTD rules on a small neural network (3 layers, 10 input dimensions, 10 hidden dimensions). They show that the root points picked by the rules are not within the local region of the input, as each rule produced outputs below 100%. It is also the case that some root points will have the exact same network output as the original input.
Evaluation \ Rule
LRPo
? 1.0
w.
Same local lincar region [expected 100%]
41.20%
38.70%
37.70%
41.10%
Same network output [expected 0%]
14.62%
13.85%
14.01%
13.99%
Proposition 4 (C4). Let f R2dh - Rd+++ be neural network, contains analytic activation function, then each recursive application of the Taylor Theorem yields higher-orde order derivative of the form:
a :as
a (fi(ai)
far {ai a (ai))
(18)
das
(a))
Thus, for a n-layered network, would get n-ordered derivatives. The problem is that it is unclear how these chains of higher-order derivatives behave.
5
Experiments
5.1 (C6) OTD-Train-Fres
We implemented the train-free DTD using an explicit computation of the root points. The network consists of 3 linear layers, each with ReLU activation. The input and each layer has 10 dimensions. We initialized the network with random weights and used non non-positive biases, Montavon et al. (2017) suggested (even though we have shown that this has not the same consoquences as claimed in Montavon et al. (2017). As we are only interested in disproving claims, it is accepaaale to show that there exists one of neural network on which the DTD delivers inconsistent. results. Therefore, we also did not train the network on any task.
We provide pscudocode for our implementation in Algorithm 2. The main simplifications of the implemen- tation are that (1) the relevance of the higher layers is computed with the input of the layer and not the root point, and (2) the root points computed using the search directions outlined in section 3.4. We tested our implementation against Captum's implementation of the DTD (Kokhlikyan et al., 2020) and found the deviation to be less than x 10-8
Verifying that the two points are within the local region would require to show that the gradients are equal and that there is a path between the two points with all points on the path also having equal gradients. As the last part is more difficult to show, we only test the necessary condition of equal gradients. Therefore, we compare the gradient of the input with the gradient of the root points [V fā()) on 1000 random inputs. The input points were sampled such that it has network output greater than 0.1.
We reported the numerical results in Table 1. Less than 100% of all root points have gradient differences that are zero, thus root point exists which must be from a different local region. This violates Proposition 1, which requires all root points to be within the function's local region. Although we only show results on an exemplary l-yyeed network, the situation would only be worse for more complex networks as the number of local regions increases exponentially with layer depth (Montufar et al., 2014).
As second analysis, we tested how the root points influence the network output. One might assume that root point will alter the network output. However, this is not always the case (see row "Same network output" in Table 1). At least, these root points do not explain the output of the neural network.
5.2 (C7) Applying Sanity Checks to (Arras et al., 2022)
10

---

Published in Transactions on Machine Learning Research (11/2022)
What is the shape tiny brown rubber thing?
shape the big abiny object?
small gray rubber object; what shape it?
There large what shape is it?
object;
Input
sphere
cylinder
ephere
Input
cylinder
aphere
cylinder
What oolor the metal sphere?
There is a large green shiny thing: what shape is it?
color of the big ball?
There large brown thing: what shape is it?
Input
cube
sphere
cube
Input
cylinder
sphere
cylinder
Figure 2: (1) Imput images from the CLEVER-XAI dataset with the LRP a1 saliency maps computed for the (2) correct class, (3) an incorrect class, and (4) different question. The original question is written above. The saliency maps do not change visually when different output class is explained. However, changing the question highlights other regions.
A recent work (Arras et al., 2022) evaluated different saliency methods on the CLEVR VQA dataset using ground truth seg- mentation masks. Interestingly, they found LRP a. (equivalent the DTD 2* -rule) to highlight the object of interest particu- larly well: "(...] high conneclion between the relevance healmaps and the taryet objects of each question". This finding seemss to contradict Sixt et al. (2020), which found that LRP a190 becomes independent of the network's deeper layer. In Arras et al. (2022), it was therefore concluded: "Maybe the phenomenon descrübed in (Sist et al., 2020) becomes predominant in the asymplotic case of neural network with very high number of layers, [...]".
10a
Correct Logit Rand Quest.
Correct Logit vs. Rand. Logit
Density
10
10-2
0.0
0.2
0.4
0.6
0.8
1.0
Absolute Difference
A simple empirical test would have been to check if LRP 1, saliency maps change when the network's last layer is changed. To perform this test, we replicated their setup and trained relation network (Santoro et al., 2017) on the CLEVR V1.0 dataset Johnson et al. (2017). The network reached an accu- racy of 93.47%, comparable to 93.3% (Arras el al., 2022) and 95.5% (Santoro al., 2017). We included more details about the model in Appendix B.
Figure 3: The histogram of absolute dif- ferences between the saliency maps for Correct Logit us. Random Question and Correct Logit us. Random Logit.
We then compared 1000 LRP a1so's saliency maps for the correct answer, incorrect answer (but from the same category), and the correct answer but different question. Tt. is valid to ask for explanation of different class, for example, to understand which evidence is present for sphere instead of cube. The saliency maps were scaled to cover the range [0,1], and the differences were measured using the mean absolute difference. Figure 3, a histogram of the differences is shown. While the saliency maps are very similar in both cases, there seems to be more variability in the question: for "correct logit vs. random question" there is an order of magnitude more pixels with a difference of 1. When looking at the resulting saliency maps in Figure In contrast, the saliency map of the wrong answer does not change.
see that the saliency maps differ quite significantly when changing the question.
First, these results validate the claim in (Sixt et al., 2020) that LRP is independent of the network's deeper layer. Second, they indicate that an information leakage between the question and LRP ARPa1.00 's saliency maps is present.
The reason for this information leakage can be found the specific architecture of the relation networks: pairs formed between all feature-map locations of the convolutional output. As the feature map has shape
11

---

Published in Transactions on Machine Learning Research (11/2022)
(8, 8, 24), 64-64 pairs are formed (i.e., wiith"". Additionally, the question embedding produced by LSTM layer is concatenated to each pair. This yields triples of (Vij, Vkt, @LSTM), where i, ki I (1)...,8). and v E 18x8x24 is the convolutional stack's output. As the convolutional layers and the LSTM layer are trained together, their representations are aligned. Thus, changing the LSTM embedding will change the internal representation in the subsequent layers. For relevance locations, the question embedding will match better with the convolutional activation and, therefore will lead to a higher saliency map at relevant locations. However, the saliency maps will still become independent of the network's deeper layer.
The implications are substantial: for example, if the model's final layers' were fine-tuned on new task, the LRP a1 explanation would not change and could not be used to explain this model. Even worse, if your model was altered to predict spheres instead of cubes, the LRP explanation would not reflect this.
It is quite fascinating that the a1P explanations highlight the right object according to the ground truth, but fail to highlight evidence for the wrong object. This result also shows how difficult it to evaluate explanation methods empirically.
6 Conclusion
We have shown that DTD, which has been cited as the theoretical foundation of numerous follow-up post-hoc explanation techniques (Ali et al., 2022; Binder et al., 2016; Kindermans et al., 2018; Arras et al., 2017; Hui & Binder, 2019; Huber et al., 2019; Eberle et al., 2020), exhibits serious flaws that explain why saliency maps created with these methods are independent of the output. From Sixt et al. (2020), we know that that the positive matrices produced by the zt-rule will converge to a rank-1 matrix. These positive matrices stem from a specific selection of the root-point, and as the selection of the root-points is not restricted, the 2 tr-le can be justified by the DTD theory, as every other explanation could be - by picking an appropriate root.
DTD as theoretical framework for explanations is under-constraint, and can be considered insufficient. Caution must be used when using explanations derived from this theory. At the core of the problem, there is no restriction and little guidance on choosing the root points. Under certain conditions (constant root points), DTD reduces to backpropagating the gradient, albeit hidden behind a complex mathematical structure. In the other case (input-dependenl root points), DTD leaves open a backdoor through which virtually any explanation can be created by crafting the root point's Jacobian accordingly. However, this again obfuscated by the thcory rather than made transparent.
Since its first ArXiv submission (Montavon et al., 2015), the DTD publication has been eited numerous times. Although even the authors have reported class-insensitive behavior (Kohlbrenner et al., 2020; Montavon et al., 2018)3, follow-up works have readily used DTD's 's key concepts, motivated by the secmingly robust mathematical foundation, instead of searching for the underlying reasons. Furthermore, explanations based on DTD were used in various applications, for example, for validating their model (Andresen et al., 2020), gain insights into geoscientific questions (Toms et al., 2020), or conduct user studies (Alqaraawi et al., 2020).
While we were able to discover serious issues of DTD, we do not see a solution how to solve them. We therefore want to point out that other theoretically well-justified methods exist: (Deletion of Information) which information can be deleted without changing the network output? One approach uses noise (Schulz et al., 2020), other discrete deletion of the input (Macdonald et al., 2019). (Testing prediction capabilities): We can test whether certain concepts are present in the network by trying to predict them from intermediate features (Kim et al., 2018). (Model inversion): How would the input need to change to predict another class? This question can be answered using invertible models (Hvilshoj el al., 2021; Dombrowski el al., 2021) or conditional generative models (Singla al., 2020). (Simple Models): If a similar performance is achieved by simpler, more interpretable model, why not simply use that? For example, (Zhang et al., 2021) replaced part of the network with linear model. All these approaches do not come with complicated mathematical superstructure, rather they have a simple and intuitive motivation.
3 3 reduced number of colle-connected layers avoids that the relewance, when redistributed backwards, looses its connection to the concept being predkcted."" 7 Montavon al. (2018, Sec 5.1.)
12

---

Published in Transactions on Machine Learning Research (11/2022)
Broader Impact Statement
Although our work focuses on the theoretical foundations of particular explanation method, we see broader implications of this work. Our work demonstrates that the theoretical loundation of explanation methods need rigorous analysis before they can support the trust that developers, users, and even regulatory bodies may put in it. This is espocially important in the field of explainable Al since empirically evaluating explanations is difficult.
The field offers variety of explanation methods, and ways to test the quality of explanations. We rec- ommend using more than just one method and employing a range of metrics and user tcsts to make sure explanations are helpful potentially critical use-cases such as medical decision making or the screening of job applications.
Acknowledgement
We want to thank the reviewers for their helpful feedback that improved this manuscripl further. The computation were done on the Curta cluster provided by the Zedat, Freie Universtität Berlin (Bennett et al., 2020). Finally, we thank Jonas Köhler for discussions about the manuscript idea.
13

---

Published in Transactions on Machine Learning Research (11/2022)
References
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klbes-Robert Müller, and Lior Wolf. XAI for transformers: Better explanations through conservative propagation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Nuu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 435 451. PMLR, 17 23 Jul 2022. CRL https:/ htttps://proeeedinge mlr wwwpppresssaae aetees/v162/all/ wat.2a//aaaaaa.../aa aamll..hmml.
Ahmed Alqaraawi, Martin Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Berthouze. Evaluating saliency map explanations for convolutional neural networks: A user study, In Proceedings of the 25th International Conference on Intelligent User Interfaces, IUI '20, pp. 275-285, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450371116. doi: 10.1 10.1145//3773325.3333
URL https:
//doi. org/oo///// 10. 1145/3377325 3315/3337732..375559
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- based attribution methods for Deep Neural Networks. In International Conference on Learning Represen- tations, February 2018. URL https://openreviie http://wwwww.eeettoorii.i...aae. het//forum2iddssyyLLRAAAM
Nick Andresen, Manuel Wöllhaf, Katharina Hohlbaum, Lars Lewejohann, Olaf Hellwich, Christa Thöne- Reineke, and Vitaly Belik. Towards a fully automated surveillance well-being status laboratory mice using deep learning: Starting with facial expression analysis. PLoS One, (5())cc0228059, 2020.
Raman Arora, Amitabh Basu, Poorya Mianjy, and Anirbit Mukherjee. Understanding Deep Neural Networks with Rectified Linear Units. In International Conference on Learning Representations (ICLR), February 2018. URL https://openrrevier www./neeottmmddl...... J_rgWRW.
Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Explaining recurrent neural network predictions in sentiment analysis. EMNLP 2017, pp. 159, 2017.
Leila Arras, Ahmed Osman, and Wojeiech Samek. Clevr-xai: A benchmark dataset for the ground truth eval- uation of neural network explanations. Information Fusion, 81:14-40, 2022. ISSN 1566-2535, doi: https: //dbi..rg/110.111/....mm htt.://......fff22222.0.000 2021.11.008. URL. $15662253521002335.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise rele- vance propagation. PLoS ONE, 10(7), 2015. ISSN 19326203. doi: 10.1371/journal.oon 0130140.. URL
ploss wwtpppo..........eeppppppppppppp httepp/lssmma/rrtccll//eeeeeeeea 1371/journal pone 0130140ktype=
printable. 00067.
David Balduzzi, Marcus Frean, Lennox Leary, P. Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The Shattered Gradients Problem: If resnets are the answer, then what is the question? In Proceedings of the 34th International Conference Machine Learning, pp. 342-350. PMLR, July 2017. URL https: / //proceedings. mlr. vr..prrsss//O//baluuuzii htt. ISSN: 2640-3498.
Loris Bennett, Bernd Melchers, and Boris Proppe. Curta: A general-purpose high-performance computer zedat, freie universität berlin. http:// http::/dx..doi.org// 100. w1t..117169///ffbimmmeoo
2020.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek.
Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers. In Artificial Newral Networks and Machine Learning ICANN 2016, Lecture Notes in Computer Science, pp. 63-71. Springer International Publishing, 2016. ISBN 978-3-319-44781-00 doi: 10.1007/998-333104447811
Jianbo Chen, Le Song, Martin J. Wainwright, and Michael I. Jordan. T-shapley and c-shapley: Efficient model interpretation for structured data. In International Conference on Learning Representations, 2019. URL https://openrevvier net//forumiidss1ISKoo0
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Rober Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in Neural Information Processing Systems, 32, 2019.
14

---

http://mwweeere.eeeeoooroooaaaaa Published in Transactions on Machine Learning Research (11/2022)
Ann-Kathrin Dombrowski, Jan E Gerken, and Pan Kessel. Diffeomorphic explanations with normalizing flows. In ICML, Workshop on Invertible Neural Networks, Normalizing Flows, and Erplicil. Likelihood Models, 2021. URL
Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus-Robert Müller, Matteo Valleriani, and Grégoire Mon- tavon. Building and interpreting deep similarity models. IEEE Transactions on Pattern Analysis and Machine Intellggence, 2020.
G. B. Folland. Advanced calculus. Prentice Hall, Upper Saddle River, NJ, 2002. ISBN 978-0-13-065265-2.
Andreas Holzinger, Anna Saranti, Christoph Molnar, Przemyslaw Biecek, and Wojeiech Samek. Esplainable Al Methods A Brief Overview, pp. 13 38. Springer International Publishing, Cham, 2022. ISBN 978-3- 031-04083-2. doi: 10.1007 10.1007/978-20 1077-8-031--40000 2.
URL https: http//i...////////////oo org/o0gg///o -0007/978-3-031-40008-----
Tobias Huber, Dominik Schiller, and Elisabeth André. Enhancing explainability of deep reinforcement learning through selective layer-wise relevance propagation. In Joint German/Austriae Conference on Artificial Intelligence (Kümstliche Intelligenz, pp. 188 202. Springer, 2019.
Lucas Y. W. Hui and Alexander Binder. BatchNorm Decomposition for Deep Neural Network Interpre- tation. In Advances in Computational Intelligence, volume 11507, pp. 280-291. Springer International Publishing, 2019. doi: 978-3-030-20518-8 24. Series Title: Lecture Notes in Computer Science.
24. URL http://link... www.iink.spinge....... com/10.100 com/100.007/
Frederik Hvilshøj, Alexandros losifidis, and Ira Assent. Ecinn: Efficient counterfactuals from invertible neural networks. ArXiv, 0tt//2111.117001 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901 2910, 2017.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668 2677. PMLR, 2018
Pieter-Jan Kindermans, Kristof Schütt, Klaus-Robert Müller, and Sven Dähne. Investigating the influence of noise and distractors the interpretation of neural networks. Technical Report arXiv:1611.0ff 161.00770,, arXiv, November 2016. URL http:://rrket http:://axxivorrg/abss//// www./osg////aeeeee//o/ /161 hatps//111.....00..1
arXiv:1611.07277 [cs, stat] lype: article.
Pieter-Jan Kindermans, Kristof T. Schüt,, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne. Learning how to explain neural networks: PatternNet and PattennAttribution. February 2018. URL
Maximilian Kohlbrenner, Alexander Bauer, Shinichi Nakajima, Alexander Binder, Wojeiech Samek, and Sebastian Lapuschkin. Towards best practice in explaining neural network decisions with Irp. In 2020 International Joint Conference on Neural Networks (IJCNN) pp. 1-7. IEEE, 2020.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Mclnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Riccardson. Captum: A unified and generie model interpretability library for pytorch, 2020.
I. Elizabeth Kumar, Suresh Venkatassubramanian. Carlos Scheidegger, and Sorelle Friedler. Problems with shapley-value based explanations as feature importance measures. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5491 5500. htmm..
PMLR, 2020. URL https: http::///occeeeeeee mlr.
Scott M Lundberg and Su-In Lec. unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
15

---

Published in Transactions on Machine Learning Research (11/2022)
Daniel D Lundstrom, Tianjian Huang, and Meisam Razaviyayn. A rigorous study of integrated gradients method and extensions to internal neuron attributions. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 14485-14508. PMLR, 17-23 Jul 2022. URL attpps///pree https:///rooceedngg.... mlr prress/v62//undastit
html.
Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rate-distortion framework for explaining neural network decisions, 2019.
Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojeiech Samek, and Klaus-Robert Müller. Explain- ing nonlinear classification decisions with deep taylor decomposition. arXie preprinl arXiv: 79, 2015.
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Tlaus-Robert Miller. Layer-wisc relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning, pp. 193 209, 2019.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Klaus-Robert Müller, Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognition, 65: 211 222, 2017. ISSN 0031-3203, doi: https://doi.org/100.1000 pttp:////...00/////ao.ooooo. 2016.11.08.. URL https://www. sciencedirect.
Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding doep neural networks. Digital Signal Processing, 73:1-15, February 2018. ISSN 10512004. doi: 10.1016/j.
1sp.2017.10.011. URL httpp:///innkinghuu.b. www.:ii.eeeeeeeeee....oooooeeee ottpr//.......i/p//////////n/nar http://wwwaaaaaaaeeeoo.orocerttt
Guido F Montufar, Razvan Pascamu, Kyunghyun Cho, and Yoshua Bengio. On the Number of Linear Regions of Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 27.
Curran Associates, Inc., 2014. URL bttps://arrse
Weili Nie, Yang Zhang, and Ankit Patel. A Theoretical Explanation for Perplexing Behaviors of Backpropagation based Visualizations. In Proceedings of the 35th International Conference on Machine Learning, pp. 3809 3818. PMTR, July 2018. URL https: http:////ccccccoo..ccoo mlr wttp://m.r..ppeee/000/ wwwpppeeee///eeep vaaess///0/nie8a... html. ISSN: 2640-3498.
Wojciech Samek, Leila Arras, Ahmed Osman, Grégoire Montavon, and Klaus-Robert Müller. Explaining the decisions of convolutional and recurrent neural networks. 2021a.
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J. Anders, and Klaus-Robert Müller. Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications. Pro- ceedings of the IEEE, 10443))247-278 March 2021b. ISSN 1558-2256. doi: 10.1009/PRRO..2221..30000 Proceedings of the IEEE.
Conference Name:
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. In 1. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (ods.), Advances Neural Informalion Processing Systems, volume 30. Curran Associates, Inc., 2017. URL sttps://proceedingg .df.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottle- necks for attribution. In International Conference on Learning Representations, 2020. URL https:
//openreview.net poneerreview.net/foormmmm mett/forrmmidd=$11Whhii.
Harshay Shah, Prateek Jain, and Praneeth Netrapalli. Do input gradients highlight discriminative features? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 2046-2059. Curran Associates, Inc., 2021. URL https: .df..
/ http://tttttteeeeeeoooooeoottttt
16

---

Published in Transactions on Machine Learning Research (11/2022)
Lloyd S. Shapley. Notes on the N-Person Game II: The Value of an N-Person Game. RAND Corporation, Santa Monica, CA, 1951. doi: 0/244/R00660.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences, 2016. URL https://arxiv.. org/ abs/
1605.01713.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by progressive exaggeration. In International Conference on Learning Representations, 2020. URL https:///pperreviee net/foruumiid::H1PFWgrr-P
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: butions fail. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9046-9057. PMLR, html.
Why many modified BP attri-
13 18 Jul 2020. URL https://proceedi
mlr wwtppp/ssseeaeieeeeeeeeeeeseeeee
Jost Tobias Springenberg, Alexey Dosovitakiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint orXiv: 1412.:1411.600 2014.
Erik Strumbelj and Igor Kononenko. A general method for visualizing and explaining black-box regression models. In International Conference on Adaptive and Natural Computing Algorithms, pp. 21-30. Springer, 2011.
Erik Strumbelj and Tgor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems, 41(3):647 665, 2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319 3328. PMLR, 2017.
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1812 6199, 2013.
Benjamin A Toms, Elizabeth A Barnes, and Imme Ebert-Uphoff. Physically interpretable neural networks for the geosciences: Applications to earth system variability. Journal of Advances in Modeling Earth Systems, 12199=22019 2020.
Tom Viering, Ziqi Wang, Marco Loog, and Elmar Fisemann. How to manipulate cmns to make them lie: the gradcam case. Xr preprint arXiv:1907. 10901, 2019.
Stephan Waeldchen, Jan Macdonald, Sascha Hauch, and Gitta Kutyniok. The computational complexity of understanding binary classifier decisions. J. Artif. Int. Res., 70:351-387, may 2021. ISSN 1076-9757. doi: 10.1613/jair...12333 12339.. URL http:::/doi..rg//// 16133/jarr.. 1. 12359.
Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh. Gradient-based analysis of NLP models is manip- ulable, In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. November 2020. Association for Computational Linguistics. doi: 10. 10.18653/vv 1o020..iniingscmmlllp URL
Online,
wwwpllggg.rrg///// wrg/2020.0000.. Findingssemmnnl 24.
H. Xiong, L. Huang, M. Yu, L. Liu, F. Zhu, and L. Shao. On the Number of Linear Regions of Convolutional Neural Networks. In ICML, 2020.
Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin IP Rubinstein. Invertible concept- based explanations for enn models with nen-tegative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11682 11690, 2021.
17

---

Published in Transactions on Machine Learning Research (11/2022)
A Proofs
A.1 Proof of proposition 3 and? 2
We will proof Propositions 2, 3, and 4 together.
We start with the partial derivative the relevance function at layer
(ai) das
d11
[ (Ji(ai))
Σ
a a (a)
(19)
dai
dain
j-1
aa (ai)
d1+1
aLt) 4mm
(ai)
2Rt (fi(ai))
Σ
(20)
sa
ča:
j=1
a--a (1) (ai) (ar)
+ a
aRR (Fi(ai))
aai - a))(aa))
(21)
da
čai
a (as)
(8)
for ReLU networks
In this first step, we applied the product rule. For ReLU networks, the higher-order terms are zero. For other networks (Transformer, LSTMs), the higher-order terms will not be zero. The terms which are zero for ReLU networks are exactly the terms from Proposition 4.
In the next step, we will apply the chain rule and rewrite dai as the k-standard basis (a one-hot vector where the k-th dimension is 1):
(ay) Ea: = Σ
dei1
2a [q]
(ai) ea:
)
(f((aa))
a (Ji(ai)) da: 0f((a))
Ck
(22)
aa=a(()(ai)
The next observation is that the gradients inside the [...] a must be the same for aj and the root point a as both are in the same local region of fi o R/t Thercfore, we can safely drop the evaluation of the gradient at the root point (I--Ja---a"
(())) and write:
aRd (ai)
di+1
@f((a:)
oa (ai)
@f((a:) aaz
an (fi(ai)
Σ
(23)
cas
(ain
eaz
Ōfi(ar)
Substituting this result into the definition of (a--1(at1) from equation 8 yields the result of Proposition 3.
To proof proposition 2, (a(()
(ai)/ōa; 0 and get:
(m))
2 (a:)
d++
af((ai)
RR (fi(ar)
(24)
das
(.((a))
af((ar)
rRt (fi(ai) Ih((a))
(25)
Janu
@f((ai)
01411(4111)
aR (f44((441)
(26)
-dau
saitipl
0f411(441)
1-1
@ff(ai)
af1+1(a111)
dia aRlt² ([+1((144))
Σ
(27)
sait1
df+1
18

---

Published in Transactions on Machine Learning Research (11/2022)
ef((a)
0ft+1(a111)
ôf (an-1)
duil 2
(28)
Jayu
8ait1
ddn-1
afilai)
ifi-1(a141)
ò (a--1(an-1)
@fnlan)
(29)
Jaiu
8ait1
dan-1
dan
(aie(a
(30)
Substituting this into the relevance function of the input R(x) Rt'(a)) and using (as & must be in the same linear region), yields:
@R(()
a
x==(()
dw
2R(x) dx
R(x) ((()))))
(x 2) R(ā(z)) Vfa((() ② (x x(x)),
(31)
mm-((m)
which finished the proof of Proposition 2.
A.2 Admissible Region for the root points of the Relevance Function
We now proof Proposition which is restated here:
Proposition (C1: Recursive Taylor cannot increase the size of admissible regions) Given ReLU network
f IRd dn++ recursive relevance functions (ar) with E (I,...,n) according to definition 3, and let index the explained logit. Then it holds for the admissible region NR (ai) for the root points at) of the relevance function R that (a.) -Nfsg....f((a))) (a:).
Let ai...., an fix the root points. Proof by induction over the number of layers. We start with the induction base case at the final layer. There, we have (²,, fiei(,,) which follows from the recursion base case. Clearly, NR (ā.) Ning (ān). Induction step: We assume N W8141(ā11)) S N/ (ā1+1). For the layer 1, the root points must be valid for the function (J(())) As we know that NR (ā1+1) S No (ā141), it must also be the case that NR (ā:) = (ā.).
B Details About the Relation Network for the CLEVR dataset
Our code-base builds upon public available implementation of relation networks and utilized Captum for computing LRP explanations Kokhlikyan et al. (2020). We also setup the CLEVR XAI dataset released on 3ithubs
19

---

Published in Transactions on Machine Learning Research (11/2022)
C Pseudo-Code
C.1 Full-backward DTD
Algorithm Pscudocode for the recursive application of the Taylor Theorem. The global state contains
the following variables: fi:...: Ju the layer functions of the network, d dut1 the dimension of the input to each layer, and the index of the output neuron.
function GRT RELEVANCE(1: layer index, az: the layer input)
if / - n then return (1))
end if
à; FIND ROOT POINT(f,L,a) ai) R[+11 - GET RELEVANCE(I + (i(r))
for 1E....d. do
a. gradrd - 0 R backward()
[1]
r3 à grad @ (a: - āz)
end for return >)=1 rj
end function
20

---

Published in Transactions on Machine Learning Research (11/2022)
C.2
Train-free DTD
Algorithm 2 DTD Train-Free
function FIND ROOT POINT(|: layer index, ay: the layer input, T the relevance)
w = W6s
if +t-rule then
aj C 2w,=0
else if w²-rule then
w clse if 7-rule then
v v= a= + +lw,30)
end if
R (WTTv)
return a; tv end function
Ensures that wij (a: āi) w//
Ru1
R
w
function GRT RELEVANCE(:: layer index, az: the layer input)
if = n- then
return 0lg
end if
R[+1 GET RELEVANCE(1 + 1, f((ai)
relevance of input instead of root point a
for
jE 11...d.44 di+1 do
a - FIND ROOT_ B_L__ROT_OOOOA_ (oiT__OOIN(f,,,FF 11
a. gragrad < 0
[Win) ai (f) + b
o. backward
rj- - ...dddd O (at āz)
end for
return 1 rj
end function
21


Reviews:
[{"review_id": "h2sXzrblJh", "reviewer": "Reviewer_u2ZG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. **A solid technical argumentation based on simple math**: The main strength of this work is that it provides theoretically solid arguments to settle a previously observed empirical phenomenon. Although very technical and specific to the DTD technique, the arguments are easy to follow and do not require heavy mathematical machinery. The fact that the root points influence the explanations through the Jacobian of the layer's output might look obvious in retrospect, but it is nice to see some work that provides analytical expressions for this dependence.\n\n2. **Concrete message**: I appreciate that this work focuses on a very specific problem with a clear objective: \"Analytically explaining how the DTD method is underspecified and why can be made to fail in practice\"\n\n# Weaknesses\n\n1. **Introduction is too technical**: Although I appreciate that this work is focused on a technical topic and goes straight to its point, I find the starting section to be too technical and hard to follow on a first read due to the use of very specific DTD jargon which has not been introduced before right from the start (e.g., $z^+$- rule or LRP).\n\n2. **Abuse of heavy notation**: I find that some sections of the paper could be better off with some polishing of the notation to reduce the overuse of incoherent subscripts and upperscripts. For example, in Section 2 the relevances of each layer are denoted as $R^l$ while later on  Section 3 they are denoted as $R^f$, $R^{x}$ and $R^{h}$.  The distinct use of \"Tn\" or \"An\" to denote the different arguments is also a bit confusing.\n\n3. **Hard-to-follow storyline**: The structure of the paper is sometimes a bit hard to follow due to the constant use of technical detours to introduce new concepts. In this regard, I believe that having a less technical introduction where the main issues are broadly discussed followed by a better structured discussion of the paper organization could help with this.\n\n4. **(Minor) Unnecesarily naive experiments**: Although I do not think it is important for the paper's argumentation, I believe there is no reason for the experiments of this work to be performed on such a toy setting. The authors could very easily provide at almost no computational cost or engineering effort the same experiment using some standard classification dataset such as CIFAR-10 and a popular architecture like ResNet18.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides a very technical theoretical study on the shortcomings of the Deep Taylor Decomposition (DTD) as a post-hoc explanation method for deep learning. Specifically, this work builds on prior studies that showed that the DTD method could produce arbitrarily different explanations depending on the root point specified by the user. The main contribution of this work is therefore a series of theoretical arguments that scrutinize this behavior and give a technical explanation for why it happens. In short, the authors show analytically that the DTD method can be arbitrarily manipulated to produce any explanation through the effect of the intermediatee layer Jacobians with respect to the root points, and empirically, that the common heuristics used in practice to identify valid root points for the DTD method do not actually satisfy its main assumptions. This series of arguments leads the authors to conclude that the DTD method is not a theoretically sound post-hoc explanation method as it is (i) underspecified in its construction, and (ii) its main assumptions are violated by the heuristics used in practice.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Overall, I think this paper meets the requirements to be published at TMLR: (i) it is technically sound, and (ii) it may appeal to some people in the community (although in this particular case, it might be a narrow set of the community interested in the technical details of the DTD method).\n\nIn any case, I still bellieve the paper could be easily improved, mostly by improving its introduction, structure and notation, as I listed under \"weaknesses\" before. Running the experiments on a less toy setting might also make the paper appeal to a broader audience (although I personally believe this is not strictly necessary).\n\nAnother way the authors could strengthen the appeal of their technical derivations is if they used their derived expressions explicitly to break a real trained network's DTD explanation by identifying root points that make the computed relevance independent of the output. This would be similar to the methods that compute adversarial examples on a model's explanations but in this case the authors would make use of their analytically derived expressions, thus providing further insights about the structure of these failure modes. ", "broader_impact_concerns": "I see no clear broader impact concerns derived from this very theoretical and technical work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ylgmRFgKD6", "reviewer": "Reviewer_rZ5T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strneghts:\n- The paper is clearly written, including an overview of the overall framework and ideas\n- The paper is well motivated, and has an important purpose: if explanations methods are to provide further understanding about the predictions of NNs, it is fundamental that this is done in a rigorous and careful way, and not resulting to loosely motivated 'hacks'.\n- The contribution is simple and important.\n\nWeaknesses:\n- Very small reorganization of content, and rephrasing of some parts might be good (see below).\n- Some related references missing (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper undertakes the study of the method called Deep Taylor Expansion DTE), which is a strategy to provide some kind of post hoc explanations for the output of neural networks. In particular, the authors present the main ideas (expanding the output in terms of a Taylor expansion), explain how this is simplified when the root points are in the same linear regions, and discuss different strategies on how to do this. Contrary to what one would expect, these different choices are shown to be problematic (e.g. resulting in relevance vectors that can be independent of input, or that can be obtained to justified any output). The empirical demonstration furthermore shows that in practice some of the underlying assumptions -such as the root corresponding to the same linear region- don't really hold either.\n\nIn a nutshell, this work attempts to shed some light onto the DTE, clarifying some concepts while disproving some common \"myths\" about this approach.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I very much appreciate this contribution, and the authors' intention to clarify what certain popular approaches do, and what they do not do. I only have a few minor comments:\n\n- As I read through the paper, I thought that some comments on other theoretically-motivated methods for explainability would be contribute to the broader picture of the field. Later, almost at the end, I discovered that the authors do in fact have a section on Related Works which addresses these points. I believe it might be better to relocate that section to near the beginning, perhaps after the introduction. In that way, the reader will understand that several other principled methods exist, and that this paper will only study one family of them (DTE). Explaining this explicitly might provide a nice context for the reader to better understand the contribution of this paper.\n\n- I think there are some works that the authors might find relevant for this paper. E.g.: [Do Input Gradients Highlight Discriminative Features?, Shah et al, 2021] provides very interesting insights showing that methods based on gradients can be provably incorrect - which supports the finding of this paper. Furthermore, the work [From Shapley back to Pearson: Hypothesis Testing via the Shapley Value, Teneggi et al, 2022] shows that Shapley values, contrary to previous interpretations and in a similar spirit to this paper, provides a formal notion of statistical importance by mathematically analyzing what those coefficients estimate.\n\n- In Section 2.2, the authors refer to the exponential number of linear regions of deep nets. While this is true in the worst case, note that more recent analysis has shown that these are significantly lesser, and can even become linear [Complexity of Linear Regions in Deep Networks, Hanin & Rolnick, 2019].\n\n- After Eq (11), in the expression $R^h$, $h$ should be bold.\n- First line in section 3.2.1, \"principle\" -> \"principled\".\n- Last line in first paragraph in Sec 3.2.1,  \"choice\" -> \"choose\"\n- Second paragraph in Sec 3.2.1, could the authors clarify what they mean exactly by \"class insensitivity\"?\n- Proposition 2: The first sentence is missing a subject. Maybe the authors meant \"Let f be a Relu network...\", or else the sentence should not break before continuing with the assumptions on the root points.\n- First line after Prop 2, the zero in $R(0)$ should be bold.\n- Last paragraph on page 8, the authors write that \"It is merely recommended to choose it within the layer’s input domain\". What do the authors mean exactly? Anything on $\\mathbb{R}^{d_l}$? Or perhaps the authors meant \"within the layer's input linear region\"?\n- Sec 5.1, the authors note that if the difference $||\\nabla f(x) - \\nabla f(\\tilde{x}) ||=0$, the root points are assumed to be in the same linear region. Rather than \"assuming\" this, the authors might want to explain that this condition is necessary but not sufficient. Thus, showing that this is not the case, suffices for the argument they are trying to make.\n- Close to the end of page 9: \"Less than 100% of all root points have gradient differences are zero,..\" -> \"Less than 100% of all root points have gradient differences *that* are zero,..\".\n", "broader_impact_concerns": "While not strictly necessary, I would recommend the authors include a Broader Impact Concerns section. This work does not represent any concerns per se, but I think this would be a good opportunity for the authors to raise concerns on using explanation methods that lack the necessary careful and precise theoretical guarantees.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pivUjr3LXA", "reviewer": "Reviewer_nb1x", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A strength of this paper is that it attempts to provide a more thorough theoretical understanding of explanation methods, which are an important problem in ML.\n\nA weakness of this paper is that the writing was not clear to me as a non expert in this subfield.  It may be the case that the writing is appropriate for readers who are experienced in this subfield, but I am not such a reader.  Here are some aspects that were unclear to me:\n- What is the nature of explanations provided by the studied methods?\n- What is the z+ rule and what does it mean to say it will \"remain the same even if the last layer is randomized\"?\n- What is the logic use in relating \"flaws\" in the DTD theory to theoretical explanations?  Is the contribution of this paper that certain scenarios do not satisfy assumptions necessary in existing theoretical work for DTD-theory to apply?\n- In (T1) the authors say they show that \"the root points must be contained in the same linear region as the input\".   What do they mean by saying \"must be\"?\n- The paper mentions \"root points\", but it is not clear to me what quantity has roots that are being studied.\n\nThese weaknesses make it difficult for me to assess whether the conclusions support the claims.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In the paper \"A Rigorous Study of the Deep Taylor Decomposition\", the authors provide some technical work related to the generation of post-hoc explanations from trained machine learning models.  The authors point out that some existing methods create explanations remain constant even if the last layer is randomized.  The authors note that the Deep Taylor Decomposition is used to provide theoretical underpinning of methods like this and establish technical claims relating root points associated with the Deep Taylor Decomposition to the explanations produced.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I recommend a substantial rewrite of the introduction (and possibly abstract) so that a reader who is not thoroughly familiar with this subfield can understand the context and the conclusions drawn.\n\nI also recommend rewriting throughout the whole paper.  It is hard to distinguish what is background material versus what is new.  It is also hard to figure out what the authors are intending to do in each section.  It would help if the authors at each section led with a paragraph explaining what was going to happen in that section.\n\nWithout substantial rewriting I do not feel I will understand the work well enough to be able to recommend for acceptance.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper undertakes the study of the method called Deep Taylor Expansion DTE), which is a strategy to provide some kind of post hoc explanations for the output of neural networks.   During the initial review period, reviewers raised some concerns about this paper which later have been addressed by the author's rebuttal.   All reviewers believe this paper meets the requirements to be published at TMLR.  Reviewer rZ5T believes that the related works section could be broader, and the authors could have done a better job at situating their contribution in light of other, theoretically-sound, explanation methods.  I hope authors could make improvements in their camera-ready version.  



Decision: Accept (poster)