Fast Axiomatic Attribution for Neural Networks
Authors: Robin Hesse
Simone Schaub-Meyer
Stefan Roth
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
interpretability
attributions
explanations
attribution priors

Abstract: Mitigating the dependence on spurious correlations present in the training dataset is a quickly emerging and important topic of deep learning. Recent approaches include priors on the feature attribution of a deep neural network (DNN) into the training process to reduce the dependence on unwanted features. However, until now one needed to trade off high-quality attributions, satisfying desirable axioms, against the time required to compute them. This in turn either led to long training times or ineffective attribution priors. In this work, we break this trade-off by considering a special class of efficiently axiomatically attributable DNNs for which an axiomatic feature attribution can be computed with only a single forward/backward pass. We formally prove that nonnegatively homogeneous DNNs, here termed $\mathcal{X}$-DNNs, are efficiently axiomatically attributable and show that they can be effortlessly constructed from a wide range of regular DNNs by simply removing the bias term of each layer. Various experiments demonstrate the advantages of $\mathcal{X}$-DNNs, beating state-of-the-art generic attribution methods on regular DNNs for training with attribution priors.

Fast Axiomatic Attribution for Neural Networks
Robin Hesse
Simone Schaub-Meyer'
Stefan Roth 1.2
'Department of Computer Science, TU Darmstadt hhessian {robin. hesse, simone.
schaub, simone.schaub, rtefa tu tu-dammstadt.de
Abstract
Mitigating the dependence spurious correlations present in the training dataset is a quickly emerging and important topic of deep learning. Recent approaches include priors the feature attribution of deep neural network (DNN) into the training process to reduce the dependence on unwanted features. However, until now one needed to trade off high-quality attributions, satisfying desirable axioms, against the time required to compute them. This in turn either led to long training times ineffective attribution priors. In this work, we break this trade-off by considering a special class of efficiently axiomatically attributable DNNs for which axiomatic feature attribution be computed with only single forward/backward pass. We formally prove that nonnegatively homogeneous DNNs, here termed X'-DNNs, are efficiently axiomatically attributable and show that they can be effortlessly constructed from wide range of regular DNNs by simply removing the bias term of each layer. Various experiments demonstrate the advantages of -DDNNN, beating state-of- of-the-art art generic attribution methods on regular DNNs for training with attribution priors]
Introduction
Many traditional machine learning (ML) approaches, such as linear models decision trees, are inherently explainable [4]. Therefore, an ML practitioner can comprehend why a method yields a particular prediction and correct the method if the explanation for the result is flawed. The prevailing ML architectures in use today [12] 25), namely deep neural networks (DNNs), unfortunately, do not come with this inherent explainability. This cause models depend dataset biases and spurious correlations in the training data. For real-world applications, e.g., credit score or insurance risk assignment, this be highly problematic and potentially lead to models discriminating against certain demographic groups 3 20 To mitigate the dependence on spurious correlations in DNNs, attribution priors have been recently proposed attribution of a DNN at training time, they allow actively controlling its behavior. As turns out, attribution priors are a very flexible tool, allowing even complex model interventions such as making an object recognition model focus on shape [22] or less sensitive to high-frequency noise (7]. However, their use brings new challenges over regular training. First, computing the feature attribution of DNN is a nontrivial task. It is critical to use an attribution method that faithfully reflects the true behavior of the deep network and ideally satisfies the axioms proposed by Sundararajan et [34] Otherwise, spurious correlations may go undetected and the attribution prior would be ineffective. Second, since the feature attribution used in each training step, needs be efficiently computable. Existing work incurs trade-off between high-quality attributions for which formal axioms hold (34) and the time required compute them [7]. Prior work on attribution priors thus had to choose whether to rely on high-quality feature attributions or allow for efficient training. In this work, we obviate this trade-off.
22 23
By enforcing priors on the feature
Code and additional resources at
wtwp:iii..iituuu..oaaaassee wo/fasttttttooooeoooeooooe...e.t a aatriicttttiibiinn
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Specifically, we make the following contributions: (i) We propose to consider a special class of DNNs, termed efficiently axiomatically attributable DNNs, for which we can compute closed-form axiomatic feature attribution that satisfies the axioms of Sundararajan et al. [4, requiring only one gradient evaluation. As a result, we can compute axiomatic high-quality attributions with only one forward/backward pass, and hence, require only fraction of the computing power that would be needed for regular DNNs, which involve a costly numerical approximation of an integral 77 34]. This significant improvement in efficiency makes our considered class of DNNs particularly well suited for scenarios where the attribution is used in the training process, such as for training with attribution priors. (ii) We formally prove that nonnegatively homogeneous DNNs (termed X XDDNNs) are efficiently axiomatically attributable DNNs and establish a new theoretical connection between Inputx Gradient [27] and Integrated Gradients [4 for nonnegatively homogeneous DNNs of different degrees. (iii) We show how X'DNNN can be instantiated from a wide range of regular DNNs by simply removing the bias term of each layer. While this may seem like a significant restriction, show that the impact on the predictive accuracy in two different application domains is surprisingly minor. In variety of experiments, (iv) admit accurate axiomatic feature attributions at fraction of the computational cost and (v) beat state-of-the-ar generic attribution methods for training regular networks with an attribution prior.
demonstrate the advantages of X'-DNNs, showing that they
2 Related work
Attribution methods can roughly be divided into perturbation-baset 13
39
41
42
and
methods. The former repeatably perturb individual in- puts or neurons to measure their impact on the final prediction. Since each perturbation requires a separate forward pass through the DNN, those methods can be computationally inefficient (28 and consequently inappropriate for inclusion into the training process. We thus consider backpropagation- based methods or, more precisely, gradient-
backpropagation-ban based
I
28
and rule-based attribution methods. They propagate an importance signal from the DNN output to its input using either the gradient or predefined rules, making them particularly efficient (28). and thus, well suited for inclusion into the training process. Gradient- based methods have the advantage of scaling to high-dimensional inputs, can be efficiently implemented using GPUs, and directly applied to any differentiable model without changing it [2].
The saliency method (30), defined as the absolute input gradient, is an early gradient-based attribution method for DNNs. Shrikumar al. (27) propose the InputxGradicns method, i.e., weighting the (signed) input gradient with the input features, to improve sharpness of the attributions for images. Bach et al. [S] introduce the rule-based Layerwise Relevance Propagation (LRP), with predefined backpropagation rules for each neural network component. As it turns out, LRP without modifications to deal with numerical instability can be reduced to apputxGradent for DNNs with ReLU 19 activation functions || 27), and hence, can be expressed in terms of gradients well. DeepLIFT (28) is another rule-based approach similar to LRP, relying on a neutral bascline input to assign contribution scores relative to the difference of the normal activation and reference activation of each neuron. Generally, sule-based approaches have the disadvantage that each DNN component requires custom modules that may have no GPU support and require a model ee-imllementttin..
Axiomatic attributions. As it is hard to empirically evaluate the quality of feature attributions. Sundararajan et al. (34 propose several axioms that high-quality attribution methods should satisfy:
Sensitivity (a) is satisfied if for every input and baseline that differ in one feature but have different predictions, the differing feature should be given a non-zero attribution.
Sensitivity (b) is satisfied if the function implemented by the deep network does not depend (mathe- matically) on some variable, then the attribution to that variable is always zero
Implementation invariance is satisfied if the attributions for two functionally equivalent networks
are always identical.
Completeness is satisfied if the attributions add up to the difference between the output of the network for the input and for the baseline.
Linearity is satisfied the attribution of linearly composed deep network aF1 + bF2 is equal to the weighted sum of the attributions for Fi and F2 with weights a and b, respectively.
Symmetry preservation satisfied if for all inputs and baselines that have identical values for symmetric variables, the symmetric variables receive identical attributions.

---

34 shows that none of the above methods satisfy all axioms, e.g., the saliency method and Inputx Gradient can suffer from the well-known problem of gradient saturation, which means that even important features can have zero attribution. To overcome this, (34] introduces Integrated Gradi- ents, a gradient-basse backpropagation method that provably satisfies these axioms; it is considered a high-quality attribution method to date 14. Its crucial disadvantage over previous methods is that an integral has to be solved, which generally requires an approximation based on ~ 20-300 gradient calculations, making it correspondingly computationally more expensive than, e.g. Input x Gradient.
Attribution priors. The above attribution methods cannot only be used for explaining model's behavior but also to actively control the behavior. To that end, the training objective can be formulated
as
arg min 0"
Σ [(Poix,)) - ()(A(Fo.))) (a.))XX
(1)
0
where a model Fe with parameters # is trained on the annotated dataset X. L denotes the regular task loss, and 92 is a scalar-valued loss of the feature attribution A, which is called the attribution prior (7]: X controls the relative weighting. For example, by forcing certain values of the attribution to be zero, we can mitigate the dependence on unwanted features [23]. But also more complex model interventions, such as making an object recognition model focus on shape [22] or less sensitive to high- fegueerrye noise 77. can be formulated using attribution priors.
An early instance of this idea is the Right for the Right Reasons (RRR) approach of Ross et al. [23], which uses the input gradient of the log prediction to mitigate the dependence on unwanted features. While this is more stable than simply using the input gradient, it still suffers from the problem of saturation. RRR may thus not reflect the true behavior of the model, and therefore, miss relevant features. Subsequent work addresses this issue using axiomatic feature attribution methods, specifically Integrated Gradients 6
which allows for more effective attribution priors 7 but incurs significant computational overhead, rendering them impractical for many scenarios. For example, Liu and Avei [1] report thirty-fold increase in training time compared to standard training. Rieger et al. [22) propose an alternative attribution prior based on rule-based contextual decomposition [18 31 (CD) as attribution method. This allows to consider clusters of features 77 instead of individual features and to define attribution priors working on feature groups. However, computing the attribution for individual features becomes computationally inefficient [7]. Additionally, since CD is rule-based attribution method, it requires custom modules and cannot be applied all types of DNNs Z The very recently proposed Expected Gradients 7 method reformulates Integrated Gradients as an expectation, allowing a sampling-base the attribution. Erion et al. 7 argue that similar to batch gradient descent, where the true gradient of the loss function is approximated over many training steps, the sampling- based approximation allows to approximate the attribution over many training steps. This results in better attributions while using fewer approximation steps. Even using as little as one reference sample, i.e., only one gradient computation, can yield advantages over the regular input gradient. However, we show that using only one reference sample still does not yield the same attribution quality as an axiomatic feature attribution method, reflected in less effective attribution priors. Schramowski et al. (26) propose a human-in -iteeiiii-ttt-oooo strategy to define appropriate attribution priors while training. Our attribution method is complementary and could be used within their framework.
14 34
approximation of
3 Efficiently axiomatically attributable DNNs
Formally, given function F: RR - R. representing single output node of a DNN, and input & E IR",, the feature attribution for the prediction at input & relative to a baseline input x' is a vector A(F, x, 2²) e R"," where each element a; is the contribution of feature I, to the prediction F(x) (34). In this work we seek an attribution method that is particularly well suited for scenarios where such an attribution is used at training time, e.g., training with attribution priors. As such, the attribution method should be of high quality, while being efficiently computable in single forward/backward pass. Since attributions obtained from Integrated Gradients (34] have strong theoretical justifications and are known to be of sugh-qualtty [14], they will serve as our starting point. In general, however, Integrated Gradients are expensive to compute for arbitrary DNNs. Therefore, in this work, we restrict our attention to special sub-class of DNNs, termed efficiently axiomatically allributable DNNs, that require only a single forward/hackwart pass to compute a closed-form solution of Integrated Gradients. We show that nonnegatively homogeneous DNNs belong to this class and use this insight
3

---

to guide the design of a concrete instantiation of efficiently axiomatically attributable DNNs, While there may be several such instantiations, we chose this particular one as can be easily constructed from wide range of regular DNNs by simply removing the bias term of each layer. This ensures comparability to prior work and allows for an easy adaptation of existing network architectures.
Definition 3.1. We call DNN F: R" + R efficiently axiomatically attributable w.r.f. a baseline r' E R", if there exists a closed-form solution of the axiomatic feature attribution method Integrated Gradients (F, x x') along the ht dimension of x E R", requiring only one forward/backward pass.
Note that all differentiable models efficiently axiomatically attributable w.E.t the trivial bascline x' = I. However, using such a baseline is not helpful. Instead, commonly chosen baselines are some kind of averaged input features baselines such that F(x') = 0, which allow an interpretation of the attribution that amounts to distributing the output to the individual input features (34).
Proposition 3.2. For a DNN F: Rn I. there exists a closed-form solution of (F, 0) w.r.t. the zero baseline O E _2,, requiring only one forward//ackkward pass, if F is strictly positive homogeneous of degree
E2,, i.c. (ax) = a (x) for Igso
Proof. Sundararajan et al. [34] define the axiomatic feature attribution method Integrated Gradients (IG) along the Th dimension for a given model F, input æ, baseline O, and straightline path ((a) = oz as
IG, 2. 0) [.' @x((a) da Assuming F is strictly positive homogeneous of degree k 2 1, we can write Integrated Gradients in Eq. 2 as
OF (T(()) @m(a) da J.¹ OF (ax)) dax; do
daz, da
OF (axr
I' a
OF (x) dz, I;
OF (I) dr,
(F, 0) lim
x; da
lim B-00
da
k Ds
(3)
8-+
daa,
The gradient expression in Eq. (3 can be computed using single forward/backward pass.
While Ancona et al. 1 already found that InputxGradient Gradient equals Integrated Gradients with the zero baseline for linear models or models that behave linearly for selected task, our Proposition (.2 is more general: We only require strictly positive homogeneity of an arbitrary order k 2 1. This allows us to consider larger class of models including nonnegatively homogeneous DNNs, which generally are not linear.
Definition 3.3. We call DNN F: R + IR nonnegatively homogeneous, if F(az)) aF(x)) for all a E R2o-
Corollary 3.4. Any nonnegatively homogeneous DNN is efficiently axiomatically attributable w.r.t. the zero baseline O E T.th and closed-form solution of the axiomatic feature attribution method Integrated Gradients, requiring only one forward/backward pass, exists.
Proof. Corollary 3.4 follows directly from Proposition 3.2 and Definitions and 3.3
Definition 3.5. We let X'DNN denote a nonnegatively homogeneous DNN. Further, for any X'-DNN F: IR + R. we let RY-Gradient (X'G) be an axiomatic feature attribution method relative the zero baseline O E IR" defined as
OF(x)
(F,z) IG, = I, Ox;
(4)
Note that while the formulas for the existing attribution method InputxGradient Gradient 227 and our novel X' X-Gradient are equal, X-Gradient is only defined for the subclass of X'-DNN and provably satisfies axioms that are generally not satisfied by ImputxGradien homogencity of X'-DNNs it follows that X-Graadeent attributions are also nonnegatively homogeneous. This allows us to define another desirable axiom that is in line with intuition about how attribution should work and that is satisfied by X' -GGradient.
Additionally, from the nonnegative
Definition 3.6. An attribution method A satisfies nonnegative homogeneity if A(F, ozz, ox') A(F, x, x²) for all Rzo.

---

Table 1: Overview of different gradient-base bassd DNN attribution methods and the axioms (34] that they provably satisfy. The left-hand side methods (Integrated Gradients, Expected Gradients) induce one to two orders of magnitude of computational overhead compared to the methods on the right-han hand side. The methods on the right-hand side require only one gradient evaluation (indicated by (1) for Expected Gradients with one reference sample), and thus, can be computed in single forward/backward pass. Note how X' X'Graaieen satisfies all axioms while requiring as little computational cost as a simple gradient evaluation, however being only defined for X-DNNS.
Integrated
Expected Gradients
Expected Gradients(1)
(Input x) Gradient
Axiom
Gradients
X-Gradient
Sensitivity (a)
x
x
Sensitivity (b)
Implementation invariance Completeness
X
X
Linearity Symmetry-preserviig
x
X
For an overview of the axioms (34] that are satisfied by popular gradient-based attribution methods, see Table [ The right-hand side methods use only one gradient evaluation, and therefore, have similar computational expense. The left-hand side methods generally require multiple gradient evaluations until convergence, making them correspondingly computationally more expensive. Note that X Y-Grrdieet satisfies all the axioms satisfied by Integrated Gradients and Expected Gradients (7]. assuming convergence of the latter, while requiring only fraction of the computational cost, however being only defined for X'-NNNs. Existing methods that have similar computational expense as X' Gradient generally do not satisfy all of the axioms, and therefore, are likely to produce lower quality attributions, which can be misleading and less effective for imposing attribution priors.
Constructing -D-NNNs With this motivation in mind, we will now study concrete instantiations of nonnegatively homogeneous DNNs. Note that this class of DNNs has already been considered by Zhang et al. [40], however, neither the same level of detail nor in the context of feature attributions. We define the output of a regular feedforward DNN F:
I² + R", for an input x E R", as recursive sequence of layers / that are applied to the output of the respective previous layer:
{ w/ (Ør (W)Fi- + bi)) if 2 1 x
Fi (x)
(5)
if ff 0.
with Wi and by being the weight matrix and bias term for layer 1, di being the corresponding activation function, and eir being the corresponding pooling function. Both di and t are optional; alternatively they are the identity function. For simplicity, we assume that the last task-specific layer, e.g., the softmax function for classification tasks, is part of the loss function. Further, for cleaner notation that aligns with R4], we assume w.l.o.g. that we are only considering one output node at time, e.g., the logit of the target class for classification tasks. This yields the DNN F: 1 + IR that we consider and allows us to directly compute the derivative of the model w.r.t. an input feature x4.- Importantly, the above formalization comprises many popular layer types and architectures. For example, fully connected and convolutional layers essentially matrix multiplications (36), and therefore, can be expressed by Eq. (5) Skip connections can also be expressed as matrix multiplication by appending the identity matrix to the weight matrix so that the input is propagated to later layers [36]. This allows
us to describe even complex architectures such as the ResNet [ variant proposed by [40]. As the above definition of DNN includes models that are generally not nonnegatively homogeneous, have to make some assumptions.
Assumption 3.7. The activation functions Φi and pooling functions t/i in the model are nonnegatively homogeneous. Formally, for all a E Rzy:
addi = (z( and oot/(=) = v1((z))
Proposition 3.8. Piecewise linear activation functions with two intervals separated by zero satisfy Assumption 3.7 For (21,..., 2n) E RR. these activation functions Φ1: Rh + RR" are defined as
[a112
if 0
Φi (z) = (6((2)),.....d)) (((n)) with (((zz)
if 2, S 0.
5

---

Proposition 399. Linear pooling functions or pooling functions selecting values based on their relative ordering satisfy Assumption 3.7 For (2),..., kn) R", these pooling functions vr: R + Th are defined as
1/2 (z) (18(())) (z))))
(8)
with z₃ being grouping of entries in z based on their spatial location and s//: TR. H TR being linear or a selection of a value based on its relative ordering, e.g., the maximum minimum value.
For proof' of Propositions 3.8 and 3.9
please refer the supplemental material. Activation functions
in Proposition 3.8 include ReLU
19 Leaky ReLU [6], and PReLU g Linear pooling functions
in Proposition 3.9 include average pooling, global average pooling, and strided convolutions. Other pooling functions in Proposition 3.9 include max pooling and min pooling (2), where the largest or smallest value selected. Therefore, DNN architectures satisfying Assumption 3.7 include, inter alia, AlexNet [II]. VGGNet (29], ResNet [9] introduced in [4)] and MLPs with ReLU activations. They alone have been cited well over one hundred thousand times, showing that we are considering substantial fraction of commonly used DNN architectures. However, these architectures are generally still not nonnegatively homogeneous. It is casy to see that even for a simple linear model F (z)) ax + b that can be expressed by Eq. homogeneity does not hold, because OF (z) = 0 + F(Oz). Therefore, a final step we set the bias term of each layer zero. As this may seem like significant restriction, we show in Sec. that the impact on the predictive accuracy in two different application domains is surprisingly minor.
55) and that satisfies Assumption 3.7 nonnegative
Corollary 3.10. Any regular DNN given by Eq. (5) satisfying Assumption 3.7 can be transformed into an X-DNN by removing the bias term of each layer.
Proof. A DNN with L layers given by Eq. 5 with all biases by set to 0 can be written as F(x)= th (&L (W. (0r (W1+)))))) As all the pooling functions 4/2 activation functions di.
and matrix multiplications W; in F are nonnegatively homogeneous, follows that F (axx) = af (3) for all a E 220-
Further discussion. We additionally note that our results have interesting consequences for DNNs
in certain application domains, e.g., in computer vision, as they allow to relate efficient axiomatic attributability to desirable properties of DNNs:
Remark 3.11. If a DNN F; IR" + R. taking an image x E Rn as input, is equivariant w.r.t. to the image contrast, it is efficiently axiomatically attributable.
This observation follows directly from the fact that contrast equivariance implies nonnegative homo- geneity. Consequentially, contrast-equivariant DNNs for regression tasks, such as image restoration or image super-resolution, are automatically efficiently axiomatically attributable. For classification tasks, such as image classification or semantic segmentation, contrast equivariance of the logits at the output implies efficient axiomatic attributability. If the classification done using a softmax, then this also implies contrast invariance of the classifier output. In other words, there is close relation between efficient axiomatic attributability and the desirable property of contrast equi-/invariance We further illustrate this experimentally in Sec. 4.4
Limitations. So far, we have discussed the advantages of XD-NNs such being able to efficiently compute high-quality feature attributions. However, we also want to mention the limitations of our method. First, our method can only be applied certain DNNs satisfying Assumption 3.7 Although this is a large class of models, our method is not completely model agnostic as other gradient-based attribution methods. Second, removing the bias term might be disadvantageous in certain scenarios. Intuitively, the bias term can be seen like an intercept in a linear function, and therefore, is important fit given data, As a matter of fact, a DNN without bias terms will always produce a zero output for the zero input, which might be problematic. Additionally, prior work argues that the bias term an important factor for the predictive performance of a DNN [36]. However, these theoretical foundations are somewhat contradictory to our own practical findings. In Sec. 4 we show that removing the bias term has less of negative impact than perhaps expected, indicating that removing it can be plausible intervention on the model architecture. As we cannot guarantee that this holds for all DNNs, we recommend that practitioners who plan on using our method, first make a preliminary analysis of whether removing the bias from the model at hand is plausible. Third, our method uses implicitly the zero baseline As F(O) O, this a reasonable choice because it can be interpreted as being neutral (34]. Nevertheless, other baselines could produce attributions that are

---

Table 2: Top-5 accuracy on the ImageNet (see supplemental material) of Input Gradient for regular DNNs resp. X' T--Grdiient for X--NNs to the numerical approximation of Integrated Gradients. Note how removing the bias (A'DNN) impairs the accuracy only marginally while reducing the mean absolute relative difference to Integrated Gradients significantly, confirming our theoretical finding that &' X-GGradient equals Integrated Gradients.
2 validation split and mean absolute relative difference
Top-S accuracy (%, f)
Mean absolute relative difference (%, 4)
Model
AlexNet
VGG16
ResNet-50
AlexNet
VGG16
ResNet-50
Regular DNN X'-DNN
79.21
90.44
92.56
79.0
97.8
93.8
78.54
90.25
91.12
1.2
0.4
0.0
better suited for certain tasks [21 33 37. For example, the zero baseline will generally assign lower attribution scores to features closer to zero, which can result in misleading attributions. Whether the advantages outweigh the disadvantages must be decided for each application, individually. In Sec. 4 we demonstrate the advantages of X'-DNNs, beating ofate-of-the-ar generic attribution methods for training with attribution priors.
4 Experiments
To demonstrate the practicability of our proposed method, we now evaluate it in various experiments using two different data domains confirm the following points: (/) It is plausible to remove the bias term in order to obtain X'-DNNs. (2) Our X'-Gradient method produces superior attributions compared other efficient gradient-bast based attribution methods. (3) Our X-Gradient method has advantages over state-of ofet-e--rttt-tttteee generic attribution methods for training with attribution priors. (4) X'-DNNs are robust to multiplicative contrast changes.
Experimental setup. For our experiments on models for image classification, Lee. Section 4. 4.2
4.4 we use the ImageNet [24] dataset, containing about 1.2 million images of 1000 different categories. We train on the training split and report numbers for the validation split. In Sec. 4.2 we
quantify the quality of attributions for image classification models by adapting the metrics proposed by Lundberg et al. [15] to work with image data. These metrics reflect how well an attribution method captures the relative importance of features by measuring the network's accuracy its output logit of the target class while masking out a progressively increasing fraction of the features based on their relative importance. For example, for the Keep Positive Mask (KPM) metric, the output logit of the target class should stay as high as possible while progressively masking out the least important features. As a mask we use Gaussian blur of the original image. For a detailed description of the metrics, please refer to [15] or the supplemental material. If not indicated otherwise, we assume numerical convergence for Integrated Gradients and Expected Gradients, which we found to occur after ~ 128 approximation steps (see supplemental material).
4.1 Removing the bias term in DNNs
Historically, the bias term plays an important role and almost all DNN architectures use one. In this first experiment, we evaluate how much removing the bias to obtain an X-DNN affects the accuracy of different DNNs. To this end, we train multiple popular image classification networks, AlexNet [I], VGG16 29). and the ResNet-50 variant of (40). as well as their corresponding X'DNNN variants obtained by removing the bias term, on the challenging ImageNet (24 dataset. The resulting top-5 accuracy on the validation split is given in Table 2 As we can observe, removing the bias decreases the accuracy of the models only marginally. This is a somewhat surprising result since prior work indicates that the bias term DNNs plays an important role [6]. We hypothesize that when removing the bias term, the DNN learns some kind of layer averaging strategy that compensates for the missing bias. For an additional comparison between a DNN with bias and its corresponding X' X'DNN in non-vision domain, see Sec. 4.3 which mirrors our findings here. Additionally, to empirically validate our finding that --Grrdiennt (XG) equals Integrated Gradients for X'DNNs, we report the mean absolute relative difference (see supplemental material) between the attribution obtained from Integrated Gradients 34 and the attribution obtained from computing Inputx Gradient for regular DNNs resp. X' X'GGradient for X' -DDNNs over the ImageNet validation split. For regular models with
7

---

Table 3: Metrics of Lundberg et al. 15 to measure the attribution quality of different attribution methods. Please refer to the experimental setup in the beginning of Sec. 4 and the supplemental material for an introduction of the metrics. We evaluate Integrated Gradients (IG) [3]], random attributions (Random), imput gradient attributions (Grad), Expected Gradients (EG) 7 and our novel
X-Grrdient (2'G) attribution on a regular AlexNet [40] and the corresponding X' -eAlexNet. The
numbers in parentheses indicate the required gradient calls. Our method on par with IG in terms of quality while requiring two orders of magnitude less computational power.
AlexNet
X'AAlexNet
Method
KPM f
KNM +
KAM f
RAM
KPM
KNM 4
KAM
RAM
IG (128)
7.57
1.67
25.22
11.12
7.38
2.21
21.79
11.68
Random
3.68
3.68
14.12
14.10
3.81
3.81
13.52
13.50
Grad (1)
3.62
3.88
20.78
11.82
3.87
4.34
19.75
11.25
EG (I)
4.92
2.97
20.49
13.76
5.41
3.19
19.47
13.19
XGG (1)
N/A
N/A
N/A
N/A
7.38
2.21
21.83
11.68
biases, Integrated Gradients produce a very different attribution compared to InputxGradient For X-DNNs on the other hand, the two attribution methods are virtually identical, as expected. The small deviation can be explained by the fact that the result of Integrated Gradients (34] is computed via numerical approximation, whereas method computes the exact integral (of course only for X X-DNNs). We make the pre-trained XDDNN models publicly available to promote a wide adoption of efficiently axiomatically attributable models.
4.2 Benchmarking gradient-basee based attribution methods
As prior work [4] but also our experiment Sec. 4.3 suggest that the quality of an attribution method positively impacts the effectiveness of attribution priors, we benchmark our method against existing gradient- gradient-based attribution methods that are commonly used for training with attribution priors. For evaluation, use the metrics from [15] adapted to work with image data. Using these metrics allows for a diverse assessment of the feature importance [15] and ensures consistency with the experimental setup in 77. Table 3 shows the resulting numbers for regular AlexNet and our corresponding XeAlleNet.. Due to the axioms satisfied by the Integrated Gradients method, it produces the best attributions for the regular network, which is in line with the results in (38) However, as it approximates an integral where each approximation step requires an additional gradient evaluation, it also introduces one two orders of magnitude of computational overhead compared to the other methods (Sundararajan et al. (34 recommend 20-300 gradient evaluations to approximate attributions). For the AlelleNet, however, Integrated Gradients and produces the best attributions while requiring only one gradient evaluation, and therefore, fraction of the compute power. Since the input gradient and Expected Gradients 77] with only one reference sample do not satisfy many of the desirable axioms (see Table D they produce clearly lower quality attributions as expected. Note that high-qualitative attribution methods should perform well across all the listed metrics, which is why the input gradient is not a competitive attribution method even though it performs well on the RAM metric. To conclude, we can see that our X' Gradient attribution yields a significant improvement in quality compared to state-of-the-an generic attribution methods that require similar computational cost. This suggests that our effort to produce an efficient and high-quality attribution method is justified and accomplished.
X' X'Grradient method on par with
4.3 Training with attribution priors
To benchmark our approach against other attribution methods when training with attribution priors, we replicate the sparsity experiment introduced in [7]. To that end, we employ the public NHANES I survey data [17] of the CDC of the United States, containing 118 one-hot encoded medical attributes, e.g., age, sex, and vital sign measurements, from 13,000 human subjects (no personally identifiable information). The objective of the binary classification task to predict if a human subject will be dead (0) or alive (1) ten years after the data was measured. A simple MLP with ReLU activations used as the model. Therefore, it can be transformed into an XDDNN by simply removing the bias terms. To emulate setting of scarce training data and to average out variance, we randomly

---

0.775
ROC-AUC AUC (Arg)
ROC-AUI (Avg)
0.77
0.750
0.725
0.76
0.700
Expected Gradients
0.75
X- X-Grddient
0.675
X2
EG (1)
RRR.
Grad
Uureg
Uureg w/o bias
100
120
w/o biss
Reference samples
Figure 1: (left) Average ROC-AUC across 200 randomly subsampled datasets for the same attribution prior using different attribution methods. "w/o bias" denotes that the bias term has been removed from the MLP. (right) Average ROC-AUC across 200 randomly subsampled datasets of Expected Gradients (EG) over the number of reference samples. The current state-of of state-of-the-an tee-hhe-rttt EG requires
approximately 32 reference samples, and thus, 32 times more computational power to outmatch YG. Confidence intervals indicate two times the standard error of the mean.
subsample 200 training and validation datasets containing 100 data points from the original dataset. Erion al. 7 proposed a novel attribution prior that maximizes the Gini coefficient, i.e., minimizes the statistical dispersion, of the feature attributions. They show that this allows to learn sparser models, which have improved generalizability on small training datasets. The more faithfully the attribution reflects the true behavior of the model, the more effective the attribution prior should be.
Comparing attribution methods. We compare different attribution methods that have previously been used for training with attribution priors and require only one gradient evaluation; thus, they have comparable computational cost. The results in Fig. [left) show that our method (A' w/o bias) outperforms all other competing methods. We can also see that for the unregularized model, removing the bias (Unreg w/o bias) has almost no effect on the average ROC-AUC of the method, once again showing that our modification for making attributions efficient, i.e., removing the bias term, is plausible various scenarios.
Since the attribution quality of Expected Gradients can be improved using more reference samples, as this yields better approximation to the true integral. we plot the average ROC-AUC Expected Gradients over the number of reference samples used in Fig. right.. We can clearly see that adding more samples improves the ROC-AUC when training with EG attribution the prior, yet again, showing that higher quality attributions lead to more effective attribution priors 77. However, we also find that approximately 32 reference samples are needed, and hence 32 times more computational power, to match the quality of our efficient X' R'-Gradient method. When using more than 32 reference samples, Expected Gradients slightly outperform our method in terms of ROC-AUC, which is due to the limitations discussed in Sec. 3 (fixed baseline, no bias terms). We argue that it is often worth accepting this small accuracy disadvantage light the significant gain in efficiency computing high-quality feature attributions.
To put this improvement in efficiency into perspective, we measure the computation time of training a ResNet-50 on the ImageNet dataset when using Expected Gradients with 32 reference samples and X' V-Gradient (see supplemental material). Using a single GPU, the computational overhead introduced when using Expected Gradients with 32 reference samples amounts to an approximately 130-fold increase in the required computation time compared to training with X X-Grradenn, and thus, would turn several days of training into several months of training.
4.4 Homogeneity of XD-NNS
The fundamental difference between X' 2'DNNN and regular DNNs is the nonnegative homogeneity of the former. To show implications on the model and its attributions, we conduct the following experiment. Similarly to Hendrycks and Dietterich [10], we reduce the contrast of the ImageNet [24] validation split by multiplying each image with varying factors a and report the top-1 accuracy of
AlexNet and the corresponding A'AAlexNet from Sec. 4.1 iee. they have not been trained specifically
to handle contrast changes. Results can be found Fig. leeft). We can observe that decreasing the
image contrast leads strong drop in accuracy of regular AlexNet. On the other hand, due the

---

1.0
AlexNeet
0.8
AlAleeNeet
Top-1 acsuracc
0.6
0.4
0.2
0.0
1.00
0.75
0.50
0.25
Image
XG
IG
IxG
IG
Scaling factor
X-AlexNet
NlexNet
Figure 2: (left) Top-1 accuracy for AlexNet and A'-AlexNet the ImageNet validation split with decreasing contrast (scaled by a).. Due to the nonnegative homogeneity of X'-AlexNet, the accuracy does not drop when reducing the contrast. (right) Qualitative examples of normalized attributions for Net and AlexNet using the attribution methods X' T-Grrdieen (2'G) resp. InputxGradient Gradient (IxG) as well Integrated Gradients (IG). The displayed attributions obtained from X'-AlexNet are almost identical, while attributions obtained from AlexNet differ significantly (see highlighted areas).
equivariance contrast of X' X'-DNNs, the accuracy of the R'-AlexNet is unaffected, showing improved robustness towards multiplicative contrast changes. To give some qualitative examples, in Fig. 2(ight) we plot the attributions for the output logit of the target class ('reflex camera') for regular AlexNet and an X-AAleNNet for an original image and the corresponding low-contrast image obtained by multiplying the normalized image with 0.3. For the X-AleeNet, our X-Gradient method and Integrated Gradients (IG) (34) produce attributions that are identical up to small approximation error, reducing the image contrast keeps the attributions unchanged up to scaling factor (not visible due to normalization for display purposes). However, the displayed attributions from the regular AlexNet differ significantly between InautxGr (see highlighted areas) We argue that the above properties of X'-NNNs generally reflect desirable properties and show that they behave more predictably with contrast changes than regular DNNs. Also note how high feature attribution scores can arise in the background (e.g., red box), showing how DNN predictions can depend parts of the input that do not appear salient for the category; this highlights possible use case for attribution priors 22 23 enabled by our approach.
Gradient (IxG) and IG, as well as for different contrasts
5 Conclusion and broader impact
In this work, we consider a special class of efficiently axiomatically attributable DNNs, for which an axiomatic feature attribution can be computed in a single forward/backward pass. We show that nonnegatively homogeneous DNNs, termed X'-DNNs, are efficiently axiomatically attributable and establish a new theoretical connection between Input Gradient (27] and Integrated Gradients (34) for nonnegatively homogeneous DNNs of different degrees, Moreover, we show that many commonly used architectures can be transformed into X'-NNNs by simply removing the bias term of each layer, which has surprisingly minor impact on the accuracy of the model in two application domains. The resulting efficiently computable and high-quality feature attributions are particularly well-suited for inclusion into the training process and potentially enable wide application of axiomatic attribution priors. This could, for example, be used to reliably mitigate dependence on unwanted features and biases induced by the training dataset, which is major challenge in today's 's ML systems.
Obermeyer et al. 20 found evidence that a widely used algorithm in the U.S. health care system contains racial biases that can be traced back to biases in the dataset that was used to develop the algorithm. Using our method to generate high-quality attributions that reflect the true behavior of an X'DNN and an appropriate attribution prior, such problems could potentially be resolved (though more research on attribution priors is necessary). However, allowing biases to be controlled by an ML practitioner can introduce new risks. Just like datasets, humans are also not free of biases [35, which can potentially be reflected such priors. We as a society need to be careful that this responsibility not exploited and used for discriminatory or harmful purposes. One way to approach this problem for applications that affect the general public to introduce an ethical review committee, which assesses whether the proposed priors are legitimate or reprehensible.
10

---

Acknowledgments and disclosure of funding
We would like to thank Jannik Schmitt for helpful mathematical advice. This project has received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No. 866008). The project has also been supported in part by the State of Hesse through the cluster projects "The Third Wave of Artificial Intelligence (3AI)" and "The Adaptive Mind (TAM)".
References
[1] M. Ancona, E. Ceolini, C. Öztireli, and M. Gross. Towards better understanding of gradient- based attribution methods for deep neural networks. In ICLR, 2018.
[2] M. Ancona, E. Ccolini, C. Öztireli, and M. H. Gross. Gradient- basee attribution methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, volume 11700 Lecture Notes in Computer Science, pages 169-191. Springer, 2019.
[3] J. Angwin, J. Larson, S. Mattu, and L. Kirchner. There's s software used across the country to predict future criminals. and it's biased against blacks. ProPublicu, 2016.
[4] A. B. Arricta, N. D. Rodríguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, and F. Herrera. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion, 58:82-115 115, 2020.
[5] S. Bach, Binder, G. Montavon, Klauschen, K.-.. Müller, and Samek. On pixel- wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1-46, 2015.
[6] Chen, X. Wu, V. Rastogi, Y. Liang, and Jha. Robust attribution regularization. Neu11PS*2019, pages 14300-14310.
[7] G. Erion, J. D. Janizek, P. Sturmfels, S. Lundberg, and S.-L. Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature Machine Intelligence, 3(7): 620-631, 2021.
[8] K. He, Zhang, Ren, and Sun. Delving deep into rectifiers: Surpassing human-lever performance on ImageNet classification. In ICCV, pages 1026-1034, 2015.
[9] K. He. X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[10] D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
[11] A. Krizhevsky, L Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. 2012, pages 1106-1114.
[12] Y. LeCnn, Y. Bengio, and G. E. Hinton. Deep learning. Nature, 521(77553:436-444 2015.
[13] Li, W. Monroe, and D. Jurafsky. Understanding neural networks through representation erasure. [cs.CL], 2016.
[14] F. Liu and B. Avci. Incorporating priors with feature attribution on text classification. In ACL, pages 6274-6283, 2019.
[15] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-L. Lec. From local explanations global understanding with explainable Al for trees. Nature Machine Intelligence, 2(1):56 67,666677667 2020.
[16] A. Maas, Y. Hamnun, and Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In ICMI. Workshop on Deep Learning for Audio, Speech, and Language Processing, 2013.
[17] H. W. Miller. Plan and operation of the health and nutrition examination survey. United States-1971-10 1971-1973. Vital and Health Statistics. Ser. 1, Programs and Collection Procedures, 1(10a):1-44 1973.
[18] W. J. Murdoch, P. J. Liu, and B. Yu. Beyond word importance: Contextual decomposition to extract interactions from Istms. In ICLR, 2018.

---

[19] V. Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, pages 807-814, 2010.
[20] Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464):447455, 2019.
[21] D. Pan, X. Li, and D. Zhu. Explaining deep neural network models with adversarial gradient integration. In IJCAI, pages 2876-2883, 2021.
[22] L. Rieger, C. Singh, W. Murdoch, and B. Yu. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In ICMI., volume 119, pages 81 16-8812, 2020.
[23] A. S. Ross, M. C. Hughes, and F. Doshi-Velez. Right for the right reasons: Training differentiable models by constraining their explanations. In IJCAI, pages
2017.
[24] O. Russakovsky, Deng, Su, J. Krause, S. Sathcesh, S. Ma. Z. Huang, Karpathy, A. Khosla, M. Bemsstein, A. C. Berg, and L. Fei-Fei. ImageNet large scale visual recognition challenge. Int. Comput. Vision, 115(13):21-225)) 2015.
[25] J. Schmidhuber. Deep learning in neural networks: An overview. Neural Nenworks, 61:85-117, 2015.
[26] P. Schramowski, W. Stammer, S. Teso, A. Brugger, F. Herhert, X. Shao, H.-G. Luigs, A.-K. Mahlein, and K. Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):476 4864466666666 2020.
[27] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kundaje. Not just a black box: Learning important features through propagating activation differences. frXiv:1605.0177 [cs.LG]. 2016.
[28] A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation differences. ICML, volume 70, pages 3145-3153, 2017.
[29] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[30] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
[31] C. Singh, J. Murdoch, and B. Yu. Hierarchical interpretations for neural network predictions. In ICLR. 2019.
[32] R. Socher, E. H. Huang, J. Pennington, A. Y. Ng. and C. D. Manning. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS*2011, pages 801-809.
[33] P. Sturmfels, S. Lundberg, and S.-I. Lee. Visualizing the impact of feature attribution baselines. Distill, 5, 2020.
[34] Sundararajan, Taly, and Q. Yan, Axiomatic attribution for deep networks. In ICML, volume 70, pages 3319-3322 3328, 2017.
[35] A. Tversky and D. Kahneman. Judgment under Uncertainty: Heuristies and Biases. Science, 185(4157): 111, 1974.
[36] S. Wang, T. Zhou, and A. Bilmes. Bias also matters: Bias attribution for deep neural network explanation. In ICMI., volume 97, pages 6659-6667, 2019.
[37] Z. Wang, M. Fredrikson, and A. Datta. Robust models are more interpretable because attributions look normal. arXiv:2103.11257 [cs. 2021.
[38] C. Yeh, C. Hsieh, A. S. Suggala. D. I. Inouye, and P. Ravikumar. On the (in)fidelity and sensitivity of Neu1IPS=2019, pages 19965-10977
explanations.
[39] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV, volume 1, pages 818 833, 2014.
[40] H. Zhang, Y. N. Dauphin, and T. Ma. Fixup initialization: Residual learning without normalization. In ICLR, 2019.
[41] J. Zhou and O. G. Troyanskaya. Predicting effects of noncoding variants with deep learning based sequence model. Nature Methods, 12(10)931-934, 2015.
[42] L. M. Zintgraf, T. S. Cohen, T. Adel, and M. Welling. Visualizing deep neural network decisions: Prediction difference analysis. In ICLR, 2017.
12


Reviews:
[{"review_id": "scqs7XsdXy", "reviewer": "Reviewer_bst1", "summary": "The authors propose a class of DNNs, which they call non-negatively homogenous, where the Integrated Gradient (which satisfy certain desirable axioms) for feature attribution can be computed in closed-form using a single gradient evaluation (as opposed to multiple gradients calls for a Riemann sum as in Sundararajan et al 2017). The authors propose instantiating such DNNs in practice by removing the bias term. They conduct experiments on two tasks where they show that removing this bias does not cause a substantial drop in accuracy while still computing IGs efficiently.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The notation is clear and the propositions/definitions are adequately explained informally which makes the paper easy to understand. Additionally, the authors make an effort to anticipate a reader's questions and answer them which makes it easy to track their specific contributions.\n\nThe primary novel contributions are proposing a new class of DNNs that can be instantiated in practice by removing a bias term and proving a closed-form IG for this class. The method is efficient because it only requires a single gradient call.\n\nRe “removing bias term”: The authors sufficiently acknowledge this limitation. It was interesting to see that at least for the experiments in the paper, accuracy does not decline too much. I imagine that in practice, a practitioner could conduct a preliminary analysis to see whether removing the bias hurts performance (through a validation set) and use the proposed DNN if it does not. I think some more suggestions for deciding when this method is appropriate (from a practitioner’s standpoint) would be valuable to add to the paper.\n\nRe “efficiency”: The authors claim that attributions methods need to be invoked at each training step. Does that only happen when attribution priors are used for regularization during training (as in Eq. (1))? Without these priors, it seems to me that attribution only needs to be computed after training a network as opposed to every step. Are there other cases where each training step needs attribution computation?\n\nTable 3 and lines 294-296: The results show that the RAM value for Grad (1) is the best (and does reasonably well for the standard AlexNet too). This is not addressed in the discussion. What are the potential reasons for this?\n\nLine 337 “leads to a strong drop inaccuracy”: I found this surprising given that AlexNet is strictly more expressive than the $\\chi$-DNN. I would have imagined that some amount of hyperparameter tuning would have prevented this drop in accuracy.\n\nFigure 2 (right): Are the qualitative attributions shown for the cases where both DNNs predict the correct class label? I am wondering if the less interpretable attribution for AlexNet is simply because it is predicting the wrong label due to the drop in accuracy.\n\nTypo in Eq (5): It should be $i \\geq 1$.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, there are two main limitations and the authors acknowledge and address both of them: (i) removing the bias term limits expressivity of the DNN and (ii) the attribution needs a baseline of zero.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "rN5UJHs5csq", "reviewer": "Reviewer_gwR5", "summary": "The paper first identifies the intensive computational cost in using Integrated Gradient (IG) as an attribution prior in training explainable deep neural networks. To this end the authors propose to switch the network architecture to negatively homogeneous networks, specifically networks without bias term, a.k.a. $\\mathcal{X}$-DNN, in this paper, where they prove that $\\texttt{input} \\times \\texttt{gradient}$ is an equivalent attribution method to IG; and therefore can decrease the number of gradient evaluations in computing IG. The paper then conducts several experiments over the ImageNet dataset and several tabular datasets to validate the performance of the proposed method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I find the theoretical results are very interesting and it opens another fresh view of connecting Integrated Gradient with inherent properties of networks, just like the connection between randomized smoothing and SmoothGrad. However, the evaluation part of the paper is confusing to me, which prevents me from giving higher scores. However, I am on the fence and expect the authors’ response to clear my concerns. Please find the detail as follows.\n\n### Originality\nThe work is a novel combination of well-known techniques. The related work is well-cited.\n\n### Quality: \nI have two major concerns on the quality side of this paper. Please find the detail as follows:\n1) Theoretical limitations of no-bias network, a.k.a.$ \\mathcal{X}$-DNN. The authors provide empirical evidence that $\\mathcal{X}$-DNNs match the Top-5 accuracy of regular networks in Section 4.1; however the experiment is only limited to one data distribution and one network, which is far away from covering all kinds of applications the author believes their work can potentially benefit for, e.g. health care. I would suggest that more theoretical analysis can be provided to convince the reader $\\mathcal{X}$-DNNs are as useful as regular networks. An example question is: assuming zero vectors as baselines can decourage the network to use features with values close to 0. For example, it will always consider darker pixels as not useful so black dogs may always have smaller attribution values compared to white dogs. Will the $\\mathcal{X}$-DNN end up with becoming a network that is always more sensitive to “bright” features and less sensitive to “darker” features that are actually important?  \n\n2) Evaluations are not well-motivated. My biggest concern is that several proposed evaluations and their metrics are not well-motivated to show the proposed network architecture is promising. \n- Firstly, as$\\mathcal{X}$G is not a new method but more like an alias of input x grad for networks without bias and is mathematically equivalent to IG in the proposed architecture, the result of Table 3 is actually comparing existing attributions using existing metrics (KPM, KNM, KAM and RAM). More analysis could be added to justify why Table 3 is important to include and why the result is useful to show the proposed $\\mathcal{X}$-DNN is promising.  Besides, the motivations of using metrics from [1] instead of other work, e.g. [2], are unclear to the reader. \n\n- Secondly, the paper is motivated to accelerate the training with attribution priors but statistics of the timing and memory cost in training $\\mathcal{X}$-DNN are missing in Section 4. More numerical results instead of descriptions, e.g. line 328-330, will show exactly how fast and efficient the proposed algorithm is. \n\n[1] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-I. Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence. \n\n[2] Yeh, C., Hsieh, C., Suggala, A.S., Inouye, D.I., & Ravikumar, P. (2019). On the (In)fidelity and Sensitivity of Explanations. NeurIPS.\n\n### Clarity\nThe motivation is clearly stated: the attribution prior regularization is usually computational expensive in terms of the number of gradient evaluations w.r.t the input; therefore, it is naturally to find another way of computing attributions with less number of gradient evaluations while the attribution used in the training is also faithful; \n2) the paper describes the motivation and the formulation of \\mathcal{X}-DNN clearly through multiple definitions and propositions. The equations and proofs are easy to understand. \n\n### Significance\nThe results are important and others (researchers or practitioners) are likely to use the ideas if the technical questions can be resolved accordingly. \n\n### Update from discussion\nThanks for the authors to post the thoughtful responses to my questions. Even though some questions are probably not able to resolve during this discussion time, e.g. the theoretical guarantees about the bias-free networks, I generally find this paper to be interesting and the authors' response is convincing and they are trying to make the changes that can answer my questions in the future revision. I therefore will increase my score from 5 to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "nglL7xxs_jp", "reviewer": "Reviewer_wVX5", "summary": "This paper proposes to use non-negatively homogeneous DNN to perform feature attribution with better computational efficiency at a minimal performance cost. The so-called non-negatively homogeneous DNN is obtained by removing the bias term from most common neural networks and using a Relu/Leaky_Relu activation function. By doing this, the model would be linearly scaled by the input data (F(ax) = a*F(x)). With this property, the integrated Gradient (IG) method can be easily computed with one call of gradient computing, which improves its original computational efficiency. In the experiment, this model shows comparable performance with the IG method but much less computation effort.\n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I’m not an expert in feature attribution research but this paper sounds like an interesting discovery based on the existing IG method, which takes advantage of neural networks that are non-negatively homogeneous after removing their \"Bias\" terms. However, the key idea of this paper that improves the IG method looks very simple and straightforward (for myself, I know the non-negatively homogeneity of DNNs without bias term a long time ago from many papers discussing the property of DNNs. e.g., for papers like \"Analytic Marching: An Analytic Meshing Solution from Deep Implicit Surface Networks\", they already kind of show these results by analyzing the local linearity of the neural networks). At least, this conclusion is not very novel from my perspective. On the other hand, this discovery could be an interesting application of this \"property\" on the feature attribution research. Therefore, I think the paper should give a better introduction to the background of this problem (feature attribution) and demonstrate that there is enough contribution based on the efficiency improvement of the IG method. For example, is \"IG\" the state-of-the-art method for feature attribution while being limited by its computational efficiency? Is there any other feature attribution method that are efficient compared with one-gradient-call IG?\n\nMinor questions:\n1. for clarity, can the author provide a more solid example of what the Integrated Gradients method really does?\n2. What is the relative attribution distance? (Mentioned in Table 2)\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "6aMJgJ3LIbC", "reviewer": "Reviewer_28k9", "summary": "The paper addresses the question of attributing the predictions of deep neural networks to the input features. It proposes to combine a particular restriction on the prediction function: non-negative homogeneity (e.g. a deep neural networks without biases) with the fast gradient x input explanation technique. With this combination, it is possible to produce explanations that satisfy the completeness axiom that can be computed quickly (from a single function evaluation). The paper also argues that non-negative homogeneity makes the model invariant to contrast, which is a useful property in image recognition.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contributes theoretical results, such as the connection between Integrated Gradients and Gradient x Input for nonnegatively homogeneous functions, the relation between gradient x input and the completeness axiom, as well as the connection between gradient x input and the sensitivity axiom. However, some of these results, in particular, the completeness of gradient x input, are already known and described in earlier papers for the case k=1, which is arguably the most important case since it covers standard deep neural networks. A practical motivation for the higher-order case (k>1) is not provided in the paper.\n\nThe paper is mathematically rigorous. However, in Table 1, I'm not sure why Input x Gradient would fail the symmetry and sensitivity(a) tests. I would encourage the authors to provide the corresponding counter-examples in the supplement. Regarding the content of the paper, references need to be added to previous work related to explaining positively homogeneous functions and on the completeness of gradient x input. \n\nThe paper is generally clear. However, Table 1 is misleading: Non-negative homogeneity is not an axiom (or some desirable property of the method) but a modeling constraint. Maybe a more suitable line to add in the table would be \"computable in a single forward/backward pass\". In Table 3, there is no description of what KPM, KNM, etc. mean (only a reference to the paper introducing them). A brief introduction of these metrics would ensure that this table can be fully understood.\n\nThe significance of the work is limited. The theoretical contributions are incremental, and the novel aspects (k>1, sensitivity) are not the focus of the experiments. Regarding the practical applicability of the method, removing biases requires retraining the model (which might offset the benefits of reducing gradient evaluations at test time). If the model is instead meant as a self-explainable model, this should be stated explicitly, and the proposed approach should be compared with other self-explainable models, e.g. bag-of-local-features.\n\nLastly, the proposed X-gradient approach satisfies the completeness axiom, but it is still based on a single gradient evaluation, which raises the question how it could reliably explain a function that contains multiple scales (or some strong local gradient variation superposed to weaker global variations).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation of methods based on a single gradient evaluation (e.g. for handling functions with multiple scales), as well as the potential cost of having to retrain the model, should be discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper focuses on feature attribution techniques (which, in a frustrating linguistic turn are often called “explanations”). These techniques are all heuristics chosen for their ability, on the basis of anecdotal evidence to highlight features that the beholder finds important. What if anything they are  “explaining” is seldom explained, and I would encourage the authors to strip the paper of gratuitous uses of the term explanation wherever it is wielded in a quasitechnical form: this includes the title “for Training Neural Networks with Explanations” and in the italicized phrase, as though introducing a technical term: "efficiently explainable”. That said, for better or ill, and the jury is still out, some of these techniques are objects of interest of a large part of the ML community, and while there has been some progress to positing axioms that these attribution maps ought to satisfy, not all methods that satisfy those axioms are known to be efficiently computable.

 In particular, the authors present a new method for computing Integrated Gradients efficiently, showing that when bias terms are removed from neural networks they are efficiently computable. Whether integrated gradients are a useful concept in the first place remains a dubious proposition (what should the baseline be and what basis could anyone possibly have for choosing one?). However, insofar as they are an object of interest and the community is willing to publish lots of papers about them, this paper strikes me as unusually useful and concrete in its offering. The reviewers debated the surprisingness of the finding and whether it was already known to some in the XAI community. However, while I have seen many papers computing integrated gradients, I have no knowledge of anyone computing integrated gradients efficiently via this trick. It seems sufficiently useful, that if it were known, you would have seen it. Questions of the value of the underlying method aside, I do not believe that a finding must require an earthshaking analysis to to be surprising to a great mathematician to be valuable and useful. If that were the case, most useful papers in ML would never have been published. The reviewer’s suggestions that a key result was already known have been rebutted by the authors (who claim that the result is only known for linear models) and the reviewer did not take the opportunity to respond. In this light, I will give the authors the benefit of the doubt. 

The authors had a constructive discussion with Reviewers bst1 and gwR5 and I trust that they will act in good faith and add the discussed modifications, clarifications, and exposition to the final draft.  I will recommend acceptance and hope that the authors will act in good faith and tone down the quasitechnical language about “explanation” and to improve the paper per the promises that arose in the discussion period.
Decision: Accept (Poster)