Exploring unfairness in Integrated Gradients based attribution methods
Authors: David Drakard
Rosanne Liu
Jason Yosinski
Venue: ICLR 2022 Submitted (2022)
Keywords: Integrated Gradients
Expected Gradients
Explainable AI
Integrated Certainty Gradients
Attribution

Abstract: Numerous  methods  have  attempted  to  explain  and  interpret  predictions  made by  machine  learning  models  in  terms  of  their  inputs.   Known  as  “attribution methods” they notably include the Integrated Gradients method and its variants.These are based upon the theory of Shapley Values, a rigorous method of fair allocation according to mathematical axioms.  Integrated Gradients has axioms derived from this heritage with the implication of a similar rigorous, intuitive notion of fairness.  We explore the difference between Integrated Gradients and more direct expressions of Shapley Values in deep learning and find Integrated Gradients’ guarantees of fairness weaker; in certain conditions it can give wholly unrepresentative results.  Integrated Gradients requires a choice of “baseline”, a hyperparameter that represents the ‘zero attribution’ case.  Research has shown that baseline choice critically affects attribution quality, and increasingly effective baselines have been developed.  Using purpose-designed scenarios we identify sources of inaccuracy both from specific baselines and inherent to the method itself, sensitive to input distribution and loss landscape. Failure modes are identified for baselines including Zero, Mean,Additive Gaussian Noise, and the state of the art Expected Gradients. We develop a new method, Integrated Certainty Gradients, that we show avoids the failures in these challenging scenarios.  By augmenting the input space with “certainty”information, and training with random degradation of input features, the model learns to predict with varying amounts of incomplete information, supporting a zero-information case which becomes a natural baseline. We identify the axiomatic origin of unfairness in Integrated Gradients, which has been overlooked in past research.

Under review as a conference paper at ICLR 2022
EXPLORING UNFAIRNESS IN INTEGRATED GRADIENTS BASED ATTRIBUTION METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
Numerous methods have attempted to explain and interpret predictions made by machine learning models in terms of their inputs. Known as "attribution methods" they notably include the Integrated Gradients method and its variants. These are based upon the theory of Shapley Values, a rigorous method of fair allocation according mathematical axioms. Integrated Gradients has axioms derived from this heritage with the implication of a similar rigorous, intuitive notion of fairness. We explore the difference between Integrated Gradients and more direct expressions of Shapley Values in deep learning and find Integrated Gradients' guarantees of fairness weaker; in certain conditions it can give wholly unrepresentative results. Integrated Gradients requires a choice of "baseline", hyperparameter that represents the 'zero attribution' case. Research has shown that baseline choice critically affects attribution quality, and increasingly effective baselines have been developed. Using purpose-desie desposdd scenarios we identify sources of inaccuracy both from specific baselines and inherent to the method itself, sensitive to input distribution and loss landscape. Failure modes are identified for baselines including Zero, Mean, Additive Gaussian Noise, and the state of the art Expected Gradients. We develop a new method, Integrated Certainty Gradients, that show avoids the failures in these challenging scenarios. By augmenting the input space with "certainty" information, and training with random degradation of input features, the model learns to predict with varying amounts of incomplete information, supporting a zero-infoon crr-oinfomatiin case which becomes a natural baseline. We identify the axiomatic origin of unfairness Integrated Gradients, which has been overlooked in past research.
INTRODUCTION
Attribution, the identification of the input features most salient to model prediction, is an increasingly important requirement for neural networks. is core part of model interpretability, which is valuable as research tool and design aid, but also increasingly as an output requirement in its own right for uses as diverse as medical imaging (Singh al. 2020) to loan applications (Bhatt et al. 2020). A "right to explanation" of machine decisions is even provided in the European Union's 's General Data Protection Regulation (Goodman & Flaxman 2017).
Gradients of predictions with respect model inputs can be calculated using backpropagation and have been used for feature attribution ("Vanilla Gradient") (Erhan al., 2009 [Simonyan et al. 20144) Yosinski et al., 2015). The gradients indicate which features causing the largest change in prediction outcome. Empirically, this method often does highlight areas relevant to prediction, but it may also miss important areas due the nonlinearity of the model and saturation of the gradients (Sundararajan et al. 2016).
sensitive to perturbation,
The method of Integrated Gradients (Sundararajan al.) 2016 2017) overcomes these issues by generating attributions based on the theory of Shapley values (Sundararajan et al.) 2017). The method integrates the gradients the input varies linearly between baseline and the final input of interest:

---

Under review as a conference paper at ICLR 2022
A
C
E
G
Figure 1: Examples of (generalized) baselines. Grey: input distribution; Red: attributed input; Green:
baselines and integration paths. [ Distribution and target input. Baselines: B: Zero: Middle;
D Distribution mean; B Maximum distance: F Additive Gaussian noise: G Expected Gradients. B are single static baselines so exhibit baseline blindness. Maximum Distance
Failure modes:
gives poor attributions due extreme O.O.D. baselines. AGN fails the Local Minimum scenario (section 5.2
EG fails the Burnt Snacks scenario (section 5.1.
(F(#)
Integrated Gradients (F,' = (x, x()
da
(1)
Oxp
x +
(x
x¹)
distance
mean gradient
where F-) denotes the prediction of the model, is the input vector of interest, x' is the baseline input vector, and i indexes to the feature of interest. A numerical approximation is normally used to calculate the integral in practice. The baseline represents a missing neutral input, concept required by the Shapley values theory, but shared with several other attribution methods (Covert et al. 2020).
In the remainder of this paper, we first give an overview of baselines discussed in the literature (Section 2 Then we discuss the theoretical cause of failure mode of IG (attribution transfer) using zero baseline example (Section B) In Section 4 we introduce a new method, Integrated Certainty Gradients. Finally we show experiments that other baselines including the state of the art Expected Gradients do not prevent attribution transfer, while in the tested scenarios Integrated Certainty Gradients does, and discuss additional failure cases of specific baselines (Section 5))
Note on notation It will be convenient to imply equivalence between attribution methods which take baseline, yo( æ', , æt, i), and those which do not (FF, xx, i). A baseline free method can be adapted to take baseline by yp (F, x', x, i) (FF, i) sF(F,'',1
2 CHOICE OF BASELINE
Baseline choice has a major effect on attribution outcome. Effective choice of baselines in practice is a topic of ongoing research. A recent review of four state-of-the-- models for tabular data did not find a best performing choice (Haug et al., (222). As this paper noted, the concept of missingness within an arbitrary space is domain-specific. Therefore applying attribution techniques to a new domain requires this concept be determined. Another review came similar conclusions for image classification attribution (Sturmfels et al. 2020).
Baselines can be generalized from single inputs distributions by taking the expected value of Integrated Gradients using each input in the distribution. In practice X' is often from an empirical (discrete uniform) distribution so the expectation becomes simply the mean over X'. A summary of common baselines used in Integrated Gradients and other attribution methods follows. For a more comprehensive review, one can refer to HHug al. (202T): Sturmfels et al. al. (2020). Visualizations of integration paths for various baselines are shown in Figure
Constant value baselines include the uniform black/zero baseline proposed with the original method (Sundararajan et al.) 2016) and choices that attempt to be unbiased with respect to the input distribution, including the component-wise mean of the input distribution, the middle of the input range, and uniform noise. If the input matches the baseline, the æ; x term in Equation I disappears, any such areas will not be attributed, even when they are essential to the model's ''s predic- tion ('baseline blindness"") (Sturmfels et al., 2020). When the baseline is constant this undesirable behavior cannot be avoided. This behavior can be seen clearly attributions of MNIST images with baseline, for example in Adebayo al. 2002
2

---

Under review as a conference paper at ICLR 2022
Distribution baselines avoid baseline blindness by combining multiple attributions. Uniform noise has been used in this way, although concerns have been identified about the risk of interaction between the baseline and high frequency features (Sundararajan et al. 2017 Sturmfels et al.) 2020). The recently developed Expected Gradients method (Erion et al. (020) uses the input (typically training) distribution as the baseline. It computes the attribution for a given input example efficiently by sampling baselines randomly from the training distribution, and interpolation weights from the the [0, 1] range (Monte Carlo integration), rather than using the many equally spaced interpolations of the more common Riemann Sum method.
Dynamic baselines change based on the attribution input. The maximum distance baseline uses the farthest point in the input space from the input vector (Sturmfels et al. (020). This avoids baseline blindness but abandons the concept of neutrality. [Sturmfels et al. (2020) cite Fong & Vedaldi (2017) as the inspiration for blurring as a possible method of feature ablation to produce a baseline. Gaussian noise may be added to the input vector during attribution and when averaged over several iterations, the resulting attributions were found to be more detailed and less noisy (Smilkov et al. 2017). Adding noise to input vectors during training was also found to give a similar and cumulative dynamic distribution
effect. The combination of imput vector with noise can be used by itself baseline (Sturmfels et al. (020) or as an augmentation techmique in combination with another baseline. The justification of this method for use with attribution appears be mainly empirical, and question regarding the theoretical justification has been raised (Sturmfels et al. 2020).
ATTRIBUTION FAIRNESS
Assessment of attribution methods is not currently straightforward because there is not yet a unified general procedure to do so. Rather, there are number of tests for various indicative characteristics (Hooker et al., 2019 Adebayo al., 2018 Yang & Kim 2019). We focus our attention on fairness: whether the relative attributions of input components are reasonable. This is the central concern of Shapley Values, a celebrated method from game theory for allocating value, with strongly justified mathematical foundation (Shapley 1952), and the background to the Integrated Gradients methods (Sundararajan et al. 2017).
Assessing fairness is not trivial. As a reference consider two other attribution methods defined using the original Shapley Values procedure: BShap (Sundararajan & Najmil 2020) and SHAP (undberg & Lee, 2017p. Shapley Values is procedure that, for given set S. requires value for every possible subset of elements: v : P(S) + R, where P is the power set of S, and gives back values for the elements themselves (the "Shapley values"): dev (v, e). The Shapley values sum Eegs Φ.. e), and certain axioms, widely considered as naturally related to fairness, are maintained. As such it gives a way of sharing the value of the complete set fairly among the elements, based on the values that are achieved by each subset.
to the value of the complete set: v(5)= =
Model inputs cannot typically represent missing components, so allocate attribution the compo- nents of an input vector using Shapley Values, presence/absence of a component must be replaced by another binary property. In the case of BShap. 'present' and 'absent' components are represented with values taken from the equivalent components of the input vector or baseline x' respectively. The value function v is simply the model's 's prediction F. SHAP represents subsets using random vectors, and the value function by the expectation of the model's prediction E o F. Each vector has a distribution f f(x|x_=2_v1) where X is the distribution of possible inputs, æ is the input being attributed, and i are the 'present' components of the vector. Further information about Shapley Values and these methods is given is Section A.1 of the supporting material. To summarize the relationships:
Der (v, @PaShapp (F xe, i) @SHAP (SHAA(F,220 2,
sets vectors random vectors
elements components components a a X
S
0
F
x x'
E F
3.1 ATTRIBUTION TRANSFER
Integrated Gradients integrates partial derivatives along a straight path between a baseline and the
input vector. This can be roughly imagined as accumulating the contours 0f(x) of the ouput- landscape that fall across the integration path. To explore fairness in IG we consider the scenario shown in Figure The scenario is classify whether either of two components æ D2 fall within
3

---

Under review as a conference paper at ICLR 2022
22
222
22
0.8
0.6
A
A
0.4
0.2
21
x²
Subfigure
Subfigure 2
0.2 0.4 0.6 0.8
Figure 2: Representation of a model that indicates whether either component of the input 00, falls within the approximate range 0.25 0.5. The background color indicates output value, from white (false, 0) to grey (true, 1). Two IG integration paths exhibiting attribution transfer are shown. Both paths start from baseline x' = (0. Path A indicates an input æ with 22 in the target range and Ø1 outside. However since the integration path crosses only a 'horizontal' gradient (indicated in the dashed circle) IG will incorrectly attribute the result (class 1) to Φ1. There is no possible change in æ1 alone that can account for the increased output of A (Subfigure 1). Conversely æ2 alone can many other places (Subfigure 2). Hence the IG
account for the full increase by changing at x', result could arguably be considered 'maximally misleading' Path indicates an input æ with both components above the target range. As the path crosses perpendicular gradients, both components are attributed with opposite signs.
Table 1: Attribution results associated with Figure 2 IG and BShap have baseline at (0, 0). SHAP values have a different sum because SHAP effectively uses the expectation across the whole input space the baseline. The input distribution is assumed uniform for SHAP. Calculations are given in section A.1.3 of the supporting material.
BShap
SHAP
IG
A
(0. 1)
(-0.11, 0.66)
(1, 0)
B
(0. 0)
(-0.22, -0.22)
(-1, 1)
the range 0.25 -.5, and the shown model gives a continuous approximation of this (transitioning linearly from 0.05 either side of the boundary). The figure shows IG attribution paths for two inputs and B. Attribution results for these inputs are shown Table [
The results for path A could be considered 'maximally misleading' as discussed in Figure 2 The other methods Table are consistent with this. To understand why IG gives this result, we compare the scenario with a case where there is a ground truth attribution. For additive models f(x) = Σ, f((;;) each component's contribution can considered independently o x', x, = fo (x;)) - f(()). Linear models are a subset. In this regime BShap and IG give matching results, as does SHAP if the baselines x' were chosen such that fa(x))) = B(f+) and the input distribution uniform. The model in Figure S deviates from additivity where the target regions of each component overlap. By passing through this area the integration path can enter the target region of æ2 without crossing
any 'horizontal' contours (f()) > 0), yielding an unintuitive result. We name this phenomenon "attribution transfer": the change in attribution caused by the integration path crossing region not representative of the wider landscape. This is a precise concept when the model is mostly additive. For general models, as it becomes harder to say what is "representative" this becomes less precisely defined, although is still useful for discussing the effect of specific perturbations the landscape, for example.
Path attributes an input vector with both components above the target range. IG explains the zero result as the balance of two competing contributions, unlike BShap and Shap which consider the components to contribute equally. Due to sharp features of the input-output landscape near the integration path, the IG attribution will undergo abrupt reversal as the input crosses the œ1 22 boundary, even though neither the input nor baseline are near a decision boundary, nor in the target range. IG attribution results be arbitrarily affected by small sharp regions that fall across the path.
4

---

Under review as a conference paper at ICLR 2022
Is it reasonable to be concerned these effects may occur in practice? We feel it is given that gradients are known to fluctuate sharply (Smilkov et al. (2017). in addition to the results we present.
3.2 AXIOMATIC FOUNDATIONS
Like the other methods, IG is based on Shapley theory and has rigorous axioms. It might then be surprising that IG permits 'maximally misleading" results. IG is based on method in the Shapley Values lineage known as Aumann-Shappe Shapley (Aumann & Shapley, 1974), and has similar but not identical axioms. The names of the axioms of IG and BShap include those of the original Shapley Values: Ffficiency, Linearity, Symmetry, and Dummy (Sundararajan & Najmi, (020). However, IG takes more liberal generalization of the Dummy axiom. For clarity we will give separate names to each variant:
Definition (Weak Dummy). An attribution method yp satisfies Weak Dummy iff
f(z) = f(z + al;) Vz, = pff 2)
This simply means that if the model does not use an input feature, it has zero attribution. Definition 2 (Strong Dummy (BShap)). Let a be a vector with each component as drawn from the corresponding component of either of two given vectors: I& or Z with a; for given Let b except that b, = x/- Then an attribution method (p satisfies Strong Dummy iff f(a) [bb = 0 Va = p(J, x', x. =
== 0
Strong Dummy is identical to Weak Dummy except that only component values from the target and baseline vector considered, not the whole range. The weaker conditions make the axiom stronger. Both IG and BShap are formulated to satisfy Weak Dummy, but BShap additionally possesses Strong Dummy. This precludes it from exhibiting attribution transfer and from producing unintuitive attributions of the kind given by IG in Table [ The situation indicated in Figure 2 Subfigure is prohibited; any positively attributed component must be responsible for an increase output value relative to another location in the input space (proof in Section A.2 of the supporting material). Attribution transfer is not possible because requires sensitivity the region between the baseline and target value. Conversely, Weak Dummy guarantees that a component does not participate in attribution transfer only it has no influence on the model at all. We feel a clear distinction between the axioms in the (original) discrete case and the continuous case may be helpful to avoid any misconception about the strength of the guarantees provided, which are considered extremely robust for Shapley Values in their original context.
4 INTEGRATED CERTAINTY GRADIENTS
We have shown that certain features of the input-output landscape can lead to unintuitive IG attribu- tions. To contrast this behavior in our experiments we introduce a new IG based method, "Integrated Certainty Gradients". which does not traverse the original input space, but rather the space of con- ditional probabilities, in a manner related to SHAP. Sundararajjn & Najmi (2020) note that SHAP ("CES" in their work) is not immediately applicable on finite continuous-valuus distributions (such as a dataset) due to their feature sparsity. Our method resolves this by having the model learn a generalization of the dataset. As Shapley Values method, direct calculation of SHAP is intractable for high dimensional data, and our method is an Aumann-Shapll alternative to SHAP in the same way IG is for BShap.
Our method adds additional information, the certainty, c, to the model input, so the model bases its prediction on the input and the certainty together: (x, The dimension of e is the same as the number of input features in x. and it takes values between [0, 1] that indicate the amount of information the corresponding feature provides. The per-pixel, per-input certainty e is drawn from a random distribution throughout the training of a model, and the model must be trained to associate low certainty values with a concept of missingness suitable for the Shapley theory. This is done using a training process that applies damage function D to the input images, so the model receives damaged imputs: (D(z, c), c). A diagram showing the damage process is given in Figure 3 First, damage data is required, of the same dimension as the undamaged input This take many forms, but for now we will use random noise where each component is i.i.d. uniformly in [o, 1]. Then damage mask is generated by treating each component e; the probability of keeping the
5

---

Under review as a conference paper at ICLR 2022
Interpdiations
Network
Network
Query network for input gradients
2
Ordinary training
Ordinary attribution (e.g. IG)
2
Imput dats
dumaged Imput
Damage
Network
Interpolations
Network
ED nolion
sample
Query network for Input gradients
2
copy
Cersainly
Certantl
ICG training
ICG attribution
Figure 3: Visualisation of ordinary vs. Integrated Certainty Gradients (ICG) training and attribution.
The top row shows ordinary training and attribution using Integrated Gradients which interpolates between a baseline (here: black) and the input example. The bottom row shows our proposed ICG approach. Blue and green correspond to damaged and undamaged components (left side), and certainty 0 and (right side). During training. we sample certainty vector, then sample damage using the probabilities from the certainty vector. The model takes this certainty vector as input and thus learns how to account for probabilistically reliable input features. ICG attribution is then simply the Integrated Gradient from no information to full information about an input example.
corresponding feature æ; If my is not kept is replaced the corresponding value from the damage data.
method is needed to generate c during training. Our method works reasonably well each component (and thus the keep probability of each input feature) is i.i.d. uniformly in [0, 1]. Regular (undamaged) inputs to the model correspond to using e; = 1 i. The features D(x, c); of an input prepared in this way have varying chances, equal to their certainties c,, to provide information about the original input æ, and therefore the class g. During training the model learns to ignore features unrelated to y. discounting features with lower certainty, and develops the required uncertainty semantics.
If the model is trained well, it will ignore features with zero certainty, because these give information about the original input:
OF (xx
c)
Rs
0
dz:
c;=0
In this case, 0 provides natural baseline to represent missing data, as the model will ignore the input values entirely.
We complete the Integrated Certainty Gradients attribution method by combining this baseline with IG. The integral of Equation is taken from e = to e 1, holding ær constant at the input we wish to attribute. Note that æ is not damaged while calculating the attribution. It is convenient note when implementing that the distance term from Equation disappears because c, - c =
Deep networks are capable of detecting and ignoring noise, so the damage data described above may not guarantee learning a correct interpretation certainties. To avoid this, samples from the input dataset can be used as damage data, ensuring certainties must be used to distinguish original and damaged features. During ICG the model will encounter interpolations with total mean certainty from O to 1. Most inputs generated with certainty sampled I.I.D. from U(O, 1) will have certainty

---

Under review as a conference paper at ICLR 2022
A
B
C
(logit)
E
F
G
H (logit)
I
Input
Zero
Baseline
Mean
Baseline
AGN
EG
ICG
Figure 4: Attribution results for the Burnt Snacks scenario. Each column shows an input image (top row) with associated attributions (other rows). Column A-D inputs are randomly chosen from class 1 (dark square present). Column E-G inputs are randomly chosen from class (no dark square). Probabilities are attributed except columns D) and H where logits are attributed. The target objects (dark squares) are marked with a green dot (not present in actual input vector). Expected Gradients is evaluated with 500 interpolations. Additive Gaussian Noise is evaluated with 50 baselines with 100 interpolations each. The other methods are evaluated with 100 interpolations.
close to 0.5. To ensure the model is trained on a wider range of certainty inputs we developed parameterized distribution based on the Continuous Bernoulli Loaiza-Ganen & Cunningham) (019) where the mean can be varied betwen 0 and 1. Full information about the training methods used is given Scction A.3.2 the supporting material.
EXPERIMENTS
5.1 BURNT SNACKS SCENARIO
We use a purpose designed scenario to demonstrate attribution transfer of methods including Expected Gradients. The scenario is notionally similar to Optical Sorting task in which overcooked food items are identified (Pedreschi et al., 2016). The model must determine whether 'overcooked" (dark) squares (25% 50% lightness)
present (class 1), or not (class 0), amidst light squares and other shapes on a dark background. The input-output landscape for an input with two squares with lightnesses æ1. 22 is similar Figure [] Two characteristies are designed Gradients. First, the average pixel value is very dark. This will tend to darken the features of an attributed image when is interpolated. Second, the light squares are lighter versions of the target objects (dark squares), but are not relevant to the prediction outcome. This combination will cause light squares to often present as dark squares interpolations, despite having no correlation with them, confusing Expected Gradients. It is easy to justify that light squares should not be attributed, because their presence does not influence the class (relative to black background). A selection of results on this scenario are shown in Figure 4 We trained certainty aware model to 98.7% accuracy (for undamaged inputs). We attribute probabilities, in line with a prominent guide (Google 2021 and also logits.
mislead Expected
7

---

Under review as a conference paper at ICLR 2022
Zero and mean baseline Although zero baseline seems appropriate, both zero and mean baselines markedly show attribution transfer. Example G for the zero baseline case is equivalent to path B of Figure There are two light squares, one with negative and one positive attribution. By checking the interpolation gradients, we saw the positive attributed square enter the target region (at 25% lightness) first, picking up attribution. The second square followed it but gathered attribution due to saturation of the output. On leaving (at 50% lightness) the situation was reversed. Attribution transfer also occurs in class I scenarios where a light square is present, for example Zero Baseline example C. This situation is equivalent to path A of Figure 2 only the light square, irrelevant to the prediction, attributed.
As a result, the process concludes with
Expected Gradients Results for Expected Gradients are shown in the 5th column of Figure 4 In no case does EG identify the dark squares when they are present when using probabilities (class 1, images A-C). Instead, it transfers attribution to the light squares, most strongly the lightest. In case appears no square is light enough to attract attribution, and artifacts from the random sampling process are shown.
Since class O is an absence class the expected outcome is not as clear. Images E and show diffuse artifacts from the interpolation process and arguably this is a fair result. However image G strongly attributes a specific square and this seems more clearly misleading, as the square has no significance to the classification outcome.
Integrated Certainty Gradients ICG does not suffer from the previously described problems. Because the integration is across the uncertainty space, the relative location of features in the input space and the input distribution are not important. For class O, ICG attributions are very diffuse and largely uniform, which could be considered an appropriate outcome, arguably the most intuitive of any method. For class 1, it always attributes a dark square (correct target feature). The attributions are strong and precisely located. However, only one square is attributed even when multiple are present. By inspecting the interpolations and using variations of the image with each square removed, we found that the model's 'ss confidence in each square transitions abruptly during the certainty increase. Because of this, the first square to gain the model's confidence causes the prediction to saturate, preventing it accruing to other squares. This is attribution transfer in the certainty space. This shows that though irrelevant objects are no longer brought into in the attribution, there can still be transfer between independent relevant objects. We leave further consideration of this aspect to future research.
Additive Gaussian Noise Additive Gaussian Noise correctly attributes the location of the dark squares in class 1 images. The attributions are noisy and not as precisely localised to the squares as those of ICG, however, all squares are attributed. In class 0 attribution focuses around the triangles. This is unsurprising because the addition of noise could easily make them resemble squares.
Logits For most methods logits show intermixed opposite attributions on both light and dark squares. Dark squares have overall more positive attribution, but nonetheless this presen- tation is imperfect and potentially confusing. ICG performs excellently in this scenario, no longer missing squares, attribut- ing all dark squares clearly.
0.5
Score
0
-0.5
5.2 LOCAL MINIMUM SCENARIO
0.25 0.5 0.75 1
Additive Gaussian Noise uses baselines drawn mainly from a region local the input vector. We introduce a scenario de- signed to challenge local baselines. Input vectors are classified according to total score S(z), which is the sum of component- wise scores f(x)). Inputs are in class 1 if S(z) > 0, class O otherwise. Note that although S is an additive function, the classifier as whole is not. The component-we -iipe score function is shown in 5 f has a positive local minimum. This local minimum makes positive contribution to class 1 even though it is less contributing than nearby values. Attribution methods based on local baselines would be expected to give the local minimum an incorrect negative attribution. We trained a certainty-awar network on this classification scenario to 93%
Component value
Figure 5: Component-wise score function f for the Local Minimum scenario. The total score is the sum of the scores of the covariates S(x) = Σ; f(x)). Component values range from 0 1, The score is -0.75 for component values 0.5 and below, and between 0.5 and for values above 0.5.
8

---

Under review as a conference paper at ICLR 2022
Table Comparison attribution methods on the Local Minimum scenario. Each row corresponds to component of the input vector. The first column shows the input vector. The second column shows the nominal "scores" the components contributing the true class, normalised maximum value 1, for comparison with attribution results. The remaining columns show attribution results, normalised maximum value I, from BShap, zero baseline Integrated Gradients, mean baseline Inte- grated Gradients, Expected Gradients, Additive Gaussian Noise and Integrated Certainty Gradients. Attributions are shown for class 1, not the predicted class (0), to aid comparison with the score values.
Input vector
Scaled score
BShap
Zero IG
Mean IG
EG
AGN
ICG
0.1
-0.94
-0.75
0.001
-0.01
-0.30
0.004
-0.75
0.2
-0.94
-0.74
0.002
-0.001
-0.73
-0.004
-0.75
0.3
-0.94
-0.74
0.003
-0.37
-0.66
0.004
-0.74
0.6
0.82
0.7
0.75
0.50
-0.24
0.89
0.70
-0.42
0.48
0.9
0.76
0.23
-0.07
-0.02
0.74
accuracy. Attribution results for this network using example input vector from class 0 are shown in T Component values of 0.50546 were used for the BShap baseline, as this was determined to be close the model's prediction boundary.
BShap, ICG and EG performed well the scenario, with ICG and BShap in excellent agreement. Only IG and BShap accurately represent the uniform negative region. All other methods performed badly. As expected, AGN incorrectly gives the local minimum negative attribution. Zero IG shows strong signs of attribution transfer. Zero IG and AGN both exhibit blindness in the negative score region due to sharing this negligible gradient region with the baselines. The results for mean IG vary widely. The shown result is a randomly chosen example. The distribution mean lies on the discontinuity of the score function (0.5) so small variations around it make major differences to the result.
CONCLUSION
We have described a situation in which Integrated Gradients produces extremely unintuitive results, and explained how this arises from areas where the prediction function is non-additive ("attribution transfer"). This behavior may be unexpected because IG founded in the theory of Shapley Values, an axiomatic framework for fair allocation. We explain this seeming contradiction by identifying difference in the formulation of an axiom derived from the original theory.
In experiments we have shown that attribution transfer occurs even for state of the art methods like Expected Gradients the presence of a highly skewed but plausible data distribution. Avoiding detrimental interactions by restricting IG to small area of the input domain, as Additive Gaussian Noise does, does not prevent mislcading results when the local prediction landscape does not reflect the wider input space.
We show these failures of IG may be avoidable, by developing a method with a new kind baseline, Integrated Certainty Gradients, that requires special training process to learn the data distribution. This method is the Aumann-Sii Shaully alternative to SHAP (as IG for BShap) and by not traversing the input space avoids the failure
demonstrate in other methods.
ACKNOWLEDGMENTS
LEFT BLANK IN ANONYMOUS VERSION

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Attribution is of key importance in many scenarios with ethical implications, for example providing reasons for life affecting decisions such as loan assessments, determining whether decisions are made based on legitimate protected personal characteristics. As such, the development of fair attribution methods is ethically important and this paper has been produced with appreciation of that. The work itself undertaken to produce this paper did not raise any ethical concerns.
REPRODUCABILITY STATEMENT
Model architectures and training hyperparameters are given in Appendix A.5 Anonymized source code is available at https//gtthhl.cmmmaccomtt99////
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL https::////.mm. htttps::/proceeeiiiii 294a8ed2 4b htw.ee/eeea222222eeeee000001111.
htt.o/eeeddnnn....rieeeeeee newrips...... pctpp/cccc/aaaee/1////eeee
pdf
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps, 2020.
R. J. Aumann and L. Shapley, Part I: The Axiomatic Approach, pp. 11-92. Princeton University 99..5
Press, 1974. httt:tt///// /www. jstor. http:////www..stoooooo///o/eeee
Umang Bhatt, Alice Xiang. Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. Explainable machine learning in deployment. Proceedings of the 2020 Conference on Fairness, Accountability, and Trunsparency, FAT '20, pp. 648-657, New York, NY, USA, 2020. Association for Computing Machinery, doi: 10. 41144//335519533356244 Utttttts:///ai. http://////o...0/11111111..
ISBN 9781450369367. 01510955.33 3375624
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining removing: A unified framework for model explanation, 2020.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. Technical report, University of Montreal, 2009.
Gabriel Erion, Joseph D. Janizek, Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients, 2020.
fast.ai. Imagenette.
2021.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful per- turbation. 2017 IEEE International Conference on Computer Vision (ICCV), October 2017. doi: 10.1 10.11099/ccv..11..331.
CRRLattp://// doi www. wrg.ii.rg//000 1109/ ICCV 20177.311
Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision-making and a "right explanation", AI Magazine, 3803) October 2017. ISSN 0738-4602. doi: 10. (tt6////iima..3333.2222.. http:////////////.....././//.//.
Google. Integrated gradients, 2021. /ttp:t/ps///.....ooooroooooooooa Liteeppeetabliiiiiiieeeess. wrw.ggrragnaaae.
Johannes Haug, Stefan Züm, Peter El-Jiz, and Gjergji Kasneci, On baselines for local feature attributions, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
10

---

Under review as a conference paper at ICLR 2022
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods deep neural networks. H. Wallach, H. Larochelle, A. Beygekzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL fi weeeeeee55500000000011100 Dcae htt.:oacaaaaaaaaacaaaaa1---e-e
neur ps.ppp... wtwpiis..cccppppp cctppppr//211//////////// neur
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998. doi: 10.1109/..26691.
Gabriel Loaiza-Ganem and John P. Cunningham. The continuous bernoulli: fixing pervasive emror in variational autoencoders, 2019.
Scott M Lundberg and Su-In Lee. unified approach to interpreting model predictions. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30, pp. ciates, Inc., 2017. URL htttp:://prcceedinngg neurips. cctpappr//011/EEE.... pdd
Curran Asso-
8 02 621 07c.227777777777646c666222666663
Pedreschi, Mery, and Marique. Chapter 22 - quality evaluation and control of potato chips. In Da-Wen Sun (ed.), Computer Vision Technology for Food Quality Evaluation (Second
Edition), pp. 591-613, Academic Press, San Diego, second edition edition, 2016. ISBN 978-0- 12-802232-0. doi: http:///wwcoooooooo/ooooo/////20
URL http:: www
sciencedirect. wtt.c/cc/cccc.eecceccet ott.apccc....aaaoo.......aoaaa//
Lloyd S. Shapley. A Value for N-Person Games. RAND Corporation, Santa Monica, CA, 1952. doi: 10.7249/P0295.
K Simonyan, Vedaldi, and A Zisserman. Deep inside convolutional networks: visualising image classification models and saliency maps. pp. 1-8. ICLR, 2014.
Amitojdeep Singh, Sourya Sengupta, and Vasudevan Lakshminarayanan Explainable deep learning models medical image analysis. Journal of Imaging, 6(6), 2020. ISSN 2313-433X. doi: 10.3390jjmaging6660022 htl.htts:: /ttp:///wwwwwoo www.:.dpp....m com wtt.l.cco.mc...... 2313-433x/6/6752
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise, 2017.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020. doi: 10.2391 00022..
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9269 9278. PMLR, July / hta/tttp.//rroceeingg...tt ml. press/ppesss/eeo wwtpppppwssss.eseseeeee http11////unarrrra jan2 html
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients counterfactuals, 2016.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. CoRR. abs/1703.01365 2017. URL http::/arxiv... www.aawwww...i.aaaaooooooooooort ottg/bab//73....3333 01365
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance, 2019.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015.
11

---

Under review as a conference paper at ICLR 2022
22
32
32
I1
31
A: BShap
B: Integrated Gradients
C: SHAP
Figure 6: Input regions considered by attribution methods. Regions are shown with nonzero area for clarity, but in reality all regions except the whole-space region of SHAP are zero width points lines of the input space. The input distribution is assumed uniform for SHAP. The locations of the target value æ and baseline x' are arbitrary.
A
APPENDIX
A.1 ADDITIONAL THEORY AND CALCULATIONS
A.1.1 SHAPLEY VALUES
Integrated Gradients is method axiomatically founded in the theory of Shapley Values (Sundararajan al. 2017). Shapley Values is way to share responsibility between contributing factors when their combined outcome (expressed numerically) need not be equal to the sum of the outcomes each factor would give their own. As such it is way of sharing the benefit (or disadvantage) of cooperation. To do this it requires counterfactual knowledge of the outcomes that would be achieved by each subset of the factors alone. Formally: given a S, and value function P(S) R, with P(S) the power set of S, there is Shapley value d (v, e) for each element e E S. Its significance is that the unique method that, given this information, allocates responsibility according to the following axioms, intuitively associated with faimess:
Efficiency Also called Completeness. The Shapley values allocated to each of the factors sum the value of the combined outcome. v(S) = Σ &w (v,e)
EES
Symmetry Factors which have the same effect have the same Shapley value. Formally: if pair of factors e1, e2 e S have v(S' Uei) v(S' U e2) for every S' not containing ej or eg, then (u, e1) = (u,
Null player Also called Dummy (Strong Dummy). If v(S) = ((e)) for all S' then the Shapley value of is 0.
Linearity The Shapley Values method is a linear mapping of the value function: es)(U1,e)
ésv(Uz e) = @w (11 + U2 e) and (av, e) = (v,
To calculate the Shapley values we define the marginal contribution factor to a S S. & S' as v(S" U (e)) e(S"). Then we consider every possible ordering of the factors in S. The Shapley value of the mean of the marginal contributions it makes to the subset of elements that precede it in each ordering.
A.1.2 BSHAP, SHAP, AND INTEGRATED GRADIENTS
Shapley Values assigns attribution to factors based on the differences in outcomes when they are absent. To represent missing components BShap uses the components of baseline vector. The normal Shapley Values procedure is then followed, representing subsets of components by vectors. These vectors take values from the target vector æ for components in the subset, and from the baseline x' for the other components. SHAP does not use a baseline. Instead it takes the expectation of the the present components. In this paper
model output over the missing components, conditioned we focus only on the model and so assume the input distribution uniform when considering SHAP. Therefore each subset in the Shapley Values calculation is represented by integral across the
12

---

Under review as a conference paper at ICLR 2022
possible values of the absent components, with the other components fixed to their values in the target vector æ. IG does not follow the Shapley Values procedure but rather the method described in Section based on the Aumann- Aumann-Shapley Shapley method for sharing attribution. Figure shows the regions of the input space used by each method.
A.1.3 CALCULATIONS
The calculations for Table follow. These follow the methods explained in the earlier parts of Section A.1 The model outputs are F 1, F(xB) 0, and F(x') O.
BShap With two features, there are two possible orderings for the Shapley calculations: 21, 22 and 22, 21 Starting from the baseline x'' and swapping in components from the input x in these orders
gives:
(x', x²/) -i (01, xt²2) m², (x1, x22) (x'). xz²) _ (at) x22) 25 (x1, 22)
For B the value at all of these locations is zero the attribution is zero for both components. Evaluating these for A:
F(O, 0) 21, F(11, 0) = -0- 22, F( F2) =
F(O, 0) 0 22, F(0, = m1 F F2) =
Which give attributions for æ1: _ 1 +
0 and æ2: 1 0+ --
IG It is not necessary to perform an explicit calculation for IG. For A the path does not encounter 'vertical' gradients so the 8F(x) term in Equation 1 must be zero for i=2. Hence the second component de;
has attribution O. The attributions of all features must sum to the difference between the target input and baseline, giving the result (1, 0). A similar argument can be developed for B by breaking the path into two parts and considering the traversal of the 'horizontal' and 'vertical' contours separately.
There again two possible orderings, this time of conditional expectations. Conditioned on both components (F = x) is just F(x). Unconditioned Ex (F())) x 0.434 is the mean value over the whole input. Conditioned on only one component, the expectation integral is if is along one of the 'arms' of the cross, and 0.25 if it is across one of the arms. This gives attributions:
SHAP
A: ::: (1 - 1 0.25 0.474) -0.11.
#2 (1 0.25 + 0.434) Rs 0.66.
B:
31: (0 0.25 + 0.434) ~ 0.22.
B2: (0 0.25 0.25 0.434) s -0.22.
A.2 PROOF THAT STRONG DUMMY IMPLIES CONSISTENCY
We define attribution method &p be "Consistent" if given model F, input and feature an attribution (F, 1) > implies there exists an a with a; = æ.; such that F(a) > F(a + Ax;) for some Xæ,, a vector zero except for component That is to say, if a component attributed positively, there must be at least one instance where changing that component of a vector to match the input vector results in increase in value. This section demonstrates that adding Strong Dummy the axioms Completeness, Symmetry, and Linearity guarantees Consistency.
BShap follows all of the four axioms. We will show is the unique such method. BShap takes an input x and baseline x'. It is evaluated using the Shapley Values procedure on all vectors a having a, = a or a; = 2; for each Let A be a set of vectors so constructed. For a given A the axioms Completeness, Symmetry, Linearity, and Strong Dummy of an attribution method on the continuous space are equivalent to the Shapley Values axioms on the a space: Completeness requires the attributions sum to the total difference in output value between the baseline and target. This transfers directly to the Shapley Values context by subtracting the baseline value from every F(a). Symmetry requires that the attribution method be invariant under permutations of the element/component labels; this is the same in both settings. In the continuous space the model functions F are all the functions from input vectors the possible attribution vectors. The functions relevant the Shapley Values
13

---

Under review as a conference paper at ICLR 2022
perspective are all those from A, subset of the input space, to the same attribution vectors as before. Hence the functions on AA are a linear subspace of the original function space, and so the linearity of an attribution method on the original space will apply to them also. Last, the definition of Strong Dummy was based on the Shapley Values Dummy axiom.
This demonstrates for any x' and æ. ie. everywhere, the four axioms equivalent to the Shapley Values axioms. Given that Shapley Values is known to be the unique solution, BShap must be unique also. Shapley Values BShap consistent (if an element has positive value it must make least one positive marginal contribution, a stronger guarantee) therefore Strong Dummy in combination with the other 3 axioms guarantees consistency. This discussion highlights the difference between Weak Dummy and the other axioms. Most of the axioms were strengthened when generalizing to the continuous setting, whereas Weak Dummy was weakened.
A.3 CERTAINTY TRAINING
A.3.1
VALIDATING TRAINING
For effective training, methods to assess correct interpretation of certainty are required. We introduce two techniques for this.
Information absence To ensure fair attributions, the model should interpret all baselines in the same way, formalized by equation 2 This may be verified by varying æ with c and confirming predictions do not change.
Integrated Gradients gives zero attribution when components match the baseline (baseline blindness). However, Integrated Certainty Gradients relies on the model itself also ignoring zero certainty pixels, so that model and attribution behavior are aligned. This means that zero certainty pixels should not accrue attribution by any method, not just ICG. This makes
0 in sense universal baseline. This motivates a distinctive visual test. If the model is well trained, it should be possible to suppress the attribution of accurate method to negligible amount by masking out that area with zero certainty during the attribution process. This can be tested by using IG with baselines z' = 0 and x' = 1: the result should show obvious suppression of attribution in the uncertain region in both cases. This test is visualized in Figure 7
Neutrality Zero certainty baselines should provide no input information to the model, so its output should be unbiased. It is therefore reasonable to expect the model prediction in the baseline space to match its expectation on the training set: F(x, e = 0) Rs ((F(Xminn) xc.
The expectation of each class in the training distribution is the proportion of samples in that class. So for example in onc-hot representation, ÿ should be approximately the proportion of the training distribution occupied by class i.
Using these tests we assessed a number of methods for certainty damage. The following methods improved training compared to using uniform distributions.
A.3.2 DAMAGE FUNCTIONS
The basic damage function described in section can produce informative attributions but more effective damage functions have been identified.
Variable damage extent With uniform random certainty, the trained model often widely violates the neutrality condition. Due to regression towards the mean, i.i.d. uniform random certainty results in an image with mean certainty 1 Σ; e; typically close to 0.5. It hypothesized that the the neutrality condition is not learned due to the lack of training inputs with mean certainty near zero. We define damage extent as the mean uncertainty of an input: 1 1 1 Σ, C₁ A damage function which results in damage extents distributed uniformly in [0, 1] will expose the model to inputs along more of the certainty interpolation space.
Sampling from the continuous Bernoulli distribution (Loaiza-Ganda & Cunningham, 2019) was chosen as a new certainty generating function because it produces samples in the range [0, I] and has parameter which allows the expectation to be varied in [0, 1] as well as other desirable properties.
14

---

Under review as a conference paper at ICLR 2022
Figure 7: Example of the In- formation Absence test, An area of the image is masked out to zero certainty during at- tribution (shown in grey). In- tegrated Gradients are taken from uniform zero and uni- form baselines. Even though these are not certainty- aware attribution methods, they show the same exclusion of attribution from zero cer- tainty areas as ICG, suggest- ing that zero certainty is "universal" baseline. Image E shows the attributions for the zero baseline and one baseline added together.
A: Input image
Zero bascline
C: One bascline
ICG attribution
E: Combined attribution
Zero attribution
G: One BL attribution
The parameter does not vary the expectation uniformly but instead concentrates towards the center of the range. To counteract this reparameterized the distribution. Let e(A) be the expectation of the distribution with parameter A. Then X' := e (A), where e is calculated numerically from the analytical form of Using this parameter gives distribution parameterized by the mean. It was found that varying damage extent using this distribution does in fact improve neutrality.
Adversarial damage Random noise is typically not strongly biased towards an input class. Uniform random damage data therefore results in input which is combination of data from the original class and unbiased data, As a result, the input is always positively biased towards the original class. Furthermore, if the model can detect and exclude random noise, it will be able to determine the true class from the remaining data. These considerations suggest the model might be able to ignore e and still achieve good predictions.
Motivated by this, adversarial damage was tested: using a random alternative sample from the test dataset as the damage data. When the alternative and original input are from different classes, damaged features D(x, c); provide evidence for different class from the true one and with large damage extent the majority of features can indicate the alternative class. It is not possible for the model to make an accurate prediction in such circumstances without considering c, forcing the model use it. Adversarial damage was found to train certainty awareness faster and with more accuracy than random uniform damage. Another consideration is that it avoids exposing the model to out of distribution feature values, although the implications this were not investigated.
Binary uncertainty Continuous Bernoulli and uniform damage distributions have been described above. Discrete Bernoulli damage was also tested, producing c; values in {0, 1}. It was found that this improves learning rate compared to uniform random damage. The reason for this has not been determined, although it may be since can be only 0 or 1, the value components no longer require probabilistic interpretation, which makes the scenario easier for the model to learn.
We found combining adversarial and uniform damage data, binary and uniform uncertainty, and using variable damage extents resulted in the most effective training in terms of speed and final evaluation. This training was used for all results which are discussed below.
A.3.3 LEARNING RATE AND ACCURACY
Certainty awareness increases the classification task complexity. To assess the impact of this trained MNIST models with and without input damage. The results are shown Figure g The red and blue curves show training with and without damage respectively. Both evaluated on undamaged inputs. They show that damage training has not resulted reduced accuracy on MNIST for this model, although training takes twice many batches. Variance in accuracy and loss is
15

---

Under review as a conference paper at ICLR 2022
25
3.8
Loo3
2
02
103 1003
Min basches
A: Training loss
Training accuracy
Figure 8: Training progress with and without damaged inputs. Blue curves show learning progress without damage. Red curves show learning progress with damage evaluated on undamaged inputs. Green curves show learning progress with damage evaluated damaged inputs. The evaluation images were drawn from the test dataset. Progress is measured in batches (of size 64) because multiple damaged inputs were created from each original input, leading differently sized epochs. Evaluations on damaged inputs are significantly less accurate, which is to be expected given that when the damage extent high they may contain little of the original image. Evaluations of the damage trained model on undamaged inputs reach similar accuracy to the model trained without damage. Their curves show roughly constant horizontal offset. On these logarithmic scales this indicates approximately constant slowdown ratio: training with damage takes roughly twice as long.
also increased. Given that on average half the components of the original image are lost during damage, this seems modest penalty. The green curve shows results for evaluation of the damage trained model on damaged inputs. This is included for completeness but not especially relevant to the comparison.
A.4 ADDITIONAL RESULTS
A.4.1 IMAGENETTE
To test ICG on more realistic scenarios, we trained a ResNet50 (He et al. 2015) using artificial uncertainty training the Imagenette dataset (fast.ai 202T). The only modifications made to the architecture were the addition of an extra channel to represent certainty, and an additional 1x1 (64 channel) convolution layer at the input, with ReLu activation and without batch normalization. The model was trained to 82.4% top-1 accuracy on the validation data.
Attributions using ICG were compared with those generated using EG, chosen because it a contem- porary baseline-free attribution method. It is established that single-path attribution methods such as ICG can produce clearer results when averaged over multiple slightly perturbed inputs (Smilkov et al. 2017). For comparison with EG, multi-path method, we averaged 5 ICG attributions over inputs with added low-intensity noise to make each final result. We tested including noise augmentation when generating FG interpolations but it did not significantly affect the outcome.
Results can be seen in Figure In most cases ICG attributions have a clear association with the object of interest. The attributions tend to be more tightly localized than those of EG. Backgrounds can form part of a model's classification decision, so stronger localization is not a full guarantee of greater attribution faithfulness. However, it is notable that in row 2 BG allocates significant attribution in the top black "border" region. seems reasonable to assume this is one of the lowest information-denssiee areas of the image, and not representative region for attribution. We have not ICG significantly attributes the border regions. Note that this is not baseline blindness: black/dark areas in images can be strongly attributed through ICG.
where
16

---

Under review as a conference paper at ICLR 2022
Image
ICG
ICG blurred
EG
EG blurred
Figure 9: Attribution results for Integrated Certainty Gradients and Expected Gradients applied to images from the Imagenette dataset. Images were randomly chosen discarding any incorrect classifications. Because methods in the Integrated Gradients family tend to produce "noisy"/"speecll"" results, a Gaussian blur has been applied to each method to make interpretation easier. The results for ICG are notably more tightly localized in most cases, An exception is the chainsaw (row 2) where ICG strongly highlights background regions. In all other cases ICG clearly indicates the object of interest, except for the golf ball (row 6), where ICG focuses attribution primarily on the golf club.
We hypothesize the reason for the difference in behavior between EG and ICG. We have noticed output artifacts which relate to objects individual EG interpolation baseline images. If EG were performed with enough interpolations to truly represent the input distribution it may be that the effects of interpolation baselines would exactly 'average out' low attribution areas. may be that this cannot be achieved in larger images due the high dimensional input space. This would also explain the excellent performance EG on the MNIST dataset (where the input distribution is rather densely represented). It would be reasonable to imagine that ICG might overcome this problem by approximating the input space as part of the deep learning model, given that deep learning methods current sfate-of-the technology for representing high dimensional data.
Whether or our hypothesis is correct, the Imagenette attributions give evidence to ICG as an effective baseline-fre free attribution method in practical scenarios.
17

---

Under review as a conference paper at ICLR 2022
A: Input image
B: Zero baseline
C: Middle baseline
D: AGN
5
Distribution mean
ICG
G: Mean baseline
H: EG
Figure 10: Attribution results for example ("5") from the MNIST dataset (Lecun al. 1998).
A Input image. E Mcan of the dataset. D Integrated Gradients with basclines uniform 0, uniform 0.5, and Additive Gaussian Noise. F Integrated Certainty Gradients. E Integrated Gradients with dataset mean baseline. H Expected Gradients. Positive values (orange) show supporting evidence. Negative values (blue) show opposing evidence.
A.4.2 MNIST IN SPACE
Figure I0 compares attribution results for several baselines on random example from the MNIST training test discussed in section A.3.3 The zero baseline attribution has strong baseline blindness: only the 'digit itself" is attributed, not the space around It shares several intuitively reasonable features with the other results. The crossbar at the top is positive, which is characteristic feature of a "5" "7" is the only other numeral having one, and it terminates on the opposite side. The bottom curve is also indicated, a feature shared only with "3". The slope in the middle is more similar 6 or an 8 (attributions for classes 6 and 8 attributions support this), and hence attributed negatively (in blue).
Of the remaining results there is a notable distinction between distribution-aww -waree methods (mean baseline and EG) and unaware (middle baseline and AGN). For example in the unaware methods there a negative attribution point where the crossbar terminates, indicating a preference for a longer crossbar, but as can be seen from the distribution mean image, so it makes sense the distribution aware methods would not indicate ICG shares many characteristics with the distribution-unavee most strongly attributed area, in the crossbar, matches to the aware methods. It is plausible it acts as an intermediate between the distribution aware and unaware methods: learning a distribution of the data during certainty training, but incorporating the model's generalization prior it does so.
the aware methods. This point is outside the region where the distribution varies,
methods, but the
To explore this idea briefly we introduce modified version of MNIST. We make the input images wider and introduce a wranslationiinvriin model architecture. We can train a model using regular MNIST images in the left region, and the model able to classify digits correctly anywhere in the extended image due to the architecture. When classifying digits in the right region they will be out-of- out-off-raining distributio.... Results from this scenario are shown in Figure II The Expected Gradients result appears more sharply defined because of baseline blindness (the right side of all training images is black). A faint 'shadow' of the training distribution is visible in the EG result. This does represent a model behavior since the model is translation invariant. EG is formulated as a fair baseline for a given distribution, but it does not follow the model's 's generalization onto OOD inputs. Because the distribution knowledge ICG uses learned by the model, is generalized by the model architecture and follows the model generalization onto new inputs. Like distribution unaware methods there no 'shadow', because knowledge of the ungeneralized distribution not retained.
A.5 EXPERIMENTAL DETAILS
Model architectures are shown in Figures 14
12 and 13
18

---

Under review as a conference paper at ICLR 2022
7
A: Training distribution mean
B: Input
C: AGN
ICG
EG
Figure 11: MNIST in Space scenario. A: mean of the training inputs; B: target input for attribution; C: Additive Gaussian Noise attribution; D: Integrated Certainty Gradients attribution; E: Expected Gradients attribution. The main attribution of EG significantly more localized due baseline blindness. A "ghost' of the training distribution is present in EG but not in AGN or ICG.
The damage function D used during training is specified by combination of damage data and certainty generator. For all models damage data was taken from either U (0, 1) adversarially from another training input. Within each input, C; values were taken i.i from either the Bernoulli or our reparameterization of the continuous Bernoulli distribution. Both have parameter equal to their expectation, and for each image this was sampled from (0, 1). The four combinations of damage data selection method and certainty generation method were each used in equal proportion.
Xavier initialization was used for network weights and network biases were initialized to zero.
Burnt snacks Training data was generated procedurally hence there is epoch size train-test split. Adam optimization was used with learning rate 0.001 and batch size of 64.
Local minimum Training data was generated procedurally hence there is no epoch size or train-test split. The network did not converge for several training attempts, and was reinitialized several times until an initial state which did converge was found. The network was initially trained with Adam optimization and learning rate of 0.001. After initial training, reduction in the learning to improve accuracy, and training was continued with reduced learning rates down to a minimum of 10-6, A batch size of 128 was used for initial training and batch size of 256 for the reduced learning rate stage.
found
MNIST
The dataset was split into 60,000 training examples and 10,000 test examples across all classes. Adam optimization was used for training with learning rate 0.001 and batch size of 64.
19

---

Under review as a conference paper at ICLR 2022
Emput
[/oos, 20,
iupru Tapuilaye
[/Non, StL, 231
concM Com20
Laguo:
(Nooss 30, 50,,
(Nors, 481, 32)
uomdll In Cuneon
aipet:
nutpa:
(Nincc 46, 46, 64)
mus_poolingii ManPoolingDD
sulpert
(Suare, 23, 23, 64)
Ingut
[None, 23, 23, 54
dripenat: Draport
oupqut
(Nome, 23, 23,
cumv22 2: Com
siper:
Liput:
(Nome, 21, 125)
201o pallingh ZeoPukking20
cumpor
(Nome, 22, 22, 1253
ieper
178)
mux.poolnggdd MasPooling21
ourput:
None,
cu Con
(None,
inve
(Noow, 9. 178)
20o paddinngd ZenuPaddligg
ourpa:
(Nene, 10, R0, 125)
impux:
128)
mucpoolingAA MusPonning22
output:
(Nrma, 5,5, 128)
deepoot I Dropeer
Clone,
inpert
(Nons, 5, 5, 125)
(Nens, 1,30))
(himss
Jane: Droee
ralpul:
(Nire,
dropoce Drppout
input:
(Nons, 32)
outger:
(Nons, In)
kspua:
doyoor 3: Dropout
silpl.
(None,
denue Dense
(Niuee 2)
Figure 12: Model architecture used for Burnt Snacks scenario comparison.
20

---

Under review as a conference paper at ICLR 2022
Lonse_input:: Layer
imput:
[(None. 6, 2)]
Continued from bottom
oulput:
[(None, 6, 2)]
inpul:
(None, 500)
dense_6 6: Dense
imput:
(None, 2)
output:
(None, 6, I)
dense: Dense
oulput:
(None, 6, 500)
input:
(None, 6, 1)
leaky_re_ied _e_______6 LeakyReLU
input:
(None, 6, 500)
output:
(None, 6, 1)
ReLU leaky_re, lu: LeakyReLLU
outpul:
(None, 6, 500)
input:
(None, 1)
flatten: Flatten
input:
(None, 500)
output:
(None. 6)
dense_1 Dense
output:
(None, 500)
input:
(None, 6)
lambda: Lambda
input:
(None, 500)
oulput:
(None, I)
leaky_re_hu.
[__l: LeakyReLLU
output:
(None, 500)
input:
(None, 1)
dense Dense
imput:
(None, 500)
output:
(None, 20)
dense_ 2: Dense
output:
(None, 500)
imput:
(None, 20)
leaky_re ee_ lu_ LeakyReLU
input:
(Nonc, 500)
output:
(None, 20)
leaky_re__ re_lu_2 Lea
output:
(None, 500)
input:
(None, 20)
dense Dense
input:
(None, 500)
output:
(None, 20)
dense_3: Dense
output:
(None, 500)
imput:
(None, 20)
leaky re Ju__ LeakyReLU
input:
(Nonc, 500)
output:
(None, 20)
leaky_re_lu_3 LeakyReLU
output:
(None, 500)
input:
(None, 20)
dense 9: Dense
input:
(None, 500)
output:
(None, 20)
dense_4: Dense
oulput:
(None, 500)
input:
(None, 20)
leaky_re_lu lu_9 LeakyReLU
input:
(None, 6, 500)
output:
(None, 20)
leaky_re bolly_re_lu lu_4 Leu
output:
(None, 500)
input:
(None, 20)
dense Dense
input:
(None, 6, 500)
output:
(None, 20)
dense_ 5: Dense
output:
(None, 500)
input:
(None, 20)
leaky_re re re_______. 10: LeakyReLU
input:
(None, 6, 500)
output:
(None, 20)
leaky_re__
makk_re__u__ lu_5: Leuk
output:
(None, 6, 500)
input: (None, 20)
Continued top
dense 11: Dense
oulput:
(None, 2)
Figure 13: Model architecture used for Local Minimum scenario comparison. The layer 'lambda: Lambda" sums the input components.
21

---

Under review as a conference paper at ICLR 2022
inpul:
[(None, 28, 2)1
conv2d_inpuff InpotLayer
output:
28, 2)
input:
(None, 28, 28, 2)
output:
(None, 26, 32)
imput:
(None, 26, 26, 32)
conv2d Conv2D
oulput:
(None, 24, 24, 64)
imput:
(None, 24, 24, 64)
max_pooling2d: MaxPoolingg2
output:
(None, 12, 12, 64)
input:
(None, 12, 12, 64)
dropout: Dropout
output:
(None, 12, 12, 64)
input:
(None, 12,
flatten: Flatten
oulput:
(None, 9216)
input:
(Nome, 9216)
output:
(None,
imput:
(None, 128)
dropout_ Dropout
output:
(None, 128)
input:
(None, 128)
dense_ I: Dense
output:
(None, 10)
Figure 14: Model architecture used for MNIST testing.
22

---

Under review as a conference paper at ICLR 2022
input:
[(None 32, 64, 2)]
conv2d input: InputLayer
output:
[(None, 32, 64, 2)]
input:
(None, 32, 64, 2)
conv2d: Conv2D
output:
(None, 30, 62, 32)
input:
(None, 30, 62, 32)
conv2d 1: Conv2D
output:
(None, 28, 60, 64)
input:
(None, 28, 60, 64)
max_poolinng max__pooling2d: MaxPooling2D
output:
(None, 14, 30, 64)
input:
(None, 14, 30, 64)
conv2d Conv2D
output:
(None, 12, 28, 64)
input:
(None, 12, 28, 64)
max naa__pooinggdd_1 1: MaxPooling2D
output:
(None, 6, 14, 64)
input:
(None, 6, 14, 64)
dropout: Dropout
output:
(None, 6, 14, 64)
input:
(None, 6, 14,
conv2d Conv2D
output:
(None, 10, 128)
input:
(None, 2, 10, 128)
conv2d 4: Conv2D
output:
(None, 2, 10, 10)
input:
(None, 2, 10, 10)
global max aaaammmaaollllaaaaa poolingood GlobalMaxPooling22
output:
(None, 10)
input: (None, 10)
activation: Activation
output:
(None, 10)
Figure 15: Model architecture used for MNIST in Space comparison.
23


Reviews:
[{"review_id": "e5M0G2bTKg6", "reviewer": "Reviewer_W9En", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "For the discussion in Figure 2. What is the exact mapping, how the numbers in Table 1 were gotten. For path A, shouldn’t the attribution IG is less for x1 because the partial derivation is uniform along the horizontal axis?\n\nFor the Figure 2 example, I can’t understand very well what is ‘maximally misleading’ and what is ‘attribution functions’. Why IG is unfair? How to measure fairness.\n\nFor my understanding, SHAP and IG are both post-hoc attributions. But the proposed ICG needs to perturb the input by certainty noise (my words) to train the model. If this is correct, what is the performance of IG or SHAP on such a trained model? Any difference with the ICG?\n\nFigure 3 also makes me confused. How to get the damaged input. What is the meaning of ‘sample’ here (bottom left). What do the green bounding box and blue bounding box mean? And blue and green rectangles as well.\n\nI’m also wondering is there any visual difference for ICG and IG in terms of the visualization results on the real image like they show in their paper?\n\nA typo, fourth line from the bottom of page 2, ‘any any’.", "summary_of_the_paper": "The paper studies the unfairness of a family of integrated gradients based attribution methods, by the what they call ‘attribution transfer’. To solve the problem, it proposes a integrated certainty gradients attribution by adding ‘noisy’ to the input during the model training. ", "summary_of_the_review": "The theoretical analysis part (section 3) is somehow hard to understand for me. This lowers my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pwt0b6IrtZ_", "reviewer": "Reviewer_FgJc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the theory side:\n\nThis paper can been as a discussion on integrated gradients. The paper evaluates different baselines and moves to investigate the (un)fairness of attribution methods that the authors define as “whether the relative attributions of input components are reasonable”. After considering the (un)fairness of attribution methods in this respect, the authors propose their attribution method. The authors then apply their method to data. I did not find enough theory backing the usefulness of the proposed methodology in various aspects. Even though the paper has applied their methodology and has shown practice, the paper lack in theoretical support.\n\nOn the application side:\n\nThe paper applies their methodology to two scenarios in the main text and also provides the results of some experiments about accuracy on the MNIST dataset in the supplementary materials.\n\nAbout the results in the main text, take Figure 4 as an example. The behavior of the proposed method is not clear. The method does not show attributions of the dark squares in columns B and C of the ICG row. Why is this the case? The authors have pointed out that repeating the algorithm leads to better results here, but the authors have not explained the reasons of this behavior clearly.\n \nAbout the results of the supplementary materials, it is unclear how the results on the MNIST dataset can be evaluated and compared against other results in the main text. Also, the results in section A.1.3 are not promising.\n\nOverall, the experiments of this paper are not convincing enough.\n\nLooking at the theory and application together, the paper lacks novelty.", "summary_of_the_paper": "This paper explores the characteristics of the method, Integrated Gradients, as an attribution method, that has been proposed to explain black box models. “Baselines” in analyzing integrated gradients are discussed and the shortcomings of integrated gradients are further evaluated. The paper then proposes Integrated Certainty Gradients and shows its application on data.", "summary_of_the_review": "Even though this paper approaches an interesting problem, it needs improvements and clarifications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "B3C5eW8g-HO", "reviewer": "Reviewer_Ejjc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The proposed Integrated Certainty Gradients training and attribution framework is novel. It makes sense that the authors utilize the fact that the integrated gradient path will be potentially less fluctuated than passing through the input space when it is only calculated along the direction of certainty change. Therefore, the impact of the previous issues on the IG could be mitigated.\n\nDetailed Comments:\n1. In Section 3.1, the example of attribution transfer is somewhat ambiguous. This part could be more readable if the authors elaborate more on how BShap, SHAP, IG gave the exact attribution value, respectively. \n\n2. In the first paragraph of Section 3.1, it is said that the model must indicate whether either of the two components falls within the range 0.25 - 0.5. However, from Figure 2 and its description, there is a differentiable region where the function value changes continuously (gradually from white to grey) at the edge of this range. I would suggest the authors modify this paragraph to be consistent with their figures.\n\n3. The formal definition of attribution transfer seems unclear. In the third paragraph of Section 3.1, it is said that \"We name this phenomenon attribution transfer\" while \"this\" here is ambiguous. Based on the context, I infer that attribution transfer refers to \"IG attribution results deviate from fair attribution due to the integration path traverses the fluctuating input space.\" However, I would like to see a more explicit definition given by the authors.\n \n4. In Section 3.2, I think the logical coherence of the paper would be better if the authors could explain how the axiomatic difference is related to the attribution transfer and the fairness of IG.\n\n5. In Section 4, it would be more convincing if the authors elaborate on the training process of ICG. I wonder whether there are changes compared with the ordinary training other than input data, such as loss function and backpropagation. Would ICG training affect the model performance? How and to what extend.\n\n6. Two purpose-designed experiments without any general scenarios are insufficient to justify the use of Integrated Certainty Gradients. I notice that the authors mention their experiments on MINIST with little details in the appendix. I expect the attribution results of this experiment.\n\n7. Since IG requires no modification of the original model while ICG trains the model with damaged data, it naturally makes me curious whether the utility of the ICG trained model decreases. I would suggest the authors show the test results of the ICG trained model and discuss whether there is a trade-off between generalization and attribution ability.", "summary_of_the_paper": "In this paper, the authors analyze the fairness of Integrated Gradient-based attribution methods. They exploit SHAP and BShap, two approaches based on the theory of Shapley Values, as the reference of \"fair\" methods. \n\nSpecifically, they present an \"attribution transfer\" phenomenon in which the Integrated Gradients are affected by some sharply fluctuated area across the integration path, thereby deviating from the ''fair'' attribution methods. To avoid the attribution transfer issue, they further propose Integrated Certainty Gradients method, where the integration path does not pass through the original fluctuated input space.  Such an objective can be achieved by training the network with perturbed inputs and corresponding certainty maps. Finally, the gradients integral can be calculated by querying the trained network with fixed inputs and varying certainty. Various purpose-designed experiments are performed to demonstrate the advantages of ICG in avoiding attribution transfer.", "summary_of_the_review": "Assessing fairness is important butt non-trivial. It is interesting to exploit SHAP and BShap, two approaches based on the theory of Shapley Values, as the reference of \"fair\" methods. Many concepts are not clear, and details are missing. Experiments are not comprehensive; please see the detailed comments.\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes analyze the fairness of Integrated Gradient based attribution methods. The authors exploit SHAP and BShap, two approaches based on the theory of Shapley Values, as the reference of "fair" methods. Specifically, they present an "attribution transfer" phenomenon in which the Integrated Gradients are affected by some sharply fluctuated area across the integration path, thereby deviating from the ''fair'' attribution methods. To avoid the attribution transfer issue, they further propose Integrated Certainty Gradients (ICG) method, where the integration path does not pass through the original fluctuated input space. Experiments are performed to demonstrate the advantages of ICG in avoiding attribution transfer. While the basic premise of the work is interesting, many conceptual details remain unclear and experimental evaluation can also be improved (please see detailed reviewer comments below). Given this, we are unable to recommend acceptance at this time. We hope the authors find the reviews helpful.
Decision: Reject