Exploring unfairness in Integrated Gradients based attribution methods

Numerous methods have attempted to explain and interpret predictions made by machine learning models in terms of their inputs . Known as “ attribution methods ” they notably include the Integrated Gradients method and its variants . These are based upon the theory of Shapley Values , a rigorous method of fair allocation according to mathematical axioms . Integrated Gradients has axioms derived from this heritage with the implication of a similar rigorous , intuitive notion of fairness . We explore the difference between Integrated Gradients and more direct expressions of Shapley Values in deep learning and find Integrated Gradients ’ guarantees of fairness weaker ; in certain conditions it can give wholly unrepresentative results . Integrated Gradients requires a choice of “ baseline ” , a hyperparameter that represents the ‘ zero attribution ’ case . Research has shown that baseline choice critically affects attribution quality , and increasingly effective baselines have been developed . Using purpose-designed scenarios we identify sources of inaccuracy both from specific baselines and inherent to the method itself , sensitive to input distribution and loss landscape . Failure modes are identified for baselines including Zero , Mean , Additive Gaussian Noise , and the state of the art Expected Gradients . We develop a new method , Integrated Certainty Gradients , that we show avoids the failures in these challenging scenarios . By augmenting the input space with “ certainty ” information , and training with random degradation of input features , the model learns to predict with varying amounts of incomplete information , supporting a zero-information case which becomes a natural baseline . We identify the axiomatic origin of unfairness in Integrated Gradients , which has been overlooked in past research . 1 INTRODUCTION . Attribution , the identification of the input features most salient to a model prediction , is an increasingly important requirement for neural networks . It is a core part of model interpretability , which is valuable as a research tool and design aid , but also increasingly as an output requirement in its own right for uses as diverse as medical imaging ( Singh et al. , 2020 ) to loan applications ( Bhatt et al. , 2020 ) . A “ right to explanation ” of machine decisions is even provided in the European Union ’ s General Data Protection Regulation ( Goodman & Flaxman , 2017 ) . Gradients of predictions with respect to model inputs can be calculated using backpropagation and have been used for feature attribution ( “ Vanilla Gradient ” ) ( Erhan et al. , 2009 ; Simonyan et al. , 2014 ; Yosinski et al. , 2015 ) . The gradients indicate which features are most sensitive to a perturbation , causing the largest change in prediction outcome . Empirically , this method often does highlight areas relevant to prediction , but it may also miss important areas due to the nonlinearity of the model and saturation of the gradients ( Sundararajan et al. , 2016 ) . The method of Integrated Gradients ( Sundararajan et al. , 2016 ; 2017 ) overcomes these issues by generating attributions based on the theory of Shapley values ( Sundararajan et al. , 2017 ) . The method integrates the gradients as the input varies linearly between a baseline and the final input of interest : Integrated Gradients ( F , x′ , x , i ) : = ( xi − x′i ) ︸ ︷︷ ︸ distance ∫ 1 α=0 ∂F ( ˙ x ) ∂ ˙ xi ∣∣∣∣ ˙ x = x′ + α ( x− x′ ) dα︸ ︷︷ ︸ mean gradient ( 1 ) where F ( · ) denotes the prediction of the model , x is the input vector of interest , x′ is the baseline input vector , and i indexes to the feature of interest . A numerical approximation is normally used to calculate the integral in practice . The baseline represents a missing or neutral input , a concept required by the Shapley values theory , but shared with several other attribution methods ( Covert et al. , 2020 ) . In the remainder of this paper , we first give an overview of baselines discussed in the literature ( Section 2 ) . Then we discuss the theoretical cause of a failure mode of IG ( attribution transfer ) using a zero baseline example ( Section 3 ) . In Section 4 we introduce a new method , Integrated Certainty Gradients . Finally we show in experiments that other baselines including the state of the art Expected Gradients do not prevent attribution transfer , while in the tested scenarios Integrated Certainty Gradients does , and discuss additional failure cases of specific baselines ( Section 5 ) . Note on notation It will be convenient to imply an equivalence between attribution methods which take a baseline , ϕ ( F , x′ , x , i ) , and those which do not ϕ ( F , x , i ) . A baseline free method can be adapted to take a baseline by ϕB ( F , x′ , x , i ) = ϕ ( F , x , i ) − ϕ ( F , x′ , i ) . 2 CHOICE OF BASELINE . Baseline choice has a major effect on attribution outcome . Effective choice of baselines in practice is a topic of ongoing research . A recent review of four state-of-the-art models for tabular data did not find a best performing choice ( Haug et al. , 2021 ) . As this paper noted , the concept of missingness within an arbitrary space is domain-specific . Therefore applying attribution techniques to a new domain requires this concept to be determined . Another review came to similar conclusions for image classification attribution ( Sturmfels et al. , 2020 ) . Baselines can be generalized from single inputs to distributions by taking the expected value of Integrated Gradients using each input in the distribution . In practice X ′ is often from an empirical ( discrete uniform ) distribution so the expectation becomes simply the mean over X ′ . A summary of common baselines used in Integrated Gradients and other attribution methods follows . For a more comprehensive review , one can refer to Haug et al . ( 2021 ) ; Sturmfels et al . ( 2020 ) . Visualizations of integration paths for various baselines are shown in Figure 1 . Constant value baselines include the uniform black/zero baseline proposed with the original method ( Sundararajan et al. , 2016 ) and choices that attempt to be unbiased with respect to the input distribution , including the component-wise mean of the input distribution , the middle of the input range , and uniform noise . If the input matches the baseline , the xi − x′i term in Equation 1 disappears , any such areas will not be attributed , even when they are essential to the model ’ s prediction ( “ baseline blindness ” ) ( Sturmfels et al. , 2020 ) . When the baseline is constant this undesirable behavior can not be avoided . This behavior can be seen clearly in attributions of MNIST images with a zero baseline , for example in Adebayo et al . ( 2020 ) . Distribution baselines avoid baseline blindness by combining multiple attributions . Uniform noise has been used in this way , although concerns have been identified about the risk of interaction between the baseline and high frequency features ( Sundararajan et al. , 2017 ; Sturmfels et al. , 2020 ) . The recently developed Expected Gradients method ( Erion et al. , 2020 ) uses the input ( typically training ) distribution as the baseline . It computes the attribution for a given input example efficiently by sampling baselines randomly from the training distribution , and interpolation weights from the the [ 0 , 1 ] range ( Monte Carlo integration ) , rather than using the many equally spaced interpolations of the more common Riemann Sum method . Dynamic baselines change based on the attribution input . The maximum distance baseline uses the farthest point in the input space from the input vector ( Sturmfels et al. , 2020 ) . This avoids baseline blindness but abandons the concept of neutrality . Sturmfels et al . ( 2020 ) cite Fong & Vedaldi ( 2017 ) as the inspiration for blurring as a possible method of feature ablation to produce a baseline . Gaussian noise may be added to the input vector during attribution and when averaged over several iterations , the resulting attributions were found to be more detailed and less noisy ( Smilkov et al. , 2017 ) . Adding noise to input vectors during training was also found to give a similar and cumulative effect . The combination of input vector with noise can be used by itself as a dynamic distribution baseline ( Sturmfels et al. , 2020 ) or as an augmentation technique in combination with another baseline . The justification of this method for use with attribution appears to be mainly empirical , and a question regarding the theoretical justification has been raised ( Sturmfels et al. , 2020 ) . 3 ATTRIBUTION FAIRNESS . Assessment of attribution methods is not currently straightforward because there is not yet a unified general procedure to do so . Rather , there are a number of tests for various indicative characteristics ( Hooker et al. , 2019 ; Adebayo et al. , 2018 ; Yang & Kim , 2019 ) . We focus our attention on fairness : whether the relative attributions of input components are reasonable . This is the central concern of Shapley Values , a celebrated method from game theory for allocating value , with a strongly justified mathematical foundation ( Shapley , 1952 ) , and the background to the Integrated Gradients methods ( Sundararajan et al. , 2017 ) . Assessing fairness is not trivial . As a reference we consider two other attribution methods defined using the original Shapley Values procedure : BShap ( Sundararajan & Najmi , 2020 ) and SHAP ( Lundberg & Lee , 2017 ) . Shapley Values is a procedure that , for a given set S , requires a value for every possible subset of elements : v : P ( S ) → R , where P is the power set of S , and gives back values for the elements themselves ( the “ Shapley values ” ) : φsv ( v , e ) . The Shapley values sum to the value of the complete set : v ( S ) = ∑ e∈S φsv ( v , e ) , and certain axioms , widely considered as naturally related to fairness , are maintained . As such it gives a way of sharing the value of the complete set fairly among the elements , based on the values that are achieved by each subset . Model inputs can not typically represent missing components , so to allocate attribution to the components of an input vector using Shapley Values , presence/absence of a component must be replaced by another binary property . In the case of BShap , ‘ present ’ and ‘ absent ’ components are represented with values taken from the equivalent components of the input vector x or a baseline x′ respectively . The value function v is simply the model ’ s prediction F . SHAP represents subsets using random vectors , and the value function v by the expectation of the model ’ s prediction E ◦ F . Each vector has a distribution f ( X|Xi=xi∀i ) where X is the distribution of possible inputs , x is the input being attributed , and i are the ‘ present ’ components of the vector . Further information about Shapley Values and these methods is given is Section A.1 of the supporting material . To summarize the relationships : φsv ( v , e ) v sets elements e S { } ϕBShap ( F , x ′ , x , i ) F vectors components i x x′ ϕSHAP ( F , x , i ) E ◦ F random vectors components i a = x a ∼ X

Summary: This paper explores the characteristics of the method, Integrated Gradients, as an attribution method, that has been proposed to explain black box models. “Baselines” in analyzing integrated gradients are discussed and the shortcomings of integrated gradients are further evaluated. The paper then proposes Integrated Certainty Gradients and shows its application on data.