Shapley Residuals: Quantifying the limits of the Shapley value for explanations

1 Introduction . There have been many recent efforts to quantify the importance of features to a model [ 19 , 4 , 1 , 15 , 12 , 13 ] . Many of these determine the importance through estimating the Shapley value of a game designed to assign importance to sets of features [ 4 , 12 , 5 , 13 , 14 , 25 ] . These Shapley-value-based feature importance methods are used widely in practice [ 2 ] . At the same time , there have been increasing concerns that these game theoretic values may not completely capture human or technical notions of feature importance [ 10 , 21 , 24 ] . A particularly salient issue is that users have misconceptions about what Shapley values represent and what actionable information can be gleaned from them [ 8 ] . Non-linear complex models , and models built on correlated features , do not have Shapley values that can be interpreted as the effect of a direct intervention [ 10 ] , e.g. , so that increasing a variable value changes the model outcome in a predictable way . The goal of this work is to quantify the extent of these concerns and provide a theoretical foundation for understanding the limits of Shapley values . In this work , we introduce Shapley Residuals , vector-valued objects that capture a specific type of quantitative information lost by Shapley values . Shapley residuals can be associated with individual 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . variables , as well as with sets of variables . When the residual of a feature exhibits a large norm , the associated Shapley value should be taken with skepticism : the resulting importance is not just due to the variable acting by itself . On the other hand , if a residual is small , most of the effect of the variable on the model is explainable by the variable acting independently ( we make these statements precise in Section 3 ) . The Shapley residual , then , communicates important details about what the explanation actually represents . To build an intuition for why this is an important problem , consider an algorithm which makes admissions decisions purely on the basis of gender and department : f ( g , d ) = g + d 2dg , where g = 1 if the applicant is male and g = 1 otherwise , and there are two departments , represented by d = 1 and d = 1 . In this contrived scenario , the applicant is admitted if f ( g , d ) > 0 ( which only happens when g and d have different signs ) and is rejected otherwise . Clearly , the admissions decision is affected by gender–yet if each of the two variables are distributed with mean 0 , the KernelSHAP values [ 12 ] which are supposed to explain the decision f ( 1 , 1 ) = 0 are both 0 , since to compute the Shapley value , each features ’ univariate and interaction influences are averaged together and cancel each other out . In this way , the computation of the Shapley value has implicitly obscured a discriminatory effect , and the corresponding nonzero Shapley residuals would demonstrate that the Shapley values are not telling the whole story . To more precisely describe what Shapley residuals capture , consider the following two motivating scenarios . First , suppose a practitioner uses Shapley values to determine the effect of data interventions on model outcomes . Consider two models f1 and f2 . In a real-world scenario , the practitioner will often only have black-box access to such models , and the models will often be significantly more complex . Here , we use these simple models : f1 ( x1 , x2 , x3 ) = x1 + x2 + x3 f2 ( x1 , x2 , x3 ) = x1 + 2x2x3 Suppose the practitioner seeks to explain the output f1 ( 1 , 1 , 1 ) = 3 or f2 ( 1 , 1 , 1 ) = 3 , using KernelSHAP to compute local feature importances . For both models , the Shapley values of x1 , x2 , and x3 are all 1 . Despite that , intervening by increasing the value of x2 changes f2 more than increasing the value of x1 ; in f1 , this clearly does not happen . The Shapley residuals for all variables in f1 are zero , indicating that variables in f1 do not interact ( as we prove in Section 3 ) . The Shapley residuals for x2 and x3 in f2 , on the other hand , are nonzero , while the Shapley residual of x1 is still zero . Finally , the Shapley residual for the set of variables { x2 , x3 } is also zero . As we show in Section 3 , these statements imply the following behavior for variables of f2 : x1 has no interactions with other variables ( its residual is zero ) ; x2 and x3 interact with other variables ( their residuals are non zero ) ; x2 and x3 only interact with each other ( the residual of the set { x2 , x3 } is zero ) . Thus , access to Shapley residuals gives warning that intervening on x2 or x3 in f2 could act differently than x1 due to an interaction between x2 and x3 . In the second scenario , consider a data generating distribution where ↵ controls the correlation between two features in X and a regression target y : ( X , y ) ⇠ ✓ N ✓ ( 0 , 0 ) , 1 ↵ ↵ 1 ◆ , hX , ( 3 , 1 ) i ◆ . We examine a regression model f ( x1 , x2 ) = 1x1+ 2x2 determined via linear least squares . Assume access to infinitely many IID samples from ( X , y ) , = ( 3 , 1 ) . Suppose a practitioner wanted to explain the output of f ( 1 , 1 ) = 1 + 2 , this time using Conditional Expectation SHAP [ 24 ] . The Shapley values are 1 + ↵ ( 2 1 ) /2 for x1 and 2 + ↵ ( 1 2 ) /2 for x2 . When ↵ ⇡ 0 , Shapley values correspond to model weights 1 , 2 , and support a ( valid ) interventional interpretation that changing x1 yields a larger change to the output of f than does x2 . However , if ↵ ⇡ 1 , Shapley values do not support this interpretation . A practitioner employing Shapley values alone lacks the information to distinguish these scenarios . Shapley residuals provide useful diagnostic information ; the norm of the residuals for x1 and x2 is exactly linearly proportional to ↵ . In these simple scenarios , it is clear that Shapley residuals capture , respectively , variable interactions and mismatches between dependent features in the data and independent variables in the model . As we show in Section 6 , these observations apply to real-world scenarios as well . In summary , we : • introduce Shapley residuals ( Section 3 ) , which characterize the limits of Shapley values as explanatory mechanisms for cooperative games , • study the properties of Shapley residuals both in general and in context of existing formulations for explanatory games ( Sections 3 , 4 and 5 ) , • show via a number of experiments that Shapley residuals capture meaningful information for model explanations in realistic scenarios ( Section 6 ) , • discuss the limitations of Shapley residuals themselves ( Section 7 ) . 2 Background . In this section , we begin by setting up the mathematical definitions and background we ’ ll need for the rest of the paper . To help illustrate these ideas , we ’ ll use the running example from the introduction of function f = x1 + 2x2x3 ; we refer to this as Example 1 . We begin by describing Shapley values and cooperative games . Games . A cooperative game consists of d players and a value function v : 2 [ d ] ! R where [ d ] , { 1 , . . . , d } . The quantity v ( S ) represents the value of the game for a coalition of players S 2 N , 2 [ d ] . Without loss of generality we will assume that v ( ; ) = 0 , and that we can identify the game with v. Let the space of games be denoted by G. The Shapley value is a way to fairly allocate the value of the grand coalition v ( [ d ] ) among the players . Definition 1 ( Shapley values [ 20 ] ) . The Shapley values i ( v ) , i 2 [ d ] are the unique values satisfying the properties Efficiency : Pd i=1 i ( v ) = v ( [ d ] ) . Dummy : If v ( S [ { i } ) = v ( S ) for all S ⇢ [ d ] \ { i } , then i ( v ) = 0 . Symmetry : If v ( S [ { i } ) = v ( S [ { j } ) for all S ⇢ [ d ] \ { i , j } , then i ( v ) = j ( v ) . Linearity : If v , v0 are two games on d players , then i ( ↵v + ↵0v0 ) = ↵ i ( v ) + ↵0 i ( v0 ) . Given a model f ( x1 , x2 , ... , xd ) , the features from 1 to d can be considered players in a game in which the payoff v is some measure of the importance or influence of a subset of features . The Shapley value i ( v ) can then be viewed as a fairly attributed “ influence '' of i on the outcome v ( [ d ] ) . In KernelSHAP , for instance , a function ’ s prediction on a certain input given a data distribution is modeled as a game as shown in Table 1 . It will be useful for us to visualize a game as a function over the vertices of a d-dimensional hypercube . Each coordinate corresponds to the presence or absence of a certain player , and each vertex corresponds to a subset of players . Specifically , we can think of the set N as the d-dimensional hypercube G = ( V = N , E ) with each vertex labeled by a set S ✓ [ d ] and edges between sets S and S [ { i } for all i 2 [ d ] , S. We depict this interpretation in Figure 1 ( a ) . Gradients on the hypercube . Let RV be the space of functions from V to R and let RE be the space of functions from E to R. In particular , the game v is an element of RV . The differential operator r : RV ! RE is then defined as rv ( S , S [ { i } ) = v ( S [ i ) v ( S ) for any v 2 RV . Essentially r is a discrete gradient operator on G , mapping functions on vertices to functions on edges ( see Figure 1 ( b ) ) . We will also define a partial gradient ri : RV ! RE : riu ( S , S [ { j } ) = ⇢ u ( S [ j ) u ( S ) i = j 0 otherwise Intuitively , ri evaluates a gradient for edges corresponding to the insertion of i , and takes the value 0 everywhere else . On the hypercube , only edges on the ith axis of riv will take a nonzero value . See the Edge Space portion of Figure 2 ( a ) for an illustration of this procedure on the running example .

Summary: This paper introduces Shapley Residuals, a new measure that can be calculated in addition to SHAP values. This measure quantifies in some sense the information lost by Shapley Value due to interactions or correlations between features. This measure can help practitioners to identify scenarios when a score given to a certain feature by SHAP may not correctly quantify the effect of this feature on the prediction due to interactions or correlations. The paper provides theoretical grounding for the residuals, along with toy examples and an experiment with UCI dataset.