Shapley Residuals: Quantifying the limits of the Shapley value for explanations

1 Introduction . There have been many recent efforts to quantify the importance of features to a model [ 19 , 4 , 1 , 15 , 12 , 13 ] . Many of these determine the importance through estimating the Shapley value of a game designed to assign importance to sets of features [ 4 , 12 , 5 , 13 , 14 , 25 ] . These Shapley-value-based feature importance methods are used widely in practice [ 2 ] . At the same time , there have been increasing concerns that these game theoretic values may not completely capture human or technical notions of feature importance [ 10 , 21 , 24 ] . A particularly salient issue is that users have misconceptions about what Shapley values represent and what actionable information can be gleaned from them [ 8 ] . Non-linear complex models , and models built on correlated features , do not have Shapley values that can be interpreted as the effect of a direct intervention [ 10 ] , e.g. , so that increasing a variable value changes the model outcome in a predictable way . The goal of this work is to quantify the extent of these concerns and provide a theoretical foundation for understanding the limits of Shapley values . In this work , we introduce Shapley Residuals , vector-valued objects that capture a specific type of quantitative information lost by Shapley values . Shapley residuals can be associated with individual 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . variables , as well as with sets of variables . When the residual of a feature exhibits a large norm , the associated Shapley value should be taken with skepticism : the resulting importance is not just due to the variable acting by itself . On the other hand , if a residual is small , most of the effect of the variable on the model is explainable by the variable acting independently ( we make these statements precise in Section 3 ) . The Shapley residual , then , communicates important details about what the explanation actually represents . To build an intuition for why this is an important problem , consider an algorithm which makes admissions decisions purely on the basis of gender and department : f ( g , d ) = g + d 2dg , where g = 1 if the applicant is male and g = 1 otherwise , and there are two departments , represented by d = 1 and d = 1 . In this contrived scenario , the applicant is admitted if f ( g , d ) > 0 ( which only happens when g and d have different signs ) and is rejected otherwise . Clearly , the admissions decision is affected by gender–yet if each of the two variables are distributed with mean 0 , the KernelSHAP values [ 12 ] which are supposed to explain the decision f ( 1 , 1 ) = 0 are both 0 , since to compute the Shapley value , each features ’ univariate and interaction influences are averaged together and cancel each other out . In this way , the computation of the Shapley value has implicitly obscured a discriminatory effect , and the corresponding nonzero Shapley residuals would demonstrate that the Shapley values are not telling the whole story . To more precisely describe what Shapley residuals capture , consider the following two motivating scenarios . First , suppose a practitioner uses Shapley values to determine the effect of data interventions on model outcomes . Consider two models f1 and f2 . In a real-world scenario , the practitioner will often only have black-box access to such models , and the models will often be significantly more complex . Here , we use these simple models : f1 ( x1 , x2 , x3 ) = x1 + x2 + x3 f2 ( x1 , x2 , x3 ) = x1 + 2x2x3 Suppose the practitioner seeks to explain the output f1 ( 1 , 1 , 1 ) = 3 or f2 ( 1 , 1 , 1 ) = 3 , using KernelSHAP to compute local feature importances . For both models , the Shapley values of x1 , x2 , and x3 are all 1 . Despite that , intervening by increasing the value of x2 changes f2 more than increasing the value of x1 ; in f1 , this clearly does not happen . The Shapley residuals for all variables in f1 are zero , indicating that variables in f1 do not interact ( as we prove in Section 3 ) . The Shapley residuals for x2 and x3 in f2 , on the other hand , are nonzero , while the Shapley residual of x1 is still zero . Finally , the Shapley residual for the set of variables { x2 , x3 } is also zero . As we show in Section 3 , these statements imply the following behavior for variables of f2 : x1 has no interactions with other variables ( its residual is zero ) ; x2 and x3 interact with other variables ( their residuals are non zero ) ; x2 and x3 only interact with each other ( the residual of the set { x2 , x3 } is zero ) . Thus , access to Shapley residuals gives warning that intervening on x2 or x3 in f2 could act differently than x1 due to an interaction between x2 and x3 . In the second scenario , consider a data generating distribution where ↵ controls the correlation between two features in X and a regression target y : ( X , y ) ⇠ ✓ N ✓ ( 0 , 0 ) , 1 ↵ ↵ 1 ◆ , hX , ( 3 , 1 ) i ◆ . We examine a regression model f ( x1 , x2 ) = 1x1+ 2x2 determined via linear least squares . Assume access to infinitely many IID samples from ( X , y ) , = ( 3 , 1 ) . Suppose a practitioner wanted to explain the output of f ( 1 , 1 ) = 1 + 2 , this time using Conditional Expectation SHAP [ 24 ] . The Shapley values are 1 + ↵ ( 2 1 ) /2 for x1 and 2 + ↵ ( 1 2 ) /2 for x2 . When ↵ ⇡ 0 , Shapley values correspond to model weights 1 , 2 , and support a ( valid ) interventional interpretation that changing x1 yields a larger change to the output of f than does x2 . However , if ↵ ⇡ 1 , Shapley values do not support this interpretation . A practitioner employing Shapley values alone lacks the information to distinguish these scenarios . Shapley residuals provide useful diagnostic information ; the norm of the residuals for x1 and x2 is exactly linearly proportional to ↵ . In these simple scenarios , it is clear that Shapley residuals capture , respectively , variable interactions and mismatches between dependent features in the data and independent variables in the model . As we show in Section 6 , these observations apply to real-world scenarios as well . In summary , we : • introduce Shapley residuals ( Section 3 ) , which characterize the limits of Shapley values as explanatory mechanisms for cooperative games , • study the properties of Shapley residuals both in general and in context of existing formulations for explanatory games ( Sections 3 , 4 and 5 ) , • show via a number of experiments that Shapley residuals capture meaningful information for model explanations in realistic scenarios ( Section 6 ) , • discuss the limitations of Shapley residuals themselves ( Section 7 ) . 2 Background . In this section , we begin by setting up the mathematical definitions and background we ’ ll need for the rest of the paper . To help illustrate these ideas , we ’ ll use the running example from the introduction of function f = x1 + 2x2x3 ; we refer to this as Example 1 . We begin by describing Shapley values and cooperative games . Games . A cooperative game consists of d players and a value function v : 2 [ d ] ! R where [ d ] , { 1 , . . . , d } . The quantity v ( S ) represents the value of the game for a coalition of players S 2 N , 2 [ d ] . Without loss of generality we will assume that v ( ; ) = 0 , and that we can identify the game with v. Let the space of games be denoted by G. The Shapley value is a way to fairly allocate the value of the grand coalition v ( [ d ] ) among the players . Definition 1 ( Shapley values [ 20 ] ) . The Shapley values i ( v ) , i 2 [ d ] are the unique values satisfying the properties Efficiency : Pd i=1 i ( v ) = v ( [ d ] ) . Dummy : If v ( S [ { i } ) = v ( S ) for all S ⇢ [ d ] \ { i } , then i ( v ) = 0 . Symmetry : If v ( S [ { i } ) = v ( S [ { j } ) for all S ⇢ [ d ] \ { i , j } , then i ( v ) = j ( v ) . Linearity : If v , v0 are two games on d players , then i ( ↵v + ↵0v0 ) = ↵ i ( v ) + ↵0 i ( v0 ) . Given a model f ( x1 , x2 , ... , xd ) , the features from 1 to d can be considered players in a game in which the payoff v is some measure of the importance or influence of a subset of features . The Shapley value i ( v ) can then be viewed as a fairly attributed “ influence '' of i on the outcome v ( [ d ] ) . In KernelSHAP , for instance , a function ’ s prediction on a certain input given a data distribution is modeled as a game as shown in Table 1 . It will be useful for us to visualize a game as a function over the vertices of a d-dimensional hypercube . Each coordinate corresponds to the presence or absence of a certain player , and each vertex corresponds to a subset of players . Specifically , we can think of the set N as the d-dimensional hypercube G = ( V = N , E ) with each vertex labeled by a set S ✓ [ d ] and edges between sets S and S [ { i } for all i 2 [ d ] , S. We depict this interpretation in Figure 1 ( a ) . Gradients on the hypercube . Let RV be the space of functions from V to R and let RE be the space of functions from E to R. In particular , the game v is an element of RV . The differential operator r : RV ! RE is then defined as rv ( S , S [ { i } ) = v ( S [ i ) v ( S ) for any v 2 RV . Essentially r is a discrete gradient operator on G , mapping functions on vertices to functions on edges ( see Figure 1 ( b ) ) . We will also define a partial gradient ri : RV ! RE : riu ( S , S [ { j } ) = ⇢ u ( S [ j ) u ( S ) i = j 0 otherwise Intuitively , ri evaluates a gradient for edges corresponding to the insertion of i , and takes the value 0 everywhere else . On the hypercube , only edges on the ith axis of riv will take a nonzero value . See the Edge Space portion of Figure 2 ( a ) for an illustration of this procedure on the running example .

Summary: In a cooperative game in characteristic function form, the value function $v$ maps from all subsets of agents, $N = {1, \ldots, n}$, producing an image with $2^N$ elements.  Shapley's value (perhaps the most widely used measure of feature importance in handling ML's attribution problem) ascribes a number to each agent, for a set of $N$ values.    As $N < 2^N$, Shapley's value does not - in general - fully encode the game.  An exception occurs when the game is _inessential_, so that $v(S) = \sum_{i \in S} v(i)$.  Specifically, when features _interact_, Shapley's value may be restrictive.  This paper builds on Stern and Tettenhorst's 2019 _Games and Economic Behavior_ paper, which provided a way of assessing a game's deviation from _inessentiality_, by means of Shapley residuals, $r_i$.  Definition 4 defines _Shapley residuals_, $r_S$, for sets of agents/features.  A series of results (Proposition 1, Corollaries 1, 2) then allow the norms of $r_S$ to be interpreted as a deviation from essentiality and - thus - a measure of Shapley's value's failure to fully encode the game.  Lemma 2 then makes precise the relationship between the absence of interactions between variables and inessentiality.  Section 5 compares the Shapley residual to the Shapley-Taylor interaction index.