L-Shapley and C-Shapley: Efficient Model Interpretation for Structured Data

1 INTRODUCTION . Although many black box machine learning models , such as random forests , deep neural networks , and kernel methods , can produce highly accurate prediction in many applications , such prediction often comes at the cost of interpretability . Ease of interpretation is a crucial criterion when these tools are applied in areas such as medicine , financial markets , and criminal justice ; for more background , see the discussion paper by Lipton ( 2016 ) as well as references therein . In this paper , we study instancewise feature importance scoring as a specific approach to the problem of interpreting the predictions of black-box models . Given a predictive model , such a method yields , for each instance to which the model is applied , a vector of importance scores associated with the underlying features . The instancewise property means that this vector , and hence the relative importance of each feature , is allowed to vary across instances . Thus , the importance scores can act as an explanation for the specific instance , indicating which features are the key for the model to make its prediction on that instance . There is now a large body of research focused on the problem of scoring input features based on the prediction of a given instance ( see , e.g. , Shrikumar et al. , 2017 ; Bach et al. , 2015 ; Ribeiro et al. , 2016 ; Lundberg & Lee , 2017 ; Štrumbelj & Kononenko , 2010 ; Baehrens et al. , 2010 ; Datta et al. , 2016 ; Sundararajan et al. , 2017 ) . Of most relevance to this paper is a line of recent work ( Štrumbelj & Kononenko , 2010 ; Lundberg & Lee , 2017 ; Datta et al. , 2016 ) that has developed methods for model interpretation based on Shapley value ( Shapley , 1953 ) from cooperative game theory . The Shapley value was originally proposed as an axiomatic characterization of a fair distribution of a total surplus from all the players , and can be applied to predictive models , in which case each feature is modeled as a player in the underlying game . While the Shapley value approach is conceptually appealing , it is also computationally challenging : in general , each evaluation of a Shapley value requires an exponential number of model evaluations . Different approaches to circumventing this complexity barrier have been proposed , including those based on Monte Carlo approximation ( Štrumbelj & Kononenko , 2010 ; Datta et al. , 2016 ) and methods based on sampled least-squares with weights ( Lundberg & Lee , 2017 ) . In this paper , we take a complementary point of view , arguing that the problem of explanation is best approached within a model-based paradigm . In this view , explanations are cast in terms of a model , which may or may not be the same model as used to fit the data . Criteria such as Shapley value , which are intractable to compute when no assumptions are made , can be more effectively computed or approximated within the framework of a model . We focus specifically on settings in which a graph structure is appropriate for describing the relations between features in the data ( e.g. , chains for sequences and grids for images ) , and distant features according to the graph have weak interaction during the computation of Shapley values . We propose two methods for instancewise feature importance scoring in this framework , which we term L-Shapley and C-Shapley ; here the abbreviations “ L '' and “ C '' refer to “ local ” and “ connected , ” respectively . By exploiting the underlying graph structure , the number of model evaluations is reduced to linear—as opposed to exponential—in the number of features . We demonstrate the relationship of these measures with a constrained form of Shapley value , and we additionally relate C-Shapley with another solution concept from cooperative game theory , known as the Myerson value ( Myerson , 1977 ) . The Myerson value is commonly used in graph-restricted games , under a local additivity assumption of the model on disconnected subsets of features . Finally , we apply our feature scoring methods to several state-of-the-art models for both language and image data , and find that our scoring algorithms compare favorably to several existing sampling-based algorithms for instancewise feature importance scoring . 2 BACKGROUND AND PRELIMINARIES . We begin by introducing some background and notation for instancewise feature importance scoring and the Shapley value . 2.1 IMPORTANCE OF A FEATURE SUBSET . We are interested in studying models that are trained to perform prediction , taking as input a feature vector x ∈ X ⊂ Rd and predicting a response or output variable y ∈ Y . We assume access to the output of a model via a conditional distribution , denoted by Pm ( ·|x ) , that provides the distribution of the response Y ∈ Y conditioned on a given vector X = x of inputs . For any given subset S ⊂ { 1 , 2 , . . . , d } , we use xS = { xj , j ∈ S } to denote the associated sub-vector of features , and we let Pm ( Y | xS ) denote the induced conditional distribution when Pm is restricted to using only the sub-vector xS . In the corner case in which S = ∅ , we define Pm ( Y | x∅ ) : = Pm ( Y ) . In terms of this notation , for a given feature vector x ∈ X , subset S and fitted model distribution Pm ( Y | x ) , we introduce the importance score vx ( S ) : = Em [ − log 1 Pm ( Y | xS ) ∣∣∣ x ] , where Em [ · | x ] denotes the expectation over Pm ( · | x ) . The importance score vx ( S ) has a codingtheoretic interpretation : it corresponds to the negative of the expected number of bits required to encode the output of the model based on the sub-vector xS . It will be zero when the model makes a deterministic prediction based on xS , and larger when the model returns a distribution closer to uniform over the output space . There is also an information-theoretic interpretation to this definition of importance scores , as discussed in Chen et al . ( 2018 ) . In particular , suppose that for a given integer k < d , there is a function x 7→ S∗ ( x ) such that , for all almost all x , the k-sized subset S∗ ( x ) maximizes vx ( S ) over all subsets of size k ; then we are guaranteed that the mutual information I ( XS∗ ( X ) , Y ) between XS∗ ( X ) and Y is maximized , over any conditional distribution that generates a subset of size k given X . The converse is also true . In many cases , class-specific importance is favored , where one is interested in seeing how important a feature subset S is to the predicted class , instead of the prediction as a conditional distribution . In order to handle such cases , it is convenient to introduce the degenerate conditional distribution P̂m ( y | x ) : = { 1 if y ∈ arg max y′ Pm ( y′ | x ) , 0 otherwise . We can then define the importance of a subset S with respect to P̂m using the modified score vx ( S ) : = Êm [ − log 1 Pm ( Y | xS ) ∣∣∣ x ] , which is the expected log probability of the predicted class given the features in S. Estimating the conditional distribution : In practice , we need to estimate—for any given feature vector x̄ ∈ X—the conditional probability functions Pm ( y | x̄S ) based on observed data . Past work has used one of two approaches : either estimation based on empirical averages ( Štrumbelj & Kononenko , 2010 ) , or plug-in estimation using a reference point ( Datta et al. , 2016 ; Lundberg & Lee , 2017 ) . Empirical average estimation : In this approach , we first draw a set of feature vector { xj } Mj=1 by sampling with replacement from the full data set . For each sample xj , we define a new vector x̃j ∈ Rd with components ( x̃j ) i equal to xji if i ∈ S and x̄i otherwise . Taking the empirical mean of Pm ( y | x̃j ) over { x̃j } then provides an estimate of Pm ( y | x̄S ) . Plug-in estimation : In this approach , the first step is to specify a reference vector x0 ∈ Rd is specified . We then define the vector x̃ ∈ Rd with components ( x̃ ) i equal to xi if i ∈ S and x0i otherwise . Finally , we use the conditional probability Pm ( y | x̃ ) as an approximation to Pm ( y | x̄S ) . The plug-in estimate is more computationally efficient than the empirical average estimator , and works well when there exist appropriate choices of reference points . We use this method for our experiments , where we use the index of padding for language data , and the average pixel strength of an image for vision data . 2.2 SHAPLEY VALUE FOR MEASURING INTERACTION BETWEEN FEATURES . Consider the problem of quantifying the importance of a given feature index i for feature vector x . A naive way of doing so would be by computing the importance score vx ( { i } ) of feature i on its own . However , doing so ignores interactions between features , which are likely to be very important in applications . As a simple example , suppose that we were interested in performing sentiment analysis on the following sentence : It is not heartwarming or entertaining . It just sucks . ( ? ) This sentence is contained in a movie review from the IMDB movie data set ( Maas et al. , 2011 ) , and it is classified as negative sentiment by a machine learning model to be discussed in the sequel . Now suppose we wish to quantify the importance of feature “ not ” in prediction . The word “ not ” plays an important role in the overall sentence as being classified as negative , and thus should be attributed a significant weight . However , viewed in isolation , the word “ not ” has neither negative nor positive sentiment , so that one would expect that vx ( { “ not ” } ) ≈ 0 . Thus , it is essential to consider the interaction of a given feature i with other features . For a given subset S containing i , a natural way in which to assess how i interacts with the other features in S is by computing the difference between the importance of all features in S , with and without i . This difference is called the marginal contribution of i to S , and given by mx ( S , i ) : = vx ( S ) − vx ( S \ { i } ) . ( 1 ) In order to obtain a simple scalar measure for feature i , we need to aggregate these marginal contributions over all subsets that contain i . The Shapley value ( Shapley , 1953 ) is one principled way of doing so . For each integer k = 1 , . . . , d , we let Sk ( i ) denote the set of k-sized subsets that contain i . The Shapley value is obtained by averaging the marginal contributions , first over the set Sk ( i ) for a fixed k , and then over all possible choices of set size k : φx ( Pm , i ) : = 1 d d∑ k=1 1 ( d−1 k−1 ) ∑ S∈Sk ( i ) mx ( S , i ) . ( 2 ) Since the model Pm remains fixed throughout our analysis , we frequently omit the dependence of φx on Pm , instead adopting the more compact notation φx ( i ) . The concept of Shapley value was first introduced in cooperative game theory ( Shapley , 1953 ) , and it has been used in a line of recent work on instancewise feature importance ranking ( Štrumbelj & Kononenko , 2010 ; Datta et al. , 2016 ; Lundberg & Lee , 2017 ) . It can be justified on an axiomatic basis ( Shapley , 1953 ; Young , 1985 ) as being the unique function from a collection of 2d numbers ( one for each subset S ) to a collection of d numbers ( one for each feature i ) with the following properties : ( i ) [ Additivity ] The sum of the Shapley values ∑d i=1 φx ( i ) is equal to the difference vx ( { 1 , . . . , d } ) − vx ( ∅ ) . ( ii ) [ Equal contributions ] If vx ( S ∪ { i } ) = vx ( S ∪ { j } ) for all subsets S , then φx ( i ) = φx ( j ) . ( iii ) [ Monotonicity ] Given two models Pm and Pm , let mx and m′x denote the associated marginal contribution functions , and let φx and φ′x denote the associated Shapley values . If mx ( S , i ) ≥ m′x ( S , i ) for all subsets S , then we are guaranteed that φx ( i ) ≥ φ′x ( i ) . Note that all three of these axioms are reasonable in our feature selection context .

Summary: The paper proposes two approximations to the Shapley value used for generating feature scores for interpretability. Both exploit a graph structure over the features by considering only subsets of neighborhoods of features (rather than all subsets). The authors give some approximation guarantees under certain Markovian assumptions on the graph. The paper concludes with experiments on text and images.