Learning Explainable Models Using Attribution Priors

1 INTRODUCTION . Recent work on interpreting machine learning models has focused on feature attribution methods . Given an input feature , a model , and a prediction on a particular sample , such methods assign a number to the input feature that represents how important the input feature was for making the prediction . Previous literature about such methods has focused on the axioms they should satisfy ( Lundberg and Lee , 2017 ; Sundararajan et al. , 2017 ; Štrumbelj and Kononenko , 2014 ; Datta et al. , 2016 ) , and how attribution methods can give us insight into model behavior ( Lundberg et al. , 2018a ; b ; Sayres et al. , 2019 ; Zech et al. , 2018 ) . These methods can be an effective way of revealing problems in a model or a dataset . For example , a model may place too much importance on undesirable features , rely on many features when sparsity is desired , or be sensitive to high frequency noise . In such cases , we often have a prior belief about how a model should treat input features , but for neural networks it can be difficult to mathematically encode this prior in terms of the original model parameters . Ross et al . ( 2017b ) introduce the idea of regularizing explanations to train models that better agree with domain knowledge . Given a binary variable indicating whether each feature should or should not be important for predicting on each sample in the dataset , their method penalizes the gradients of unimportant features . However , two drawbacks limit the method ’ s applicability to real-world problems . First , gradients don ’ t satisfy the theoretical guarantees that modern feature attribution methods do ( Sundararajan et al. , 2017 ) . Second , it is often difficult to specify which features should be important in a binary manner . More recent work has stressed that incorporating intuitive , human priors will be necessary for developing robust and interpretable models ( Ilyas et al. , 2019 ) . Still , it remains challenging to encode meaningful , human priors like “ have smoother attribution maps ” or “ treat this group of features similarly ” by penalizing the gradients or parameters of a model . In this work , we propose an expanded framework for encoding abstract priors , called attribution priors , in which we directly regularize differentiable functions of a model ’ s axiomatic feature attributions during training . This framework , which can be seen as a generalization of gradient-based regularization ( LeCun et al. , 2010 ; Ross et al. , 2017b ; Yu et al. , 2018 ; Jakubovitz and Giryes , 2018 ; Roth et al. , 2018 ) , can be used to encode meaningful domain knowledge more effectively than existing methods . Furthermore , we introduce a novel feature attribution method - expected gradients - which extends integrated gradients ( Sundararajan et al. , 2017 ) , is naturally suited to being regularized under an attribution prior , and avoids hyperparameter choices required by previous methods . Using attribution priors , we build improved deep models for three different prediction tasks . On images , we use our framework to train a deep model that is more interpretable and generalizes better to noisy data by encouraging the model to have piecewise smooth attribution maps over pixels . On gene expression data , we show how to both reduce prediction error and better capture biological signal by encouraging similarity among gene expression features using a graph prior . Finally , on a patient mortality prediction task , we develop a sparser model and improve performance when learning from limited training data by encouraging a skewed distribution of the feature attributions . 2 ATTRIBUTION PRIORS . In this section , we formally define an attribution prior , and give three example priors for different data types . Let X ∈ Rn×p denote a dataset with labels y ∈ Ro , where n is the number of samples , p is the number of features , and o is the number of outputs . In standard deep learning we aim to find optimal parameters θ by minimizing loss , subject to a regularization term Ω′ ( θ ) on the parameters : θ = argminθL ( θ ; X , y ) + λ′Ω′ ( θ ) . For some model parameters θ , let Φ ( θ , X ) be a feature attribution method , which is a function of θ and the data X . Let φ ` i be the feature importance of feature i in sample ` . We formally define an attribution prior as a scalar-valued penalty function of the feature attributions , Ω ( Φ ( θ , X ) ) , which represents a log-transformed prior probability distribution over possible attributions . θ = argminθL ( θ ; X , y ) + λΩ ( Φ ( θ , X ) ) , where λ is the regularization strength . We note that the attribution prior function Ω is agnostic to the attribution method Φ . While in Section 3 we propose a feature attribution method for attribution priors , other attribution methods can be used . This includes existing methods like integrated gradients or simply the gradients themselves . In the latter case , we can see the method proposed in Ross et al . ( 2017b ) as a specific instance of an attribution prior : θ = argminθL ( θ ; X , y ) + λ′′||A ∂L ∂X ||2F where the attribution method Φ ( θ , X ) is the gradients of the model , represented by the matrix ∂L∂X whose ` , ith entry is the gradient of the loss at the ` th sample with respect to the ith feature . A is a binary matrix indicating which features should be penalized in which samples . Often , however , we do not know which features are important in advance . Instead , we can define different attribution priors for different tasks depending on the data and our domain knowledge . To demonstrate how attribution priors can capture human intuition in a variety of domains , in the following sections we first define and then apply three different priors for three different data types . 2.1 PIXEL ATTRIBUTION PRIOR FOR IMAGE CLASSIFICATION . Prior work on interpreting image models has focused on creating pixel attribution maps , which assign a value to each pixel indicating how important that pixel was for a model ’ s prediction ( Selvaraju et al. , 2017 ; Sundararajan et al. , 2017 ) . These attribution maps can be noisy and often highlight seemingly unimportant pixels in the background . Such attributions can be difficult to understand , and may indicate the model is vulnerable to adversarial attacks ( Ross and Doshi-Velez , 2018 ) . Although we may desire a model with smoother attributions , existing methods only post-process attribution maps and do not change model behavior ( Smilkov et al. , 2017 ; Selvaraju et al. , 2017 ; Fong and Vedaldi , 2017 ) . Such techniques may not be faithful to the original model ( Ilyas et al. , 2019 ) . In this section , we describe how to apply our framework to train image models with naturally smoother attributions . To regularize pixel-level attributions , we use the following intuition : neighboring pixels should have a similar impact on an image model ’ s output . To encode this intuition , we apply a total variation loss on pixel-level attributions as follows : Ωpixel ( Φ ( θ , X ) ) = ∑ ` ∑ i , j |φ ` i+1 , j − φ ` i , j |+ |φ ` i , j+1 − φ ` i , j | , where φ ` i , j is the attribution for the i , j-th pixel in the ` -th training image . Including the λ scale factor , this penalty is equivalent to placing a Laplace ( 0 , λ−1 ) prior on the differences between adjacent pixel attributions . For further details , see Bardsley ( 2012 ) and the Appendix . 2.2 GRAPH ATTRIBUTION PRIOR FOR GENE EXPRESSION DATA . In the image domain , our attribution prior took the form of a penalty encouraging smoothness over adjacent pixels . In other domains , we may have prior information about specific relationships between features that can be encoded as an arbitrary graph ( such as social networks , knowledge graphs , or protein-protein interactions ) . For example , prior work in bioinformatics has shown that proteinprotein interaction networks contain valuable information that can be used to improve performance on biological prediction tasks ( Cheng et al. , 2014 ) . These networks can be represented as a weighted , undirected graph . Formally , say we have a weighted adjacency matrix W ∈ Rp×p+ for an undirected graph , where the entries encode our prior belief about the pairwise similarity of the importances between two features . For a biological network , Wi , j encodes either the probability or strength of interaction between the i-th and j-th genes ( or proteins ) . We can encourage similarity along graph edges by penalizing the squared Euclidean distance between each pair of feature attributions in proportion to how similar we believe them to be . Using the graph Laplacian ( LG = D −W ) , where D is the diagonal degree matrix of the weighted graph this becomes : Ωgraph ( Φ ( θ , X ) ) = ∑ i , j Wi , j ( φ̄i − φ̄j ) 2 = φ̄TLGφ̄ . In this case , we choose to penalize global rather than local feature attributions . So we define φ̄i to be the importance of feature i across all samples in our data set , where this global attribution is calculated as the average magnitude of the feature attribution across all samples : φ̄i = 1n ∑n ` =1 |φ ` i | . Overall , Ωgraph is equivalent to placing a Normal ( 0 , λ−1 ) prior on the differences between attributions for features that are adjacent in the graph . See Bardsley ( 2012 ) and the Appendix for details . 2.3 SPARSITY ATTRIBUTION PRIOR FOR FEATURE SELECTION . Feature selection and sparsity are popular ways to alleviate the curse of dimensionality , facilitate interpretability , and improve generalization by building models that use a small number of input features . A straightforward way to build a sparse deep model is to apply an L1 penalty to the first layer ( and possibly subsequent layers ) of the network . Similarly , the sparse group lasso ( SGL ) penalizes all weights connected to a given feature ( Feng and Simon , 2017 ; Scardapane et al. , 2017 ) , while Ross et al . ( 2017a ) penalize the gradients of each feature in the model . These approaches suffer from two problems : First , a feature with small gradients or first-layer weights may still strongly affect the model ’ s output ( Shrikumar et al. , 2017 ) . A feature whose attribution value ( e.g. , integrated or expected gradient ) is zero , is much less likely to have any effect on predictions . Second , successfully minimizing the L1 or SGL penalty is not necessarily the best way to create a sparse model . A model that puts weight w on 1 feature is penalized more than one that puts weight w 2p on each of p features . Prior work on sparse linear regression has shown that the Gini coefficient G of the weights , proportional to 0.5 minus the area under the CDF of sorted values , avoids such problems and corresponds more directly to a sparse model ( Hurley and Rickard , 2009 ; Zonoobi et al. , 2011 ) . We extend this analysis to deep models by noting that the Gini coefficient can be written differentiably and using it to develop an attribution penalty based on the global feature attributions φ̄i : Ωsparse ( Φ ( θ , X ) ) = −2G ( Φ ) = − ∑p i=1 ∑p j=1 |φ̄i − φ̄j | n ∑p i=1 φ̄i . This is similar to the total variation penalty Ωimage , but normalized and with a flipped sign to encourage differences . The corresponding attribution prior is maximized when global attributions are zero for all but one feature , and minimized when attributions are uniform across features .

Summary: The paper improves the existing feature attribution method by adding regularizers to enforce (human) expectations about a model’s behavior. Three different datasets (i.e. image, gene expression, health-care) are chosen to evaluate the proposed model’s effectiveness, while different regularizers (i.e. image prior, graph prior, and sparsity prior) are explored for the respective task.