Regularizing Black-box Models for Improved Interpretability

1 Introduction . Complex learning-based systems are increasingly shaping our daily lives , and , in order to monitor and understand these systems , we require clear explanations of model behavior . While model interpretability has many definitions and is often largely application specific ( Lipton , 2016 ) , local explanations are a popular and powerful tool ( Ribeiro et al. , 2016 ) . Recent work on local interpretability in machine learning ranges from proposals of new models that are interpretable by-design ( e.g. , Wang and Rudin , 2015 ; Caruana et al. , 2015 ) to model-agnostic post-hoc algorithms for interpreting complex , black-box predictors such as ensembles and deep neural networks ( e.g. , Ribeiro et al. , 2016 ; Lei et al. , 2016 ; Lundberg and Lee , 2017 ; Selvaraju et al. , 2017 ; Kim et al. , 2018 ) . Despite the variety of technical approaches , the underlying goal of all of these works is to develop an interpretable predictive system that produces two outputs : a prediction and its underlying explanation . Both interpretability by-design and post-hoc explanation strategies have limitations . On one hand , the by-design approaches are restricted to working with model families that provide inherent interpretability , potentially at the cost of accuracy . On the other hand , by performing two disjoint steps , there is no guarantee that post-hoc explainers applied to an arbitrary model will produce explanations of suitable quality . Moreover , recent approaches that claim to overcome this apparent trade-off between prediction accuracy and explanation quality are in fact by-design proposals that impose certain constraints on the underlying model families they consider ( e.g. , Al-Shedivat et al. , 2017 ; Plumb et al. , 2018 ; AlvarezMelis and Jaakkola , 2018a ) . In this work , we propose a novel alternative strategy called Explanation-based Optimization ( ExpO ) that aims to address both of these shortcomings by adding an interpretability regularizer to the loss function of an arbitrary predictive model . We illustrate how ExpO can influence the interpretability and accuracy of a model in Figure 1 ( left ) . Illustration . Consider a situation where Bob ’ s loan application is denied by a machine learning system ; see the toy illustration in Figure 1 ( right ) . Here , a good local explanation can help Bob understand how to improve his application in order to get the loan . Unfortunately , a standard model , a multi-layer perceptron trained with SGD , is not very interpretable because it has many abrupt changes . Indeed , we can quantitatively measure the local interpretability of the model using the standard fidelity ( Ribeiro et al. , 2016 ; Plumb et al. , 2018 ) and stability ( Alvarez-Melis and Jaakkola , 2018a ) explanation metrics . To make the learned model more amenable to local explanation , ExpO augments the objective function with fidelity- or stability-based regularizers , effectively controlling the degree of local explainability . The specific contributions of our work are as follows : 1 . Interpretability regularizers . We introduce two interpretability regularizers associated with the fidelity and stability explanation metrics . The first , ExpO-Fidelity , is designed for semantic features and explainers that directly make predictions , such as those in Ribeiro et al . ( 2016 ) ; Lundberg and Lee ( 2017 ) ; Plumb et al . ( 2018 ) . The second , ExpO-Stability , is tailored for non-semantic features ( e.g. , pixels ) and explainers that identify features that are influential on a prediction , such as saliency maps ( Simonyan et al. , 2013 ) . Both regularizers are differentiable and can be used to augment the objective function of an arbitrary model . In Section 3.1 , we discuss how they differ from the classical approaches for local approximation and function smoothing . 2 . Generalizable explanation quality . We analyze the properties of the explanation quality metrics and show that the benefits of our regularization generalize to unseen points . Specifically , we derive a bound on the gap between the fidelity of explanations on training and held out points and connect it with the local variance of the learned model . Empirical results . We evaluate models trained with and without our regularizers on a variety of regression and classification tasks with semantic and image features.1 Empirically , they slightly improve predictive performance across the nine datasets we consider ( seven UCI regression tasks , a medical classification task , and MNIST ) . From an interpretability perspective , our results demonstrate significant improvement in terms of explanation quality as measured by the fidelity and stability metrics . In particular , our regularizers improved explanation fidelity by at least 25 % on the UCI datasets and on the medical classification task while stability on MNIST was improved by orders of magnitude . Additionally , we found , qualitatively , that our regularizers produce models with simpler and more comprehensible explanations . In summary , black-box explanation systems work better on models trained with ExpO and , as an additional benefit , those models tend to be more accurate . 2 Background and Related Work . Consider a supervised learning problem , where our goal is to estimate a model , f : X 7→ Y , f ∈ F , that maps input feature vectors , x ∈ X , to targets , y ∈ Y , and is trained using data , { xi , yi } Ni=1 . If the class of functions used for modeling the data is complex , we can 1 The code for our regularizers and all experiments is at : https : //github.com/ForReview11235/CodeForICLR2020 understand the behavior of f in some neighborhood , Nx ∈ P [ X ] ( where P [ X ] is the space of probability distributions over X ) , by generating a local explanation . We denote algorithms that produce explanations ( i.e. , explainers ) as e : X × F 7→ E , where E is the set of possible explanations . The choice of E generally depends on whether or not X consists of semantic features , and will be defined more precisely next . 2.1 Semantic Features . We call features semantic if people can reason about them and understand what it means when their values change ( e.g. , a person ’ s income , the concentration of a chemical , etc. ) . Consequently , local explanations try to predict how the model ’ s output would change if the input was perturbed ( Ribeiro et al. , 2016 ; Lundberg and Lee , 2017 ; Plumb et al. , 2018 ) . Thus , we can define the output space of the explainer as Es : = { g ∈ G | g : X 7→ Y } , where G is a class of interpretable ( typically linear ) functions . Fidelity metric . When the explainer ’ s output space is Es , the explanation is defined as a function g : X 7→ Y , and it is natural to evaluate how accurately g models f in a neighborhood Nx ( Ribeiro et al. , 2016 ; Plumb et al. , 2018 ) : F ( f , g , Nx ) : = Ex′∼Nx [ ( g ( x′ ) − f ( x′ ) ) 2 ] , ( 1 ) which we refer to as the neighborhood-fidelity ( NF ) metric . This metric is sometimes evaluated with Nx as a point mass on x and we call this version the point-fidelity ( PF ) metric . While Plumb et al . ( 2018 ) argued that point-fidelity can be misleading because it does not measure generalization of e ( x , f ) across Nx , it has been used for evaluation in the prior work ( Ribeiro et al. , 2016 ; Lundberg and Lee , 2017 ; Ribeiro et al. , 2018 ) and we report it in our experiments along with the neighborhood-fidelity for completeness . Black-box explanation systems . Various explainers have been proposed to generate local explanations of the form g : X 7→ Y , typically assuming that g is linear . In particular , LIME ( Ribeiro et al. , 2016 ) , one of the most popular black-box explanation systems2 , solves the following optimization problem : e ( x , f ) : = arg min g∈Es F ( f , g , Nx ) + Ω ( g ) , ( 2 ) where Ω ( e ) stands for an additive regularizer that encourages certain desirable properties of the explanations ( e.g. , sparsity ) . LIME ’ s objective function is closely related to the fidelity metric and subsequently to our proposed ExpO-Fidelity regularizer . Consequently , we expect our regularizer to improve the quality of LIME-generated explanations . Our experimental results in Section 4.1 corroborate this hypothesis . Along with LIME , we consider another black-box explanation tool , called MAPLE ( Plumb et al. , 2018 ) . It differs substantially from LIME in that its neighborhood function is learned from the data rather than specified as a parameter . In our experiments , we evaluate the quality of MAPLE-generated local explanations for models regularized via ExpO-Fidelity , but do not use MAPLE ’ s learned neighborhood function to define ExpO-Fidelity . We view this as a good test case to see how optimizing the fidelity metric for one neighborhood generalizes to another one ( see Section 3 for a more detailed discussion of this point ) . In Section 4.1 , we see that regularizing for LIME neighborhoods improves MAPLE ’ s explanation quality . 2.2 Non-Semantic Features . Non-semantic features lack an inherent interpretation , with images being a canonical example . When X consists of non-semantic inputs , we can not assign meaning to the difference between x and x′ , hence it does not make sense to explain the difference between the predictions f ( x ) and f ( x′ ) . As a result , fidelity is not an appropriate explanation metric . Instead , in this 2SHAP ( Lundberg and Lee , 2017 ) is another popular method that proposes a theoreticallymotivated neighborhood sampling function , but requires explanations to be linear models that act on binary features . This requirement is too limiting in our case . context , local explanations try to identify which parts of the input are particularly influential on a prediction ( Sundararajan et al. , 2017 ) . Consequently , we consider explanations of the form Ens : = Rd , where d is the number of features in X . Stability metric and saliency maps . When the explainer ’ s output space is Ens , the explanation is a vector in Rd , and can not be directly compared to the underlying model itself , as in the case of the fidelity metric . Instead , the focus in this setting is on the degree to which the explanation changes between points in a local neighborhood , which we measure using the stability metric ( Alvarez-Melis and Jaakkola , 2018a ) : S ( f , e , Nx ) : = Ex′∼Nx [ ||e ( x , f ) − e ( x′ , f ) ||22 ] ( 3 ) Various explainers ( Sundararajan et al. , 2017 ; Zeiler and Fergus , 2014 ; Shrikumar et al. , 2016 ; Smilkov et al. , 2017 ; Adebayo et al. , 2018 ) have been proposed to generate local explanations in Ens , with saliency maps ( Simonyan et al. , 2013 ) being the approach that we consider in this work . Saliency maps assign importance weights to image pixels based on the magnitude of the gradient of the predicted class with respect to the corresponding pixels . Recent work on model interpretability emphasizes that more stable explanations tend to be more trustworthy ( Alvarez-Melis and Jaakkola , 2018a ; Ghorbani et al. , 2017 ; Alvarez-Melis and Jaakkola , 2018b ) . Note that the stability metric can also be considered in the context of semantic features in addition to the fidelity metric .

Summary: The paper considers the problem of training black box models for improved interpretability, and  proposes to penalize black-box models at training time using two regularizers that correspond to fidelity and stability explanation metrics. As computing the regularizers exactly is computationally intensive they propose two approximating algorithm. In addition, as the one for fidelity is still prohibitive, a randomized variant is proposed. Connections are established between the regularizers and the model's Lipschitz constant or total variation. A generalization bound is presented for local linear explanations. The proposed approach is evaluated on a variety of datasets.