RL-LIM: Reinforcement Learning-based Locally Interpretable Modeling

1 INTRODUCTION . Artificial Intelligence ( AI ) is advancing at a rapid pace , particularly with recent advances in deep neural networks and ensemble methods ( Goodfellow et al. , 2016 ; He et al. , 2016 ; Chen & Guestrin , 2016 ; Ke et al. , 2017 ) . This progress has been fueled by ‘ black-box ’ machine learning models where the decision making is controlled by complex non-linear interactions between many parameters that are difficult for humans to understand and interpret . However , in many real-world applications AI systems are not only expected to perform well but are also required to be interpretable : doctors need to understand why a particular treatment is recommended , and financial institutions need to understand why a loan was declined . Use cases of model interpretability vary across applications : it can provide trust to users by showing rationales behind decisions , enable detection of systematic failure cases , and provide actionable feedback for improving models ( Rudin , 2018 ) . Many studies have suggested a trade-off between performance and interpretability ( Virág & Nyitrai , 2014 ; Johansson et al. , 2011 ) . This is correct in that globally interpretable models , which attempt to explain the entire model behavior , typically yield considerably worse performance than ‘ blackbox ’ models ( Lipton , 2016 ) . To go beyond the performance limitations of globally interpretable models , another promising direction is locally interpretable models , which instead of explaining the entire model explain a single prediction ( Ribeiro et al. , 2016 ) . Methodologically , while a globally interpretable model fits a single inherently interpretable model ( such as a linear model or a shallow decision tree ) to the entire training set , locally interpretable models aim to fit an inherently interpretable model locally , i.e . for each instance individually , by distilling knowledge from a high performance black-box model . Such locally interpretable models are very useful for real-world AI deployments to provide succinct and human-like explanations to users . They can be used to identify systematic failure cases ( e.g . by seeking common trends in input dependence for failure cases ) , detect biases ( e.g . by quantifying feature importance for a particular variable ) , and provide actionable feedback to improve a model ( e.g . understand failure cases and what training data to collect ) . To be useful in practice , locally interpretable models need to maximize two objectives : ( i ) the overall prediction performance ( how well it predicts compared to the ground truth labels ) – for the model to be accurate , and ( ii ) fidelity ( how well it approximates the ‘ black-box ’ model predictions ) – to ensure the model is reliably approximating the black-box model ’ s predictions in the neighborhood of interest ( Plumb et al. , 2019 ; Lakkaraju et al. , 2019 ) . To this end , a few methods have recently been proposed for locally interpretable modeling : Local Interpretable Model-agnostic Explanations ( LIME ) ( Ribeiro et al. , 2016 ) , Supervised Local modeling methods ( SILO ) ( Bloniarz et al. , 2016 ) , and Model Agnostic Supervised Local Explanations ( MAPLE ) ( Plumb et al. , 2018 ) . LIME in particular has gained notable popularity and has been deployed in many applications due to its simplicity . However , the overall prediction performance and fidelity metrics are not reaching desired levels in many cases ( Alvarez-Melis & Jaakkola , 2018 ; Zhang et al. , 2019 ; Ribeiro et al. , 2018 ; Lakkaraju et al. , 2017 ) . Indeed , as we show in our experiments , there are frequent cases where existing locally interpretable models even underperform commonly low-performing globally interpretable models . One of the fundamental challenges to fit a locally interpretable model is the representational capacity difference while applying distillation . Black-box machine learning models , such as deep neural networks or ensemble models , have much larger representational capacity than locally interpretable models . This can result in underfitting with conventional distillation techniques , leading to suboptimal performance ( Hinton et al. , 2015 ; Wang et al. , 2019 ) . We address this fundamental challenge by proposing a novel Reinforcement Learning-based method to fit Locally Interpretable Models which we call RL-LIM . RL-LIM efficiently utilizes the small representational capacity of locally interpretable models by training with a small number of samples that are determined to have the highest value contribution to the fitting of a locally interpretable model . In order to select these highest-value instances , we train instance-wise weight estimators ( modeled with deep neural networks ) using a reinforcement signal that quantifies the fidelity metric ( i.e . how well does the model approximate the black-box model predictions ) . The contributions of this paper can be summarized as : 1 . We introduce the first method that tackles interpretability through data-weighted training , and show that reinforcement learning is highly effective for end-to-end training of such a model . 2 . We show that distillation of a black-box model into a low-capacity interpretable model can be significantly improved by fitting with a small subset of relevant samples that is controlled efficiently by our method . 3 . On various classification and regression datasets , we demonstrate that RL-LIM significantly outperforms alternative models ( LIME , SILO and MAPLE ) in overall prediction performance and fidelity metrics – in most cases , the overall performance of locally interpretable models obtained by RL-LIM is very similar to complex black-box models . 2 RELATED WORK . Locally interpretable models : There are various approaches to interpret black-box models – ( Gilpin et al. , 2018 ) provides a good overview . One approach is to directly decompose the prediction into feature attributions by considering what-if cases . Shapley values ( Štrumbelj & Kononenko , 2014 ) and their computationally-efficient variants ( Lundberg & Lee , 2017 ) are commonly-used methods in this category . Other notable methods are based on activation differences , e.g . DeepLIFT ( Shrikumar et al. , 2017 ) , or saliency maps using the gradient flows , e.g . CAM ( Zhou et al. , 2016 ) and Grad-CAM ( Selvaraju et al. , 2017 ) . In this paper , we focus on the direction of locally interpretable modeling – distilling a black-box model into an interpretable model for each input instance . Locally Interpretable Model-agnostic Explanation ( LIME ) ( Ribeiro et al. , 2016 ) is the most popular method for locally interpretable modeling . LIME is based on modifying a data instance by tweaking the feature values and then learning from the impact of the modifications on the output . A fundamental challenge for LIME is the need for a meaningful distance metric to determine neighborhoods , as simple metrics like Euclidean distance may yield poor fidelity in some cases and the estimation can be highly-sensitive to normalization ( Alvarez-Melis & Jaakkola , 2018 ) especially with categorical variables . Supervised Local modeling methods ( SILO ) ( Bloniarz et al. , 2016 ) ) aims to improve LIME by determining the neighborhoods for each instance using ad-hoc tree-based ensemble methods . Model Agnostic Supervised Local Explanations ( MAPLE ) ( Plumb et al. , 2018 ) furthers adds a method for feature selection on top of SILO – it utilizes ad-hoc tree-based ensemble methods to determine the weights of training instances for each target instance and uses the weights to optimize a locally interpretable model . However , SILO and MAPLE still have shortcomings because the tree-based ensemble methods are optimized independently from the locally interpretable model – lack of joint optimization results in suboptimal fidelity for the locally interpretable model . Overall , to construct a locally interpretable model , a key problem is how to select the optimal training instances for each testing instance , because the selected training instances mostly determine the constructed locally interpretable model . The number of possibilities for training instance selection is extremely large ( exponential in the number of training instances ) . LIME heuristically utilizes Euclidean distances , whereas SILO and MAPLE use ad-hoc tree-based ensemble methods . Our proposed method , RL-LIM , takes a very different perspective : to properly and efficiently explore the large possible solution space , RL-LIM utilizes reinforcement learning to find the optimal policy that selects the training instances that maximize the fidelity of the locally interpretable model . Data-weighted training : Optimal weighing of training data is a paramount problem in machine learning . By upweighting valuable instances and downweighting the low quality or problematic instances , better performance can be obtained in certain learning scenarios , such as imbalanced or noisy labels ( Jiang et al. , 2018 ) . One approach for data weighting is utilizing Influence Functions ( Koh & Liang , 2017 ) , that are based on oracle access to gradients and Hessian-vector products . Jointly-trained student-teacher methods constitute another approach ( Jiang et al. , 2018 ; Bengio et al. , 2009 ) to learn a data-driven curriculum . Using the feedback from the teacher network , training instance-wise weights are learned for the student model . Aligned with our motivations , meta learning is considered for data weighting in Ren et al . ( 2018 ) . Their proposed method utilizes gradient descent-based meta learning , guided by a small validation set , to maximize the target performance . In this work we consider data-weighted training for a novel purpose : interpretability . Unlike gradient descent-based meta learning , our approach uses reinforcement learning to integrate the reward directly with the fidelity metric . Aforementioned works estimate the same ranking of training instances for the entire dataset . Instead , our method yields an instance-wise ranking of training data points , different for each testing instance . This enables efficient distillation of a black-box model prediction into a locally interpretable model . 3 REINFORCEMENT LEARNING-BASED MODELING . We consider a training dataset D = { ( xi , yi ) , i = 1 , ... , N } ∼ P for training of a black-box model f , where xi ∈ X is the feature vector in a d-dimensional feature space X and yi ∈ Y is the corresponding label in a label space Y . We also assume that there exists a probe dataset Dp = { ( xpj , y p j ) , j = 1 , ... , M } ∼ P where M is the number of probe instances . The probe dataset is used to evaluate the model performance to guide meta-learning as in Ren et al . ( 2018 ) . If there is no explicit probe dataset , we can randomly partition a subset of the training dataset as the probe dataset and the remainder as the training dataset . RL-LIM is composed of three models : 1 . Black-box model f : X → Y – any machine learning model that needs to be explained ( e.g . a deep neural network or a decision tree-based ensemble model ) , 2 . Locally interpretable model gθ : X → Y – an inherently interpretable model by design ( e.g . a linear model or a shallow decision tree ) , 3 . Instance-wise weight estimation model hφ : X × X × Y → [ 0 , 1 ] – a function that outputs the instance-wise weights to fit the locally interpretable model . It uses concatenation of a probe feature , a training feature , and a corresponding black-box model prediction on the training feature as its inputs . It can be a complex machine learning model – e.g . here a deep neural network . Our objective is to construct an accurate locally interpretable model gθ such that the predictions made by it are similar to the predictions of the given black-box model f∗ – i.e . the locally interpretable model has high fidelity . We use a loss function , L : Y × Y → R to quantify the fidelity of the locally interpretable model ( e.g . mean absolute error , lower the better ) . In RL-LIM , the three necessary components of an RL framework are as follows : the state is the vector of input features , the action is the selection vector , and the reward is the fidelity which depends on the input features and the selection vector . The instance-wise weight estimator model is the agent that outputs the actions based on the state ( input features ) . The environment is comprised of the input feature generating process , as well as the black-box model for the target task . The representational capacity difference between the black-box model and the locally interpretable model is the bottleneck we aim to address . Ideally , to avoid underfitting , locally interpretable models should be learned with a minimal number of training instances that are most effective in capturing the model behavior . We propose an instance-wise weight estimation model hφ to estimate the probabil- ities of training instances that should be used for fitting the locally interpretable model . Integrating with the accurate locally interpretable modeling goal , we propose the following objective : min hφ Exp∼PX [ L ( f∗ ( xp ) , g∗θ ( xp ) ( x p ) ) ] + λExp , x∼PX [ hφ ( xp , x , f∗ ( x ) ) ] s.t . g∗θ ( xp ) = argmingθ Ex∼PX [ hφ ( xp , x , f∗ ( x ) ) × Lg ( f∗ ( x ) , gθ ( x ) ) ] ( 1 ) where λ ≥ 0 is a hyper-parameter that controls the number of training instances used to fit the locally interpretable model ( we study the impact of performance on λ in Section 4.2 ) , and hφ ( xp , x , f∗ ( x ) ) represents the instance-wise weight for each training pair ( x , f∗ ( x ) ) for the probe data xp . Lg is the loss function to fit the locally interpretable model , for which we use the mean squared error between predicted values for regression and logits for classification . φ and θ are the trainable parameters , whereas f∗ ( the pre-trained black-box model ) is fixed . The first term in the objective function Exp∼PX [ L ( f∗ ( xp ) , g∗θ ( xp ) ( x p ) ) ] represents the local prediction differences between black-box model and locally interpretable model ( referred to as fidelity metric ) . The second term in the objective function Exp , x∼PX [ hφ ( xp , x , f∗ ( x ) ) ] represents the expected number of selected training points to fit the locally interpretable model . Lastly , the constraint ensures that the locally interpretable model is derived from weighted loss function , where weights are the output of the instance-wise weight estimator hφ . Our formulation does not assume any constraint on gθ – it could be any inherently interpretable model suitable for the data type of interest . Next , we describe how Eq . ( 1 ) can be efficiently addressed with reinforcement learning .

Summary: This paper studies the problem of interpreting predictions of blackbox models. In particular, they study local interpretable models, which are used to study interpretability at the level of one or a few data points. A key challenge is that in order for local models to be interpretable, they need to be simple in form and therefore lower in capacity (i.e. linear); thus, if they are trained on entire datasets they will underfit.