Title: RL-LIM: REINFORCEMENT LEARNING-BASED LOCALLY INTERPRETABLE MODELING

Abstract: Understanding black-box machine learning models is important towards their widespread adoption. However, developing globally interpretable models that explain the behavior of the entire model is challenging. An alternative approach is to explain black-box models through explaining individual prediction using a locally interpretable model. In this paper, we propose a novel method for locally interpretable modeling -Reinforcement Learning-based Locally Interpretable Modeling (RL-LIM). RL-LIM employs reinforcement learning to select a small number of samples and distill the black-box model prediction into a low-capacity locally interpretable model. Training is guided with a reward that is obtained directly by measuring agreement of the predictions from the locally interpretable model with the black-box model. RL-LIM near-matches the overall prediction performance of black-box models while yielding human-like interpretability, and significantly outperforms state of the art locally interpretable models in terms of overall prediction performance and fidelity.

1. INTRODUCTION:

Artificial Intelligence (AI) is advancing at a rapid pace, particularly with recent advances in deep neural networks and ensemble methods (Goodfellow et al., 2016; He et al., 2016; Chen & Guestrin, 2016; Ke et al., 2017) . This progress has been fueled by 'black-box' machine learning models where the decision making is controlled by complex non-linear interactions between many parameters that are difficult for humans to understand and interpret. However, in many real-world applications AI systems are not only expected to perform well but are also required to be interpretable: doctors need to understand why a particular treatment is recommended, and financial institutions need to understand why a loan was declined. Use cases of model interpretability vary across applications: it can provide trust to users by showing rationales behind decisions, enable detection of systematic failure cases, and provide actionable feedback for improving models (Rudin, 2018) .

Many studies have suggested a trade-off between performance and interpretability (Virág & Nyitrai, 2014; Johansson et al., 2011) . This is correct in that globally interpretable models, which attempt to explain the entire model behavior, typically yield considerably worse performance than 'blackbox' models (Lipton, 2016) . To go beyond the performance limitations of globally interpretable models, another promising direction is locally interpretable models, which instead of explaining the entire model explain a single prediction (Ribeiro et al., 2016) . Methodologically, while a globally interpretable model fits a single inherently interpretable model (such as a linear model or a shallow decision tree) to the entire training set, locally interpretable models aim to fit an inherently interpretable model locally, i.e. for each instance individually, by distilling knowledge from a high performance black-box model. Such locally interpretable models are very useful for real-world AI deployments to provide succinct and human-like explanations to users. They can be used to identify systematic failure cases (e.g. by seeking common trends in input dependence for failure cases), detect biases (e.g. by quantifying feature importance for a particular variable), and provide actionable feedback to improve a model (e.g. understand failure cases and what training data to collect).

To be useful in practice, locally interpretable models need to maximize two objectives: (i) the overall prediction performance (how well it predicts compared to the ground truth labels) -for the model to be accurate, and (ii) fidelity (how well it approximates the 'black-box' model predictions) -to ensure the model is reliably approximating the black-box model's predictions in the neighborhood of interest (Plumb et al., 2019; Lakkaraju et al., 2019) . To this end, a few methods have recently been proposed for locally interpretable modeling: Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016) , Supervised Local modeling methods (SILO) (Bloniarz et al., 2016) , and Model Agnostic Supervised Local Explanations (MAPLE) (Plumb et al., 2018) . LIME in particular has gained notable popularity and has been deployed in many applications due to its simplicity. However, the overall prediction performance and fidelity metrics are not reaching desired levels in many cases (Alvarez-Melis & Jaakkola, 2018; Zhang et al., 2019; Ribeiro et al., 2018; Lakkaraju et al., 2017) . Indeed, as we show in our experiments, there are frequent cases where existing locally interpretable models even underperform commonly low-performing globally interpretable models.

One of the fundamental challenges to fit a locally interpretable model is the representational capacity difference while applying distillation. Black-box machine learning models, such as deep neural networks or ensemble models, have much larger representational capacity than locally interpretable models. This can result in underfitting with conventional distillation techniques, leading to suboptimal performance (Hinton et al., 2015; Wang et al., 2019) . We address this fundamental challenge by proposing a novel Reinforcement Learning-based method to fit Locally Interpretable Models which we call RL-LIM. RL-LIM efficiently utilizes the small representational capacity of locally interpretable models by training with a small number of samples that are determined to have the highest value contribution to the fitting of a locally interpretable model. In order to select these highest-value instances, we train instance-wise weight estimators (modeled with deep neural networks) using a reinforcement signal that quantifies the fidelity metric (i.e. how well does the model approximate the black-box model predictions). The contributions of this paper can be summarized as:

1. We introduce the first method that tackles interpretability through data-weighted training, and show that reinforcement learning is highly effective for end-to-end training of such a model. 2. We show that distillation of a black-box model into a low-capacity interpretable model can be significantly improved by fitting with a small subset of relevant samples that is controlled efficiently by our method. 3. On various classification and regression datasets, we demonstrate that RL-LIM significantly outperforms alternative models (LIME, SILO and MAPLE) in overall prediction performance and fidelity metrics -in most cases, the overall performance of locally interpretable models obtained by RL-LIM is very similar to complex black-box models.

2. RELATED WORK:

Locally interpretable models: There are various approaches to interpret black-box models - (Gilpin et al., 2018) provides a good overview. One approach is to directly decompose the prediction into feature attributions by considering what-if cases. Shapley values ( Štrumbelj & Kononenko, 2014) and their computationally-efficient variants (Lundberg & Lee, 2017) are commonly-used methods in this category. Other notable methods are based on activation differences, e.g. DeepLIFT (Shrikumar et al., 2017) , or saliency maps using the gradient flows, e.g. CAM (Zhou et al., 2016) and Grad-CAM (Selvaraju et al., 2017) . In this paper, we focus on the direction of locally interpretable modeling -distilling a black-box model into an interpretable model for each input instance.

Locally Interpretable Model-agnostic Explanation (LIME) (Ribeiro et al., 2016) is the most popular method for locally interpretable modeling. LIME is based on modifying a data instance by tweaking the feature values and then learning from the impact of the modifications on the output. A fundamental challenge for LIME is the need for a meaningful distance metric to determine neighborhoods, as simple metrics like Euclidean distance may yield poor fidelity in some cases and the estimation can be highly-sensitive to normalization (Alvarez-Melis & Jaakkola, 2018) especially with categorical variables. Supervised Local modeling methods (SILO) (Bloniarz et al., 2016) ) aims to improve LIME by determining the neighborhoods for each instance using ad-hoc tree-based ensemble methods. Model Agnostic Supervised Local Explanations (MAPLE) (Plumb et al., 2018) furthers adds a method for feature selection on top of SILO -it utilizes ad-hoc tree-based ensemble methods to determine the weights of training instances for each target instance and uses the weights to optimize a locally interpretable model. (Jiang et al., 2018) . One approach for data weighting is utilizing Influence Functions (Koh & Liang, 2017) , that are based on oracle access to gradients and Hessian-vector products. Jointly-trained student-teacher methods constitute another approach (Jiang et al., 2018; Bengio et al., 2009) to learn a data-driven curriculum. Using the feedback from the teacher network, training instance-wise weights are learned for the student model. Aligned with our motivations, meta learning is considered for data weighting in Ren et al. (2018) . Their proposed method utilizes gradient descent-based meta learning, guided by a small validation set, to maximize the target performance.

In this work we consider data-weighted training for a novel purpose: interpretability. Unlike gradient descent-based meta learning, our approach uses reinforcement learning to integrate the reward directly with the fidelity metric. Aforementioned works estimate the same ranking of training instances for the entire dataset. Instead, our method yields an instance-wise ranking of training data points, different for each testing instance. This enables efficient distillation of a black-box model prediction into a locally interpretable model.

3. REINFORCEMENT LEARNING-BASED MODELING:

We consider a training dataset D = {(x i , y i ), i = 1, ..., N } ∼ P for training of a black-box model f , where x i ∈ X is the feature vector in a d-dimensional feature space X and y i ∈ Y is the corresponding label in a label space Y. We also assume that there exists a probe dataset D p = {(x p j , y p j ), j = 1, ..., M } ∼ P where M is the number of probe instances. The probe dataset is used to evaluate the model performance to guide meta-learning as in Ren et al. (2018) . If there is no explicit probe dataset, we can randomly partition a subset of the training dataset as the probe dataset and the remainder as the training dataset. RL-LIM is composed of three models: Our objective is to construct an accurate locally interpretable model g θ such that the predictions made by it are similar to the predictions of the given black-box model f * -i.e. the locally interpretable model has high fidelity. We use a loss function, L : Y × Y → R to quantify the fidelity of the locally interpretable model (e.g. mean absolute error, lower the better). In RL-LIM, the three necessary components of an RL framework are as follows: the state is the vector of input features, the action is the selection vector, and the reward is the fidelity which depends on the input features and the selection vector. The instance-wise weight estimator model is the agent that outputs the actions based on the state (input features). The environment is comprised of the input feature generating process, as well as the black-box model for the target task.

1. Black-box model f : X → Y -

The representational capacity difference between the black-box model and the locally interpretable model is the bottleneck we aim to address. Ideally, to avoid underfitting, locally interpretable models should be learned with a minimal number of training instances that are most effective in capturing the model behavior. We propose an instance-wise weight estimation model h φ to estimate the probabil-ities of training instances that should be used for fitting the locally interpretable model. Integrating with the accurate locally interpretable modeling goal, we propose the following objective:

min h φ E x p ∼P X L(f * (x p ), g * θ(x p ) (x p )) + λE x p ,x∼P X h φ (x p , x, f * (x)) s.t. g * θ(x p ) = arg min g θ E x∼P X h φ (x p , x, f * (x)) × L g (f * (x), g θ (x)) (1)

where λ ≥ 0 is a hyper-parameter that controls the number of training instances used to fit the locally interpretable model (we study the impact of performance on λ in Section 4.2), and h φ (x p , x, f * (x)) represents the instance-wise weight for each training pair (x, f * (x)) for the probe data x p . L g is the loss function to fit the locally interpretable model, for which we use the mean squared error between predicted values for regression and logits for classification. φ and θ are the trainable parameters, whereas f * (the pre-trained black-box model) is fixed.

The first term in the objective function

E x p ∼P X L(f * (x p ), g * θ(x p ) (x p ))

represents the local prediction differences between black-box model and locally interpretable model (referred to as fidelity metric). The second term in the objective function E x p ,x∼P X h φ (x p , x, f * (x)) represents the expected number of selected training points to fit the locally interpretable model. Lastly, the constraint ensures that the locally interpretable model is derived from weighted loss function, where weights are the output of the instance-wise weight estimator h φ . Our formulation does not assume any constraint on g θ -it could be any inherently interpretable model suitable for the data type of interest. Next, we describe how Eq. ( 1) can be efficiently addressed with reinforcement learning.

3.1. TRAINING AND INFERENCE:

The RL-LIM method, shown in Fig. 1 , can be thought of as encompassing 5 stages: 

= arg min f 1 N N i=1 L f (f (x i ), y i ).

If the pre-trained black-box model is already saved, we can skip this stage and retrieve the given pre-trained black-box model to f * .

• Stage 1 -Auxiliary dataset construction: Using the pre-trained black-box model f * , we create auxiliary training and probe datasets, as D = {(x i , ŷi ), i = 1, ..., N } (where ŷi = f * (x i )) and Dp = {(x p j , ŷp j ), j = 1, ..., M } (where ŷp j = f * (x p j )), respectively. These auxiliary datasets ( D, Dp ) are used for instance-wise weight estimation models and locally interpretable model training. To encourage exploration, we consider probabilistic selection, with a sampler block that is based on the output of the instance-wise weight estimatorh φ (x p j , x i , ŷi ) represents the probability that (x i , ŷi ) is selected to train locally interpretable model for the probe instance x p j . Let the binary vector c(x p j ) ∈ {0, 1} N represent the selection operation, such that (x i , ŷi ) is selected for training locally interpretable model for x p j when c i (x p j ) = 1. Correspondingly, ρ φ (x p ) is the probability mass function for c(x p j ) given h φ (•):

ρ φ (x p j , c(x p j )) = N i=1 h φ (x p j , x i , f * (x i )) ci(x p j ) • (1 -h φ (x p j , x i , f * (x i ))) 1-ci(x p j )

As the original form of the optimization problem in Eq. ( 1) is intractable due to the expectation operations, we employ approximations:

-The sample mean is used as an approximation of the first term of the objective function as

1 M M j=1 L(f * (x p j ), g * θ(x p j ) (x p j ))).

-The second term of the objective, which represents the average selection probability, is approximated as the number of selected instances (divided by N ) to have ||c(

x p j )|| 1 = 1 N N i=1 |c i (x p j )|.

-The constraint term is approximated using the sample mean of the training loss as g

* θ(x p j ) = arg min g θ 1 N N i=1 c i (x p j ) • L g (f * (x i ), g θ (x i ))

. The sampler block yields a non-differential objective, and we cannot train the instance-wise weight estimator using conventional gradient descent-based optimization. There are approximations such as training in expectation (Raffel et al., 2017) or Gumbel-softmax (Jang et al., 2016) . Instead, motivated by its many successful applications (Ranzato et al., 2015; Zaremba & Sutskever, 2015; Zhang & Lapata, 2017) , we use REINFORCE algorithm (Williams, 1992) such that the selection action is rewarded by the performance of its impact. The loss function for the instance-wise weight estimator l(φ) is expressed as:

l(φ) = E x p j ∼P X E c(x p j )∼ρ φ (x p j ,•) L(f * (x p j ), g * θ(x p j ) (x p j ))) + λ||c(x p j )|| 1

To apply the REINFORCE algorithm, we directly compute the gradient ∇ φ l(φ) as:

∇ φ l(φ) = E x p j ∼P X E c(x p j )∼ρ φ (x p j ,•) L(f * (x p j ), g * θ(x p j ) (x p j ))) + λ||c(x p j )|| 1 ∇ φ log ρ φ (x p j , c(x p j ))

Using the gradient ∇ φ l(φ), we employ the following steps iteratively to update the parameters of the instance-wise weight estimator φ:

1. Estimate instance-wise weights w i (x p j ) = h φ (x p j , x i , ŷi ) and instance-wise selection vector c i (x p j ) ∼ Ber(w i (x p j )) for each training and probe instance in a mini-batch.

2. Optimize the locally interpretable model with the selection vector for each probe instance:

g * θ(x p j ) = arg min g θ N i=1 c i (x p j ) • L g (f * (x i ), g θ (x i ))

3. Update the instance-wise weight estimation model parameter φ:

φ ← φ - α M M j=1 L(f * (x p j ), g * θ(x p j ) (x p j )) -L b (x p j ) + λ||c(x p j )|| 1 • ∇ φ log ρ φ (x p j , c(x p j ))

where α > 0 is a learning rate and L b (x p j ) = L(f * (x p j ), g * b (x p j )) is the baseline loss against which we benchmark the performance improvement. We repeat the steps above until convergence.

• Stage 4 -Interpretable inference: Unlike when training, we use a fixed instance-wise weight estimator (without the sampler and interpretable baseline) and merely fit the locally interpretable model at inference. Given the test instance x t , we obtain the selection probabilities from the instance-wise weight estimator, and using these as the weights, we fit the locally interpretable model via weighted optimization. The outputs of the trained interpretable model are the instancewise predictions and the corresponding explanations (e.g., local dynamics of the black-box model predictions at x t given by the coefficients of the fitted linear model).

3.2. COMPUTATIONAL COST:

In Interpretable inference: To infer with the locally interpretable model, we need to fit the locally interpretable model after obtaining the instance-wise weights from the trained instance-wise weight estimator. Thus, for each testing instance, the computational complexity is O(d 2 N ).1 

For instance, on a single NVIDIA V100 GPU, on Facebook Comment dataset (consisting ∼ 600,000 samples), RL-LIM yields a training time of less than 5 hours (including Stage 1, 2 and 3) and an interpretable inference time of less than 10 seconds per a testing instance. On the other hand, LIME results in much longer interpretable inference time (around 30 seconds per a testing instance) due to acquiring a large number of black-box model predictions for the inputs perturbations, whereas SILO and MAPLE are similar to RL-LIM.

4. EXPERIMENTS:

We compare RL-LIM to multiple benchmarks on 3 synthetic datasets and 5 UCI public datasets.

Datasets: The 3 public datasets for regression problems are: (1) Blog Feedback, (2) Facebook Comment, (3) News Popularity; the other 2 public datasets for classification problems are: (4) Adult Income, (5) Weather. Details of the data descriptions can be found in the hyper-links of each dataset (colored in blue). Data statistics can be found in Table 3 in Appendix A. In this section, we mainly focus on the tabular datasets because the local dynamics are more important and useful to explain for them; however, RL-LIM method can be generalized to other data types in a straightforward way.

Black-box models::

We focus on approximating black-box models that are shown to yield competitive performance on the target tasks: 3 tree-based ensemble methods (1) XGBoost (Chen & Guestrin, 2016) , (2) LightGBM (Ke et al., 2017) , (3) Random Forests (RF) (Breiman, 2001) ; and deep neural networks (4) Multi-layer Perceptron (MLP). Also, we use (5) Ridge Regression (RR) and ( 6) Regression Tree (RT) (for regression) and ( 7) Logistic Regression (LR) and (8) Decision Tree (DT) (for classification) as globally interpretable models to benchmark. 2 We focus on two types of locally interpretable models: (1) Ridge regression, (2) Shallow regression tree (with a max depth of 3). We report the performance with ridge regression for regression and with shallow regression tree for classification in this section. The results of the other two combinations (with ridge regression for classification and with shallow regression tree for regression) are described in Appendix E.

Comparisons (Plumb et al., 2018) .

Performance metrics: To evaluate the performance of locally interpretable models using real-world datasets, we quantify the overall prediction performance and its fidelity. We assume a disjoint testing dataset D t = {(x t k , y t k )} L k=1 for evaluation. For the overall prediction performance, we compare the predictions of the locally interpretable models with the ground-truth labels. We use Mean Absolute Error (MAE) for regression and Average Precision Recall (APR) for classification. For fidelity, we compare the outputs (predicted values for regression and logits for classification) of the locally interpretable models and of the black-box model. We consider two metrics: R 2 score (Legates & McCabe, 1999) and Local MAE (LMAE). The details of the metrics are described in Appendix C.

Implementation details::

We implement instance-wise weight estimator using a multi-layer perceptron with tanh activation. The number of hidden units and layers are optimized by the crossvalidation. In most cases, 5-layer perceptron with 100 hidden units performs reasonably-well across all datasets. All features are normalized to be between zero and one, using standard minmax scaler. Categorical variables are transformed using one-hot encoding.

4.1. EXPERIMENTS ON SYNTHETIC DATASETS -RECOVERING LOCAL DYNAMICS:

On real-world datasets it is challenging to directly evaluate the explanation quality of the locally interpretable models due to the absence of ground-truth explanations. Thus we initially focus on synthetic datasets (with known ground-truth explanations) to directly evaluate how well the locally interpretable models can recover the underlying local dynamics. We construct three synthetic datasets such that the 11-dimensional input features X are sampled from N (0, I) and Y are:

1. Syn1: Y = X 1 + 2X 2 if X 10 < 0 and Y = X 3 + 2X 4 if X 10 ≥ 0 2. Syn2: Y = X 1 + 2X 2 if X 10 + e X11 < 1 and Y = X 3 + 2X 4 if X 10 + e X11 ≥ 1 3. Syn3: Y = X 1 + 2X 2 if X 10 + X 3 11 < 0 and Y = X 3 + 2X 4 if X 10 + X 3 11 ≥ 0

All three datasets have different local dynamics in different input regimes. We directly use the ground truth function as the black-box model and focus on how well locally interpretable modeling can capture the local dynamics. We evaluate the performance of capturing local dynamics using Absolute Weight Difference (AWD): AWD = ||w -ŵ||, where w is the ground truth coefficients to generate Y and ŵ is the derived coefficient from the locally interpretable models. We use the estimated coefficients of the ridge regression as the derived local dynamics ( ŵ).

As shown in Fig. 2 , RL-LIM significantly outperforms other benchmarks in discovering the local dynamics on all three datasets and in different regimes. RL-LIM can actively learn the linear and non-linear decision boundaries for the local dynamics. Note that LIME completely fails to recover the local dynamics as it uses the Euclidean distance uniformly for all features and cannot distinguish the special properties of the features that alter the local dynamics. SILO and MAPLE only use the predictions to discover the local dynamics; thus, it is hard to discover the decision boundary that depends on the other variables which are independent to the predictions. We exclude LIME in these graphs due to its poor performance in terms of AWD (it is higher than 1.6 in all distance regimes for all three synthetic datasets).

4.2. THE EFFECT OF THE NUMBER OF SELECTED SAMPLES ON FIDELITY:

In RL-LIM, optimal distillation is enabled by using a small subset of training instances to fit the low-capacity locally interpretable model. The number of selected instances is controlled by λ in our method -if λ is high/low, RL-LIM penalizes more/less on the number of selected instances; thus, less/more instances are selected to construct the locally interpretable model. We analyze the efficacy of λ in controlling the likelihood of selection and the dependency of fidelity on λ. We expect that if we select a too small/large number of training instances, the locally interpretable model will overfit/underfit which negatively affects the fidelity in both cases. Fig. 3 shows that there is a clear relationship between λ and the local fidelity. If λ is too large, RL-LIM selects too small number of instances; thus, the fitted locally interpretable model is less accurate (due to overfitting). On the other hand, if λ is too small, RL-LIM selects too large number of instances and deteriorates fidelity (due to underfitting). To achieve the optimal λ, we conduct cross-validation experiments and select λ which achieves the best validation fidelity (e.g. λ = 0.5 in Syn2). Fig. 3 shows the average selection probability of the training instances for each λ. As λ increases, the average selection probabilities monotonically decrease due to the higher penalty on the number of selected training instances. Note that even using a small portion of training instances, RL-LIM can accurately distill the predictions of black-box models into locally interpretable models which is crucial to understand and interpret the predictions using the most relevant training instances.

4.3. EXPERIMENTS ON REAL DATASETS -OVERALL PERFORMANCE AND FIDELITY:

On multiple real datasets, we evaluate the overall prediction performance and fidelity. For the regression and classification problems, we use ridge regression and shallow regression trees as the locally interpretable model. More results can be found in Appendix E.

As can be seen in Table 1 , the performance of globally interpretable ridge regression (trained on the entire dataset from the scratch) is much worse than other complex non-linear models, implying that modeling non-linear relationships between the features and the labels is important towards high prediction performance. For other locally interpretable modeling methods (LIME, SILO, MAPLE), the performance is far worse than the original black-box model, showing that they fail at efficiently distilling the non-linear black-box models. In some cases (especially on the Facebook dataset), the performance of the benchmarks is even worse than the performance of global ridge regression (highlighted in red), questioning the value of using these locally interpretable models instead of globally interpretable ridge regression.

Datasets Models XGBoost LightGBM MLP RF (RR-MAE) Metrics MAE R 2 MAE R 2 MAE R 2 MAE R 2 Original 5

In contrast, RL-LIM achieves similar overall prediction performance to the black-box models and significantly outperforms global ridge regression. Table 1 also compares the fidelity in terms of R 2 score for regression using ridge regression as the locally interpretable model (LMAE results can be found in Appendix E.3). We observe that R 2 scores for some cases (especially on Facebook dataset and LIME) are negative which represent that the outputs of the locally interpretable models are even worse than the constant mean value estimator. On the other hand, RL-LIM achieves higher and positive R 2 values consistently for all datasets and black-box models than other benchmarks.

Table 2 shows a similar analysis for classification using shallow regression trees (with max depth of 3) as the locally interpretable model (Regression trees are used to model logit outputs for classification.). The overall prediction performance of four black-box models are significantly better than the globally interpretable decision tree which demonstrates the superior fitting by complex black-box models. Among the locally interpretable models, RL-LIM achieves the best APR and R 2 score for most cases, underlining its strength in distilling the predictions of the black-box model accurately.

In some cases, the benchmarks (especially for LIME) achieve lower overall prediction performance than the globally interpretable decision tree (highlighted in red). The overall prediction performance and fidelity metrics of all locally interpretable models seem better for classification problems than regression problems. We expect that the predictions of black-box models are mostly highly confident, i.e. located near 0 or 1; thus, locally interpretable models can easily distill the predictions of the black-box models for classification than regression.

4. .6991 .9680 .7052 .9452 .6997 .8864 .7042 .8398 MAPLE .6973 .9675 .7056 .9446 .6983 .8856 .6983 .8856 Table 2 : Real-world classification dataset results. Overall prediction performance (metric: APR, higher is better) and fidelity (metric: R 2 score, higher is better) on classification problems with shallow regression tree as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. APR of global decision tree (DT) can be found below the data name. five subgroups in predicting the annual income using XGBoost as the black-box model. We use ridge regression as the locally interpretable model and the absolute value of fitted coefficients as the estimated feature importance. As can be observed in Fig. 4 , for age subgroups, capital gain seems much more important for mature people (older than 25) than young people (younger than 25). For education subgroups, capital gain/loss, occupation, and native countries are more critical for highly-educated people (Doctorate, Prof-school, and Masters graduates) than the others. We do not discover notable biases of black-box models for gender, marital status, and race subgroups.

5. CONCLUSIONS:

We propose a novel method for locally interpretable modeling of pre-trained black-box models. Our proposed method employs reinforcement learning to select a small number of valuable instances and use them to train a low-capacity locally interpretable model. The selection mechanism is guided with a reward obtained from the similarity of predictions of the locally interpretable model and the blackbox model. Our approach near-matches the performance of black-box models and significantly outperforms alternative techniques in terms of overall prediction performance and fidelity metrics consistently across various datasets and black-box models.

C PERFORMANCE METRICS:

• Mean Absolute Error (MAE):

MAE = E (x t ,y t )∼P ||g * θ(x t ) (x t ) -y t )|| 1 1 L L k=1 ||g * θ(x t k ) (x t k ) -y t k || 1 , • Local MAE (LMAE): LMAE = E x t ∼P X ||g * θ(x t ) (x t ) -f * (x t )|| 1 1 L L k=1 ||g * θ(x t k ) (x t k ) -f * (x t k ))|| 1 , • R 2 score (Legates & McCabe, 1999): R 2 = 1- E x t ∼P X ||f * (x t ) -g * θ(x t ) (x t )|| 2 2 E x t ∼P X ||f * (x t ) -E xt ∼P X [f * (x t )]|| 2 2 1- 1 L L k=1 ||f * (x t k ) -g * θ(x t k ) (x t k )|| 2 2 1 L L k=1 ||f * (x t k ) -1 L L k=1 [f * (x t k )]|| 2 2 .

If R 2 = 1, the predictions of the locally interpretable model perfectly match the predictions of the black-box model. On the other hand, if R 2 = 0, the locally interpretable model performs as similar as the constant mean value estimator. If R 2 < 0, the locally interpretable model performs worse than the constant mean value estimator. As can be seen in Fig. 7 (a)-(c), the instance-wise weights have quite skewed distribution. Some samples (e.g. with average instance-wise weights above 0.5) are much more critical to interpreting the probe sample than many others (e.g. average instance-wise weights below 0.1)

D LEARNING CURVES OF RL-LIM:

In Fig. 7 (d) , there is a clear trend that the samples near the probe sample have higher average instance-wise weights, which shows that RL-LIM learns the meaningful distance metrics to measure the relevance while interpreting the probe samples. This trend is consistent across all three synthetic datasets.

:

A subset of the training dataset can be used to reduce complexity (with decreased fidelity).

We use python packages (including Sklearn and Tensorflow) to implement those predictive models and the details can be found in the hyper-links (colored in blue) of each model and Appendix B.

Figures and Tables:

Stage 2 -Interpretable baseline training: To improve the stability of the instance-wise weight estimator training, a baseline model is observed to be beneficial. As the baseline model g b : X → Y, we use a globally interpretable model (such as a linear model or shallow decision tree) optimized to replicate the predictions of the black-box model: g * b = arg min g 1 N N i=1 L(g(x i ), ŷi ). • Stage 3 -Instance-wise weight estimator training: We train an instance-wise weight estimator using the auxiliary datasets ( D, Dp ).

this subsection, we analyze the computational cost of RL-LIM for training and inference. As a representative and commonly used example, we assume linear regression as the locally interpretable model, which has a computational complexity of O(d 2 N ) + O(d 3 ) to fit, where d is the number of features and N is the number of training instances. When N >> d (which is often the case in practice), the training computational complexity is approximated as O(d 2 N )(Tan, 2018).Training: Given a pre-trained black-box model, Stage 1 involves running inference N times and the total complexity depends on the complexity of the black-box model. Unless the black-box model is very complex, the computational complexity of Stage 1 becomes much smaller than Stage 3. Stage 2 has negligible computational overhead. At Stage 3, we iteratively train the instance-wise weight estimator and fit the locally interpretable model from scratch using weighted optimization. Therefore, the computational complexity is O(d 2 N N I ) where N I is the number of iterations in Stage 3 (typically N I < 10, 000 until convergence). Thus, the training complexity scales roughly linearly with the number of training instances.

Fig. 5 in Appendix D shows the learning curves of RL-LIM demonstrating the efficiency of reinforcement learning.

Figure2: Synthetic dataset results. Mean absolute weight difference (AWD) with 95% confidence intervals (of 10 independent runs) on three synthetic datasets. X-axis: Distance from the boundary where the local dynamics change, such as X 10 = 0 for Syn1 (in percentile), Y-axis: AWD (the lower, the better). We exclude LIME in these graphs due to its poor performance in terms of AWD (it is higher than 1.6 in all distance regimes for all three synthetic datasets).

Figure 3: Fidelity & average selection probability of training instances as a function of the number of selected samples on three synthetic datasets. X-axis: λ, Y-axis: LMAE and average selection probability of training instances. LMAE is Local MAE -lower is better.

Figure 4: Qualitative interpretability results. The analyses of feature importance (derived by RL-LIM) for 5 types of subgroups in Adult Income dataset: (a) Age, (b) Gender, (c) Marital status, (d) Race, (e) Education. The color represents the feature importance for each subgroup.

Figure 5: Learning curves of RL-LIM on three synthetic datasets. X-axis: The number of iterations on instance-wise weight estimator training, Y-axis: Rewards (LMAE of baseline (globally interpretable model) -LMAE of RL-LIM), higher the better.

Figure 7: (a) Instance-wise weight distributions -Syn1, (b) Instance-wise weight distributions -Syn2, (c) Instance-wise weight distributions -Syn3, (d) Average instance-wise weights of training samples in terms of distance from the probe sample (percentile)



Table content: <table><tr><td>(e.g. a</td></tr><tr><td>deep neural network or a decision tree-based ensemble model),</td></tr><tr><td>2. Locally interpretable model g θ : X → Y -an inherently interpretable model by design (e.g. a</td></tr><tr><td>linear model or a shallow decision tree),</td></tr><tr><td>3. Instance-wise weight estimation model h φ : X × X × Y → [0, 1] -a function that outputs</td></tr><tr><td>the instance-wise weights to fit the locally interpretable model. It uses concatenation of a probe</td></tr><tr><td>feature, a training feature, and a corresponding black-box model prediction on the training feature</td></tr><tr><td>as its inputs. It can be a complex machine learning model -e.g. here a deep neural network.</td></tr></table>

Stage 0 -Black-box model training: This stage is the preliminary stage for RL-LIM. Given the training set D, the black-box model f is trained to minimize a loss function (L f ) (e.g. mean squared error for regression or cross-entropy for classification), i.e., f *

Table content: <table/>

Real-world regression dataset results. Overall prediction performance (metric: MAE, lower is better) and fidelity (metric: R 2 score, higher is better) on regression problems with ridge regression as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. MAE of global ridge regression (RR) can be found below the data name. Red represents performance that is worse than global ridge regression and the negative R 2 scores. Bold represents the best results.

Table content: <table><tr><td/><td/><td>.131</td><td>1.0</td><td>4.965</td><td>1.0</td><td>4.893</td><td>1.0</td><td>5.203</td><td>1.0</td></tr><tr><td/><td colspan="3">RL-LIM 5.289 .8679</td><td>4.971</td><td>.9069</td><td>4.994</td><td>.7177</td><td>4.993</td><td>.8573</td></tr><tr><td>Blog</td><td>LIME</td><td colspan="8">9.421 .3440 10.243 .3019 10.936 -.2723 19.222 -.2143</td></tr><tr><td>(8.420)</td><td>SILO</td><td colspan="2">6.261 .0005</td><td>6.040</td><td>.2839</td><td>5.413</td><td>.4274</td><td>6.610</td><td>.4500</td></tr><tr><td/><td colspan="3">MAPLE 5.307 .8248</td><td>4.981</td><td>.8972</td><td>5.012</td><td>.5624</td><td>5.058</td><td>.8471</td></tr><tr><td/><td colspan="2">Original 24.18</td><td>1.0</td><td>20.22</td><td>1.0</td><td>18.36</td><td>1.0</td><td>30.09</td><td>1.0</td></tr><tr><td/><td colspan="3">RL-LIM 22.92 .7071</td><td>24.84</td><td>.4268</td><td>20.23</td><td>.5495</td><td>22.65</td><td>.4360</td></tr><tr><td>Facebook</td><td>LIME</td><td colspan="2">35.20 .2205</td><td>38.19</td><td>.2159</td><td>38.82</td><td>.2463</td><td>51.77</td><td>.1797</td></tr><tr><td>(24.64)</td><td>SILO</td><td colspan="5">31.41 -.4305 39.10 -1.994 22.35</td><td>.3307</td><td colspan="2">42.05 -.7929</td></tr><tr><td/><td colspan="3">MAPLE 23.28 .6803</td><td colspan="6">41.86 -3.233 24.77 -.1721 44.75 -1.078</td></tr><tr><td/><td colspan="2">Original 2995</td><td>1.0</td><td>3140</td><td>1.0</td><td>2255</td><td>1.0</td><td>3378</td><td>1.0</td></tr><tr><td/><td colspan="3">RL-LIM 2958 .7534</td><td>2957</td><td>.5936</td><td>2260</td><td>.9761</td><td>2396</td><td>.6523</td></tr><tr><td>News</td><td>LIME</td><td colspan="2">5141 -.2467</td><td>6301</td><td>-2.008</td><td>2289</td><td>.5030</td><td>9435</td><td>-7.477</td></tr><tr><td>(2989)</td><td>SILO</td><td colspan="2">3069 .4547</td><td>3006</td><td>.4025</td><td>2257</td><td>.9617</td><td>3251</td><td>.3816</td></tr><tr><td/><td colspan="3">MAPLE 2967 .7010</td><td>3005</td><td>.3963</td><td>2259</td><td>.9534</td><td>3060</td><td>.5901</td></tr></table>

4 QUALITATIVE ANALYSES -INTERPRETATIONS OF RL-LIM ON ADULT INCOME DATASET In this subsection, we qualitatively analyze the local explanations provided by RL-LIM on the Adult Income dataset. Although RL-LIM is able to provide local explanations for each individual separately, we analyze its explanations in subgroup granularity for better visualization and understanding. Fig. 4 represents the feature importance (derived by RL-LIM as the local explanations) for

Table content: <table><tr><td>Datasets</td><td>Models</td><td colspan="2">XGBoost</td><td colspan="2">LightGBM</td><td>MLP</td><td/><td>RF</td></tr><tr><td colspan="2">(DT-APR) Metrics</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td></tr><tr><td/><td colspan="2">Original .8096</td><td>1.0</td><td>.8254</td><td>1.0</td><td>.7678</td><td>1.0</td><td>.7621</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM .8011 .9889 .8114 .9602 .7710 .9451 .7881 .8788</td></tr><tr><td>Adult</td><td>LIME</td><td colspan="8">.6211 .5009 .6031 .3798 .4270 .2511 .6166 .3833</td></tr><tr><td>(.6388)</td><td>SILO</td><td colspan="8">.8001 .9869 .8107 .9583 .7708 .9470 .7833 .8548</td></tr><tr><td/><td colspan="9">MAPLE .7928 .9794 .8034 .9405 .7719 .9410 .7861 .8622</td></tr><tr><td/><td colspan="2">Original .7133</td><td>1.0</td><td>.7299</td><td>1.0</td><td>.7205</td><td>1.0</td><td>.7274</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM .7071 .9734 .7118 .9601 .7099 .9124 .7102 .9008</td></tr><tr><td>Weather</td><td>LIME</td><td colspan="8">.6179 .7783 .6159 .6913 .5651 .3417 .6209 .3534</td></tr><tr><td>(.5838)</td><td>SILO</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Real-world regression dataset results. Overall prediction performance (metric: MAE, lower is better) and fidelity (metric: R 2 score, higher is better) on regression problems with shallow regression tree as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. MAE of global regression tree (RT) can be found below the data name. Red represents performance that is worse than global regression tree and the negative R 2 scores. Bold represents the best results.

Table content: <table><tr><td colspan="3">E ADDITIONAL RESULTS</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="10">E.1 REGRESSION WITH SHALLOW REGRESSION TREE AS THE LOCALLY INTERPRETABLE</td></tr><tr><td>MODEL</td><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Datasets</td><td>Models</td><td colspan="2">XGBoost</td><td colspan="2">LightGBM</td><td>MLP</td><td/><td>RF</td><td/></tr><tr><td>(RT-MAE)</td><td>Metrics</td><td>MAE</td><td>R 2</td><td>MAE</td><td>R 2</td><td>MAE</td><td>R 2</td><td>MAE</td><td>R 2</td></tr><tr><td/><td colspan="2">Original 5.131</td><td>1.0</td><td>4.965</td><td>1.0</td><td>4.939</td><td>1.0</td><td>5.203</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM 5.121 .8242 4.778 .8939 4.587 .6375 4.652 .8990</td></tr><tr><td>Blog</td><td>LIME</td><td colspan="8">11.80 .2658 13.22 .1483 7.396 -.6201 19.61 -.4116</td></tr><tr><td>(5.955)</td><td>SILO</td><td colspan="8">5.149 .8035 4.818 .8816 4.649 .6177 4.715 .8774</td></tr><tr><td/><td colspan="9">MAPLE 5.329 .7991 5.024 .8660 4.609 .6339 5.016 .8201</td></tr><tr><td/><td colspan="2">Original 24.18</td><td>1.0</td><td>20.22</td><td>1.0</td><td>18.36</td><td>1.0</td><td>30.09</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM 21.82 .9307 21.35 .9194 18.56 .8832 22.44 .7236</td></tr><tr><td>Facebook</td><td>LIME</td><td colspan="8">36.69 .3278 44.21 .1809 40.85 -.1513 51.70 .2301</td></tr><tr><td>(22.28)</td><td>SILO</td><td colspan="8">22.42 .8655 22.33 .7235 19.57 .8566 24.41 .6917</td></tr><tr><td/><td colspan="9">MAPLE 22.15 .8824 23.43 .8581 20.32 .8035 27.12 .3134</td></tr><tr><td/><td>Original</td><td>2995</td><td>1.0</td><td>3140</td><td>1.0</td><td>2255</td><td>1.0</td><td>3378</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM 2938 .9382 2504 .4104 2226 .9016 2431 .2768</td></tr><tr><td>News</td><td>LIME</td><td colspan="8">6272 -.6267 7737 -2.960 2390 .0013 9637 -7.075</td></tr><tr><td>(3093)</td><td>SILO</td><td colspan="8">2910 .1020 2854 .3461 2274 .8201 2874 .2278</td></tr><tr><td/><td colspan="9">MAPLE 2968 .9288 2846 .3631 2284 .8021 2888 .1872</td></tr><tr><td/><td>Datasets</td><td>Models</td><td colspan="4">XGBoost LightGBM MLP</td><td>RF</td><td/><td/></tr><tr><td/><td/><td>RL-LIM</td><td/><td>.7530</td><td>1.358</td><td colspan="3">1.273 1.413</td><td/></tr><tr><td/><td>Blog</td><td>LIME SILO</td><td/><td>9.160 .8325</td><td>11.16 1.379</td><td colspan="3">5.006 17.461 1.178 1.934</td><td/></tr><tr><td/><td/><td>MAPLE</td><td/><td>1.029</td><td>1.598</td><td colspan="3">1.359 2.158</td><td/></tr><tr><td/><td/><td>RL-LIM</td><td/><td>7.240</td><td>6.867</td><td colspan="3">5.596 15.77</td><td/></tr><tr><td/><td>Facebook</td><td>LIME SILO</td><td/><td>31.52 8.459</td><td>37.75 9.149</td><td colspan="3">30.58 45.58 6.997 18.63</td><td/></tr><tr><td/><td/><td>MAPLE</td><td/><td>7.985</td><td>8.644</td><td colspan="3">7.290 23.17</td><td/></tr><tr><td/><td/><td>RL-LIM</td><td/><td>389.0</td><td>1072</td><td colspan="3">116.6 957.1</td><td/></tr><tr><td/><td>News</td><td>LIME SILO</td><td/><td>4455 496.7</td><td>6243 1214</td><td>504.0 160.6</td><td>9969 1175</td><td/><td/></tr><tr><td/><td/><td>MAPLE</td><td/><td>440.7</td><td>1201</td><td>163.6</td><td>1196</td><td/><td/></tr></table>

Fidelity results (Metric: LMAE, lower the better) on regression problems with shallow regression tree as the locally interpretable model. Bold represents the best results. E.2 CLASSIFICATION WITH RIDGE REGRESSION AS THE LOCALLY INTERPRETABLE MODEL RL-LIM .7140 .9879 .7290 .9801 .7212 .9755 .7331 .9450 Weather LIME .6376 .7898 .6392 .6873 .6395 .5321 .6387 .4513 (.7009) SILO .7134 .9888 .7281 .9773 .7220 .9797 .7277 .9024 MAPLE.7134 .9897 .7273 .9778 .7213 .9702 .7308 .9323

Table content: <table><tr><td>Datasets</td><td>Models</td><td colspan="2">XGBoost</td><td colspan="2">LightGBM</td><td>MLP</td><td/><td>RF</td><td/></tr><tr><td>(LR-APR)</td><td>Metrics</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td><td>APR</td><td>R 2</td></tr><tr><td/><td colspan="2">Original .8096</td><td>1.0</td><td>.8254</td><td>1.0</td><td>.7678</td><td>1.0</td><td>.7621</td><td>1.0</td></tr><tr><td/><td colspan="9">RL-LIM .7977 .9871 .8039 .9439 .7670 .9791 .7977 .9217</td></tr><tr><td>Adult</td><td>LIME</td><td colspan="8">.6803 .7195 .6805 .6259 .6957 .8310 .7057 .6759</td></tr><tr><td>(.7553)</td><td>SILO</td><td colspan="8">.7912 .9750 .7884 .9301 .7655 .9778 .7664 .9140</td></tr><tr><td/><td colspan="9">MAPLE .7947 .9840 .8011 .9386 .7683 .9636 .7958 .8961</td></tr><tr><td/><td colspan="2">Original .7133</td><td>1.0</td><td>.7299</td><td>1.0</td><td>.7205</td><td>1.0</td><td>.7274</td><td>1.0</td></tr></table>

Real-world classification dataset results. Overall prediction performance (metric: APR, higher is better) and fidelity (metric: R 2 score, higher is better) on classification problems with ridge regression as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. APR of global logistic regression (LR) can be found below the data name. Red represents the results that are worse than global logistic regression and the negative R 2 scores. Bold represents the best results.E.3 REGRESSION WITH RIDGE REGRESSION AS THE LOCALLY INTERPRETABLE MODEL -FIDELITY ANALYSIS IN TERMS OF LOCAL MAE (LMAE)

Table content: <table><tr><td>Datasets</td><td>Models</td><td colspan="3">XGBoost LightGBM MLP</td><td>RF</td></tr><tr><td/><td>RL-LIM</td><td>.8679</td><td>1.135</td><td colspan="2">1.432 1.651</td></tr><tr><td>Blog</td><td>LIME SILO</td><td>6.534 2.220</td><td>8.037 3.046</td><td colspan="2">8.207 17.01 2.393 3.909</td></tr><tr><td/><td>MAPLE</td><td>.9690</td><td>1.416</td><td colspan="2">1.550 1.984</td></tr><tr><td/><td>RL-LIM</td><td>6.394</td><td>21.29</td><td colspan="2">8.217 33.64</td></tr><tr><td>Facebook</td><td>LIME SILO</td><td>32.57 19.51</td><td>33.70 30.07</td><td colspan="2">27.38 48.03 11.52 40.14</td></tr><tr><td/><td>MAPLE</td><td>7.664</td><td>31.25</td><td colspan="2">13.31 44.38</td></tr><tr><td/><td>RL-LIM</td><td>436.9</td><td>1049</td><td colspan="2">74.11 905.8</td></tr><tr><td>News</td><td>LIME SILO</td><td>3317 657.2</td><td>4766 1253</td><td colspan="2">327.4 8828 79.85 1345</td></tr><tr><td/><td>MAPLE</td><td>500.5</td><td>1261</td><td colspan="2">88.19 1157</td></tr></table>

Fidelity results (Metric: LMAE, lower the better) on regression problems with ridge regression as the locally interpretable model. Bold represents the best results.

Table content: <table/>

Peer review (iclr2020):
In this paper, the authors aim to learn a locally interpretable model via the reinforcement learning approach, to address the fundamental challenge which is that the previous locally interpretable model has smaller representation capacity than black-box models, and causes under-fitting with conventional distillation techniques. Overall speaking, the paper is well organized, and the proposed approach is well tested, but in my opinion, there is a conceptual error. You claimed your method is REINFORCEMENT LEARNING based, but the REINFORCEMENT LEARNING definition for your task is weird, or wrong. In section 3, you didn*t give an explicit explanation for the state transition. With your given RL-like objective function, it seems that the state transition is from features to features. However, there is no specific correlated explanation in your paper on why you make such an assumption. Besides, the state transition in RL relies on decision making at each time step, while it has not reflected in your paper and code, namely, the state-transition independents on the decision making. To sum up, I don’t think the proposed method is RL-based, it would be more appropriate to define it as a MAB problem, and this paper should solve this problem before publishing.