RL-LIM: Reinforcement Learning-based Locally Interpretable Modeling
Authors: Jinsung Yoon
Sercan O. Arik
Tomas Pfister
Venue:  (2020)
Keywords: Interpretability
Explanable AI
Explanability

Abstract: Understanding black-box machine learning models is important towards their widespread adoption. However, developing globally interpretable models that explain the behavior of the entire model is challenging. An alternative approach is to explain black-box models through explaining individual prediction using a locally interpretable model. In this paper, we propose a novel method for locally interpretable modeling -- Reinforcement Learning-based Locally Interpretable Modeling (RL-LIM). RL-LIM employs reinforcement learning to select a small number of samples and distill the black-box model prediction into a low-capacity locally interpretable model. Training is guided with a reward that is obtained directly by measuring agreement of the predictions from the locally interpretable model with the black-box model. RL-LIM near-matches the overall prediction performance of black-box models while yielding human-like interpretability, and significantly outperforms state of the art locally interpretable models in terms of overall prediction performance and fidelity. 

Under review as a conference paper at ICLR 2020
RL-LIM: REINFORCEMENT LEARNING-BASED LOCALLY INTERPRETABLE MODELING
Anonymous authors
Paper under double-blind review
ABSTRACT
Understanding black-box machine learning models is important towards their widespread adoption. However, developing globally interpretable models that ex- plain the behavior of the entire model is challenging, An alternative approach is to explain black-box models through explaining individual prediction using locally interpretable model. In this paper, we propose a novel method for locally inter- pretable modeling - Reinforcement Learning-basee baased Locally Interpretable Model- ing (RL-LIM). RL-LIM employs reinforcement learning to select small number samples and distill the black-box model prediction into a low-capacity locally interpretable model. Training is guided with a reward that is obtained directly by measuring agreement of the predictions from the locally interpretable model with the black-box model. RL-LIM near-matches the overall prediction performance of black-box models while yielding human-like interpretabilityy and significantly outperforms state of the art locally interpretable models terms of overall pre- diction performance and fidelity.
INTRODUCTION
Artificial Intelligence (AI) is advancing at a rapid pace, particularly with recent advances in deep neural networks and ensemble methods (Goodfellow al., 2016 He et al. 2016 Chen & Guestrin 2016 Ke et al. (2017). This progress has been fueled by 'black-box machine learning models where the decision making is controlled by complex non-linear interactions between many parameters that are difficult for humans to understand and interpret. However, in many real-world applications AI systems are not only expected to perform well but are also required to be interpretable: doctors need to understand why a particular treatment is recommended, and financial institutions need to understand why a loan was declined, Use cases of model interpretability vary across applications: it can provide trust to users by showing rationales behind decisions, enable detection of systematic failure cases, and provide actionable feedback for improving models (Rudin) 2018).
Many studies have suggested trade-off between performance and interpretability (Virág & Nyitrai 2014 Johansson et al. 2011). This is correct in that globally interpretable models, which attempt explain the entire model behavior, typically yield considerably worse performance than "black- box' models (Lipton) 2016). To go beyond the performance limitations of globally interpretable models, another promising direction is locally interpretable models, which instead of explaining the entire model explain a single prediction (Ribeiro et al. 2016). Methodologically, while a globally interpretable model fits single inherently interpretable model (such as linear model or shal- low decision tree) to the entire training set, locally interpretable models aim to fit an inherently interpretable model locally, i.e. for each instance individually, by distilling knowledge from a high performance black-box model. Such locally interpretable models are very useful for real-world AI deployments to provide succinct and human-like explanations to users. They can be used identify systematic failure cases (e.g. by seeking common trends in input dependence for failure cases), de- tect biases (c.g. by quantifying feature importance for a particular variable), and provide actionable feedback to improve model (e.g. understand failure cases and what training data to collect).
To be useful in practice, locally interpretable models need to maximize two objectives: (i) the overall prediction performance (how well it predicts compared to the ground truth labels) for the model to be accurate, and (ii) fidelity (how well approximates the 'black-box' model predictions) - to ensure the model is reliably approximating the black-box model's predictions in the neighhorhood

---

Under review as a conference paper at ICLR 2020
of interest (Plumb al. 2019 Lakkaraju et al. 2019). To this end, few methods have recently been proposed for locally interpretable modeling: Local Interpretable Model- -ggnosicc Explanations (LIME) (Ribeiro et al.] 2016), Supervised Local modeling methods (SILO) (Bloniarz et al. 2016), and Model Agnostic Supervised Local Explanations (MAPLE) (Plumb et al. 2018). LIME in partic- ular has gained notable popularity and has been deployed in many applications due to its simplicity. However, the overall prediction performance and fidelity metrics are not reaching desired levels in
many cases (Alvarez-Melis & Jaakkolal 2018 Zhang et al. 2019; Ribeiro et al., 2018 Lakkaraju et al. 2017). Indeed, as we show in our experiments. there are frequent cases where existing locally interpretable models even underperform commonly low- wwwpffrrrrmnn globally interpretable models.
One of the fundamental challenges to fit locally interpretable model is the representational capac- ity difference while applying distillation. Black-box machine learning models, such as deep neural networks or ensemble models, have much larger representational capacity than locally interpretable models. This can result in underfitting with conventional distillation techniques, leading to subopti- mal performance (Hinton et al. 2015 Wang et al. 2019). We address this fundamental challenge by proposing a novel Reinforcement Learning-based method to fit Locally Interpretable Models which call RL-LIM. RL-LIM efficiently utilizes the small representational capacity of locally inter- pretable models by training with a small number of samples that are determined to have the highest value contribution the fitting of locally interpretable model. In order to select these highest-valuu instances, we train instance-win -wise weight estimators (modeled with deep neural networks) using re- inforcement signal that quantifies the fidelity metric (i.e. how well does the model approximate the black-box model predictions). The contributions of this paper can be summarized as:
1. We introduce the first method that tackles interpretability through data-weighted training, and show that reinforcement learning is highly effective for end-to-ens training of such a model.
2. We show that distillation of a black-box model into low-cat cowccppcctty interpretable model can be sig- nificantly improved by fitting with a small subset of relevant samples that is controlled efficiently by our method.
3. On various classification and regression datasets, we demonstrate that RL-LIM significantly out- performs alternative models (LIME, SILO and MAPLE) in overall prediction performance and fidelity metrics in most cases, the overall performance of locally interpretable models obtained by RL-LIM is very similar to complex black-box models.
2 RELATED WORK
Locally interpretable models: There are various approaches to interpret black-box models Gilpin et al., 2018) provides a good overview. One approach is to directly decompose the predic- tion into feature attributions by considering what-if cases. Shapley values (Strumbelj & Kononenko variants (Lundberg & Leel 2017) are commonly-used
2014) and their computationally-effciiiee methods in this category. Other notable methods are based on activation differences, e.g. DeepLIFT (Shrikumar et al. 2017), saliency maps using the gradient flows, e.g. CAM (Zhou et al. (016) and Grad-CAM (Selvaraju et al.] 2017). In this paper, we focus on the direction of locally interpretable modeling- distilling black-box model into an interpretable model for each input instance.
Locally Interpretable Model-agnostic Explanation (LIME) (Ribeiro et al., 2016) is the most popular method for locally interpretable modeling. LIME is based on modifying a data instance by tweaking the feature values and then learning from the impact of the modifications on the output. A funda- mental challenge for LIME is the need for a meaningful distance metric to determine neighborhoods, as simple metrics like Euclidean distance may yield poor fidelity in some cases and the estimation can be highly-sensitive to normalization
Aalvarez-Melis & Jaakkola 2018) especially with categor- ical variables. Supervised Local modeling methods (SILO) (Bloniarz et al. 2016)) aims to improve LIME by determining the neighborhoods for each instance using ad-hoc tree- ensemble meth- ods. Model Agnostic Supervised Local Explanations (MAPLE) (Plumb et al., 20188) furthers adds a method for feature selection on top of SILO - it utilizes ad-hoc tree-based ensemble methods to determine the weights of training instances for each target instance and uses the weights to opti- mize a locally interpretable model. However, SILO and MAPLE still have shortcomings because the tree-based ensemble methods are optimized independently from the locally interpretable model lack of joint optimization results in suboptimal fidelity for the locally interpretable model. Over- all, to construct locally interpretable model, a key problem is how to select the optimal training
2

---

Under review as a conference paper at ICLR 2020
instances for each testing instance, because the selected training instances mostly determine the constructed locally interpretable model. The number of possibilities for training instance selection is extremely large (exponential in the number of training instances). LIME heuristically utilizes Euclidean distances, whereas SILO and MAPLE use ad-hoc tree-based ensemble methods. Our proposed method, RL-LIM, takes very different perspective: to properly and efficiently explore the large possible solution space, RL-LIM utilizes reinforcement learning to find the optimal policy that selects the training instances that maximize the fidelity of the locally interpretable model.
Data-weighted training: Optimal weighing of training data is a paramount problem in machine learnin. By upweighting valuable instances and downweighting the low quality or problematic instances, better performance can be obtained in certain learning scenarios, such as imbalanced or noisy labels (Jiang et al., 2018). One approach for data weighting is utilizing Influence Func- tions (Koh & Liang, (017). that are based on oracle access to gradients and Hessian- ucts. Jointly-trained
prod-
methods constitute another approach (Jiang et al., [2018] [Bengio] et [ to learn data-driven curriculum. Using the feedback from the teacher network, training instance-wise weights are learned for the student model. Aligned with our motivations, meta learn- ing is considered for data weighting in Ren et al. (2018). Their proposed method utilizes gradient descent-based meta learning, guided by a small validation set, to maximize the target performance.
In this work we consider data- weighted training for novel purpose: interpretability. Unlike gradi- ent descent-based meta learning, our approach uses reinforcement learning to integrate the reward directly with the fidelity metric. Aforementioned works estimate the same ranking of training in- stances for the entire dataset. Instead, our method yields an instance-wise ranking of training data points, different for each testing instance. This enables efficient distillation of a black-box model prediction into a locally interpretable model.
3 REINFORCEMENT LEARNING- MODELING
We consider a training dataset D (xx,,)),, N} P for training of a black-box model f. where x; E X' is the feature vector in a d-dimensional feature space X' and y/: E y is the corresponding label in label space V. We also assume that there exists probe dataset DD {(x", v I,... M} ~ P where M is the number of probe instances. The probe dataset is used evaluate the model performance guide meta-learning as in Ren et al. (2018). If there is no explicit probe dataset, we can randomly partition subset of the training dataset as the probe dataset and the remainder as the training dataset. RL-LIM is composed of three models:
1. Black-box model ʃ X + + y- - any machine learning model that needs be explained (e.g. deep neural network or a decision tree-based ensemble model),
2. Locally interpretable model gn : X' + y an inherently interpretable model by design (e.g. a linear model or a shallow decision tree),
Instance-wise weight estimation model h& X' x X' x + [0, 1] function that outputs the instance-wise weights to fit the locally interpretable model. It uses concatenation of probe feature, training feature, and a corresponding box model prediction on the training feature as its inputs. It can be a complex machine learning model - e.g. here deep neural network.
Our objective is to construct an accurate locally interpretable model ga such that the predictions made by it are similar to the predictions of the given black-box model f"- i.e. the locally inter- pretable model has high fidelity. We use loss function, L 3 x y + R to quantify the fidelity of the locally interpretable model (e.g. mean absolute error, lower the better). In RL-LIM, the three necessary components of an RL framework are as follows: the state is the vector of input features, the action is the selection vector, and the reward the fidelity which depends on the input fea- tures and the selection vector. The instance-wise weight estimator model is the agent that outputs the actions based on the state (input features). The environment is comprised of the input feature generating process, well as the black-box model for the target task.
The representational capacity difference between the black- box model and the locally interpretable model is the bottleneck aim to address. Ideally, to avoid underfitting, locally interpretable models should be learned with minimal number of training instances that are most effective in capturing the model behavior. We propose instance-wise weight estimation model h& estimate the probabil-
3

---

Under review as a conference paper at ICLR 2020
ities of training instances that should be used for fitting the locally interpretable model. Integrating with the accurate locally interpretable modeling goal, we propose the following objective:
min hig
Ex [x(f((x*) (o (x*))] + AlEx xx (xx
(I)
s.t. 30(x=) arg ming, Ex [x (x)) Lg (x), go
where À 2 0 is a hyper-parameter that controls the number of training instances used to fit the locally interpretable model (we study the impact of performance on À in Section 4.2), and ha(x,, x, f (x)) represents the instance-wise weight for each training pair (x, f (x)) for the probe data x". Lg the
loss function to fit the locally interpretable model, for which we use the mean squared error between
predicted values for regression and logits for classification. & and are the trainable parameters, whereas /* (the pre-trained black-box model) is fixed.
The first term in the objective function Ex--Px [C(f*(x") (x")) (x*))] represents the local predic- tion differences between black-box model and locally interpretable model (referred to as fidelity metric). The second term in the objective function Ex xx [xo x. (x))x represents the ex- pected number of selected training points to fit the locally interpretable model. Lastly, the constraint ensures that the locally interpretable model is derived from weighted loss function, where weights are the output of the iistance-wise weight estimator hgg Our formulation does not assume any con- straint on Go g it could be any inherently interpretable model suitable for the data type of interest. Next, we describe how Eq. can be efficiently addressed with reinforcement learning.
3.1 TRAINING AND INFERENCE
The RL-LIM method, shown in Fig. can be thought of as encompassing stages:
Black box model
Black-bux
Trainig dataeci
9
Interpretable baseline
9
Los
Stage 0: Black-box model training
Stage 1: Auxiliary dataset construction
Stage 2: Interpretable
baseline training
Grey Trainable model
White block: Fixed model
Probe sample
Instancs--iie estimator
Solection pootebiiiiize
Testing sample
Sampler
Invlance-wine weight estimater
vector
pubabbliticc
Weighted
Weightul Interprctable model
opcimization
Interpretable model
Interpretable
baseline
Low
Loss
Instance-wie expluations
Interpretable prodictions
Reward
Stage 3: Instance-wise weight estimator training
Stage 4: Interpretable inference
Figure 1: The proposed RI.LLIM method. White blocks represent fixed (not learnable) models, and grey blocks represent learnable (trainable) models. Stage O: Black-box model training. Stage 1: Auxiliary dataset construction. Stage 2: Interpretable baseline training. Stage 3: Instance-wise weight estimator training. Stage 4: interpretable inference.
4

---

Under review as a conference paper at ICLR 2020
Black-box model training: This stage is the preliminary stage for RL-LIM. Given the training set D, the blx model f is trained to minimize loss function (C2) (e.g. mean squared enror for regression cross-yntropy for classification), ie., f arg miny N E (x( vi). If the pre-trained black-box model is already saved, we can skip this stage and retrieve the given pre-trained black-box model to f*
Stage 0
Stage 1 Auxiliary dataset construction: Using the pre-trained black-box model f", we create auxiliary training and probe datasets, D {(x,; f.), ,,...,, N] (where ü (x,)) and DP {(x]. 0f). l...., ,) (where if f (x()), respectively. These auxiliary datasets (D, Dr) used for mistance-wise weight estimation models and locally interpretable model training.
Stage 2 Interpretable baseline training: To improve the stability of the instance-wis wise weight estimator training, a baseline model is observed to be beneficial. As the baseline model gb : X - V. we use globally interpretable model (such as linear model or shallow decision tree) opti- mized replicate the predictions of the black-box model: g arg min, t Σi ((g(x), Uc).
Stage 3 - Instance-wise weight estimator training: We train an instance-wise weight estimator using the auxiliary datasets (D, DP). To encourage exploration, we consider probabilistic se- lection, with sampler block that is based on the output of the instance-wise weight estimator ha X,, Xc) represents the probability that (x,, V: ) is selected to train locally interpretable model for the probe instance x²,. Let the binary vector e(x})) {0. 1 represent the selection opera- tion, such that (x,, úi) is selected for training locally interpretable model for xf, when e: (x²)) 1. Correspondingly, Pa(x)) the probability mass function for c(x²) given h-):):
(x(x),c(x] - II (x) (xx) (1 has (x): X,, ( (x:)))=-<(04)
As the original form of the optimization problem in Eq. is intractable due to the expectation operations, employ approximations:
The sample mean is used an approximation of the first term of the objective function as (x)(()))
1े (( Se(gg ( (x)))..
The second term of the objective, which represents the average selection probability, is
approximated as the number of selected instances (divided by N) to have [|c(x))]l
E (xx())
The constraint term is approximated using the sample mean of the training loss as 9
arg ming, N MM (x(x))) (f))
The sampler block yields a non-differential objective. and we cannot train the instance- weight estimator using conventional gradient descent-based optimization. There are approxi- mations such as training in expectation (Raffel et al., 2017) or Gumbel-softmax (Jang et al,, 2016). Instead, motivated by its many successful applications (Ranzato et al.] 2013 Zaremba & Sutskever] 2015 Zhang & Lapatal (017), we use REINFORCE algorithm (Williams) 1992) such that the selection action is rewarded by the performance of its impact. The loss function for the instance-wint wise weight estimator (()) is expressed as:
(Φ)) = Ex E [e [C(("(()) (x) + X1]e(x))111]
To apply the REINFORCE algorithm, we directly compute the gradient V Vol( as:
V = Besssr |Cf (x)) 90xg) (x())) + X||c(x3))l1 V log (x c(x)))
Using the gradient V (Ø)( we employ the following steps iteratively to update the parameters of the aistance-wise weight estimator Φz
1. Estimate instance-wise weights wx(x)) = hx ùt) and instance-wise selection vector es (xx)) Ber(w: (x))) for each training and probe instance in a mini-batch.
5

---

Under review as a conference paper at ICLR 2020
2. Optimize the locally interpretable model with the selection vector for each probe instance:
9 = arg min
cxxx)) Lg(
3. Update the instance-wise weight estimation model parameter Φ6:
M (x))) (x7)) Lo(x)) + + X1|c(x)) V log (x c(x)))
where > is learning rate and Lx(x)) (x )) is the baseline loss against which we benchmark the performance improvement. We repeat the steps above until convergence.
fixed instance-wise weight estimator (without the sampler and interpretable baseline) and merely fit the locally interpretable model at inference. Given the test instance x", we obtain the selection probabilities from the instance-wise weight estimator, and using these as the weights, we fit the locally interpretable model via weighted optimization. The outputs of the trained interpretable model are the instance- wise predictions and the corresponding explanations (e.g., local dynamics of the black-bor box model predictions at x given by the coefficients of the fitted linear model).
Stage 4 Interpretable inference: Unlike when training,
3.2 COMPUTATIONAL COST
In this subsection, we analyze the computational cost of RL-LIM for training and inference. As representative and commonly used example, we assume linear regression as the locally interpretable model, which has computational complexity of (d + O(d)) to fit, where d is the number of features and N is the number of training instances. When >> d (which is often the case in practice), the training computational complexity is approximated as Odd N) (Tan 2018).
Given pre-trained black-box model, Stage involves running inference N times and the total complexity depends on the complexity of the black-box model. Unless the black-box model is very complex, the computational complexity of Stage becomes much smaller than Stage 3. Stage has negligible computational overhead. At Stage 3, we iteratively train the instance-wise weight estimator and fit the locally interpretable model from scratch using weighted optimization. Therefore, the computational complexity Od22NN, where N, is the number of iterations in Stage 3 (typically N, < 10, 000 until convergence). Thus, the training complexity scales roughly linearly with the number of training instances.
Training:
Interpretable inference: To infer with the locally interpretable model, we need to fit the locally interpretable model after obtaining the instance-wise weights from the trained instance- weight estimator. Thus, for each testing instance, the computational complexity is (dd
For instance, on a single NVIDIA V100 GPU, on Facebook Comment dataset (consisting ~ 600,000 samples), RL-LIM yields training time of less than 5 hours (including Stage 1, 2 and 3) and an interpretable inference time of less than 10 seconds per a testing instance. On the other hand, LIME results in much longer interpretable inference time (around 30 seconds per testing instance) due to acquiring a large number black-box model predictions for the inputs perturbations, whereas SILO and MAPLE are similar RL-LIM.
4 EXPERIMENTS
We compare RL-LIM to multiple benchmarks on 3 synthetic datasets and 5 UCI public datasets.
Datasets: The 3 public datasets for regression problems are: (1) Blog Feedback, (2) Facebook Comment, (3) News Popularity the other 2 public datasets for classification problems are: Adult Income, (5) Weather Details of the data descriptions can be found in the hyper-links of each dataset (colored in blue). Data statistics can be found in Table 3 in Appendix A In this section, we mainly focus on the tabular datasets because the local dynamics are more important and useful to explain for them; however, RL-LIM method can be generalized to other data types in straightforward way.
A subset of the training dataset used reduce complexity (with decreased fidelity).

---

Under review as a conference paper at ICLR 2020
Black-box models: We focus approximating black-box models that are shown to yield competi-
tive performance on the target tasks: 3 tree-based ensemble methods (1) XGBoost Chen & Guestrin
2016). (2) LightGBM [Ke et al., 2017). (3) Random Forests (RF) [Breiman 2001 and deep neural
networks (4) Multi-layer Perceptron ML.)) Also, we use (5) Ridge Regression (RR) and (6) Re-
gression Tree (RT) (for regression) and (7) Logistic Regression (LR) and (8) Decision Tree (DT)
(for classification) as globally interpretable models to benchmark We focus on two types of lo- cally interpretable models: (1) Ridge regression, (2) Shallow regression tree (with max depth of
3). We report the performance with ridge regression for regression and with shallow regression tree for classification in this section. The results of the other two combinations (with ridge regression for classification and with shallow regression tree for regression) are described in Appendix E
Comparisons to previous work: We compare the performance of RL-LIM with three competing methods: (1) Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016). (2)
Supervised Local modeling methods (SIL.)) (Bloniarz et al. 2016), (3) Model Agnostic Supervised Local Explanations (MAPLE) (Plumb al. 2018).
Performance metries: To evaluate the performance of locally interpretable models using real-world datasets, we quantify the overall prediction performance and its fidelity. We assume a disjoint testing dataset D' = {xx vf ) for evaluation. For the overall prediction performance, we compare the predictions of the locally interpretable models with the ground-trui labels. We use Mean Absolute Error (MAE) for regression and Average Precision Recall (APR) for classification. For fidelity, compare the outputs (predicted values for regression and logits for classification) of the locally interpretable models and of the black-box model. We consider two metrics: R²² score (Legates & McCabe 1999) and Local MAE (LMAE). The details of the metries are described in Appendix [
Implementation details: We implement instance-wise weight estimator using a multi-layer per- ceptron with tanh activation. The number of hidden units and layers are optimized by the cross- validation. In most cases, 5-layer perceptron with 100 hidden units performs reasonably-wel. across all datasets. All features are normalized to be between zero and one, using standard minmax scaler. Categorical variables are transformed using one-hot encoding.
4.1 EXPERIMENTS ON SYNTHETIC DATASETS RECOVERING LOCAL DYNAMICS
On real-world datasets it is challenging to directly evaluate the explanation quality of the locally in- terpretable models due to the absence of ground- toound-truth explanations. Thus initially focus syn- thetic datasets (with known ground-trume explanations) to directly evaluate how well the locally in- terpretable models can recover the underlying local dynamics. We construct three synthetic datasets such that the -idimensioonl input features X are sampled from \(, I) and Y are:
1. Syn1: Y = X1 + 2X, if X10 < 0 and Y X3 2X, if X10 2 0
2. Syn2: Y = X1 + 2X, if X10 + ex1 < 1 and Y = X3 2X4 if X10 + en 21
3. Syn3: Y = X1 + 2X, if X10 + x < 0 and Y = X3 + 2X, if X10 + X 20 0
All three datasets have different local dynamics in different input regimes. We directly use the ground truth function as the black-box model and focus on how well locally interpretable modeling can capture the local dynamics. We evaluate the performance of capturing local dynamics using Absolute Weight Difference (AWD): AWD = ||w - w/l. where w is the ground truth coefficients generate Y and w is the derived coefficient from the locally interpretable models. We use the estimated coefficients of the ridge regression as the derived local dynamics (w).
As shown in Fig. RL-LIM significantly outperforms other benchmarks in discovering the local dynamics on all three datasets and in different regimes. RL-LIM can actively learn the linear and non-linear decision boundaries for the local dynamics. Note that LIME completely fails to recover the local dynamics as it uses the Euclidean distance uniformly for all features and cannot distinguish the special properties of the features that alter the local dynamics. SII.O and MAPLE only use the predictions to discover the local dynamics; thus, it is hard to discover the decision boundary that depends on the other variables which are independent to the predictions. Fig. 5 in Appendix D shows the learning curves of RL-LIM demonstrating the efficiency of reinforcement learning.
We use python packages (including Skleam and Tensorflow) to implement those predictive models and the details can he found in the hyper-links (colored in blue) of each model and Appendix

---

Under review as a conference paper at ICLR 2020
RLUM
RL-M
UUUM
Diviunuu from the bounday (our (unkil))
Dalanke from tha boundayy (psrenttio)
Divlumnu lum the boundayy (se (onkk)
Figure 2: Synthetic dataset results. Mean absolute weight difference (AWD) with 95% confidence intervals (of 10 independent runs) on three synthetic datasets. X-axis: Distance from the boundary where the local dynamics change, such as X1o 0 for Synl (in percentile), Y-axis: AWD (the lower, the better). We exclude LIME in these graphs due to its poor performance in terms of AWD (it is higher than 1.6 all distance regimes for all three synthetic datasets).
4.2 THE EFFECT THE NUMBER OF SELECTED SAMPLES ON FIDELITY
In RL-LIM, optimal distillation is enabled by using small subset of training instances to fit the low-capacity locally interpretable model. The number of selected instances is controlled by X in our method - if À is high/low, RL-LIM penalizes more/less the number of selected instances; thus, less/more instances are selected to construct the locally interpretable model.
Syn2 Syn
Auy Salus
Syn?
Synl Synz Syn2
Syn3
8.6
05
26
Figure 3: Fidelity & average selection probability of training instances as a function of the number of selected samples on three synthetic datasets. X-axis: X, Y-axis: LMAE and average selection probability of training instances. LMAE is Local MAE - lower is better.
We analyze the efficacy of À in controlling the likelihood of selection and the dependency of fidelity on A. We expect that if we select a too small/large number of training instances, the locally inter- pretable model will overfit/underfit which negatively affects the fidelity in both cases. Fig. 3 shows that there clear relationship between À and the local fidelity. If too large, RL-LLM selects too small number of instances; thus, the fitted locally interpretable model is less accurate (due to overfitting). On the other hand, if À is too small, RL-LIM selects too large number of instances and deteriorates fidelity (due to underfitting). To achieve the optimal X, we conduct cross-validation experiments and select À which achieves the best validation fidelity (e.g. = 0.5 in Syn2). Fig. 3 shows the average selection probability of the training instances for each A. As X increases, the average selection probabilities monotonically decrease due to the higher penalty on the number of selected training instances. Note that even using small portion of training instances, RL-LIM can accurately distill the predictions of black-box models into locally interpretable models which is crucial to understand and interpret the predictions using the most relevant training instances.
4.3 EXPERIMENTS ON REAL DATASETS OVERALL PERFORMANCE AND FIDELITY
On multiple real datasets, we evaluate the overall prediction performance and fidelity. For the re- gression and classification problems, use ridge regression and shallow regression trees as the locally interpretable model. More results can be found in Appendix
As can be seen in Table [ the performance of globally interpretable ridge regression (trained on the entire dataset from the scratch) is much worse than other complex non-linear models, implying that modeling non-linear relationships between the features and the labels is important towards high prediction performance. For other locally interpretable modeling methods (LIME, SILO. MAPLE), the performance is far worse than the original black-box model, showing that they fail efficiently
8

---

Under review as a conference paper at ICLR 2020
Datasets
Models
XGBoost
LightGBM
MLP
RF
(RR-MAE)
Metrics
MAE
R?2
MAE
R2
MAE
R
MAE
R2
Original
5.131
1.0
4.965
1.0
4.893
5.203
RL-LIM
5,289
.8679
4,971
.9069
4.994
.7177
4.993
.8573
Blog (8.420)
LIME
9.421
.3400
10.243
.3019
10.936
-22733
19.222
-2143
SILO
6.261
.0005
6,040
.2899
5,413
.4274
6.610
.4500
MAPLE
5.307
.8248
4.981
.8972
5.012
.5624
5.058
.8411
Original
24.18
1.0
20.22
1.0
18.36
1.0
30.09
1.0
RL-LIM
22.92
.7071
24.84
.2268
20.23
.5495
22.65
.3600
Facebook (24.64)
LIME
35.20
.2055
38.19
.2159
38.82
.2463
51.77
.1797
SILO
31.41
-,4305
39.10
-1.994
22.35
.3307
42.05
-7929
MAPLE
23.28
.6803
41.86
-3.233
24.77
-.1721
44.75
-1.78
Original
2995
1.0
3140
1.0
2255
1.0
3378
1.0
RL-LIM
2958
.7534
2957
.5936
2260
.9611
2396
.6523
News (2989)
LIME
5141
-,2467
6301
-2.008
2289
.5030
9435
-7,477
SILO
3069
.4547
3006
.4025
2257
.9617
3251
.3816
MAPLE
2967
.7010
3005
.3663
2259
.9544
3060
.5901
Table Real-world regression dataset results. Overall prediction performance (metrie: MAE, lower is better) and fidelity (metric: R² score, higher is better) on regression problems with ridge regres- sion as the locally interpretable model. 'Original' is the performance the original black-box model that the models are approximating. MAE of global ridge regression (RR) can be found below the data name. Red represents performance that is worse than global ridge regression and the negative R scores. Bold represents the best results.
distilling the non-linear black-box models. some cases (especially on the Facebook dataset), the performance of the benchmarks is even worse than the performance of global ridge regression (highlighted red), questioning the value of using these locally interpretable models instead of globally interpretable ridge regression.
In contrast, RL-LIMM achieves similar overall prediction pertormane the black-box models and significantly outperforms glohal ridge regression. Table I also compares the fidelity in terms of R² score for regression using ridge regression as the locally interpretable model (LMAE results can be found in Appendix E.3 We observe that 2 scores for some cases (especially on Facebook dataset and LIME) are negative which represent that the outputs the locally interpretable models are even worse than the constant mean value estimator. On the other hand, RL-LIM achieves higher and positive 1² values consistently for all datasets and black-box models than other benchmarks.
2 similar analysis for classification using shallow regression trees (with max depth of 3) as the locally interpretable model (Regression trees are used to model logit outputs for classifica- tion.). The overall prediction performance of four black-box models significantly better than the globally interpretable decision tree which demonstrates the superior fitting by complex black-box models. Among the locally interpretable models, RL-LIM achieves the best APR and Rt score for most cases, underlining its strength in distilling the predictions of the black-box model accurately. In some cases, the benchmarks (especially for LIME) achieve lower overall prediction performance than the globally interpretable decision tree (highlighted in red). The overall prediction performance and fidelity metrics of all locally interpretable models seem better for classification problems than regression problems. We expect that the predictions of black-box models are mostly highly confi- dent, i.c. located near or I; thus, locally interpretable models can easily distill the predictions of the black-box models for classification than regression.
4.4 QUALITATIVE ANALYSES - INTERPRETATION AMALYSES-INTEERPEETTIIN
OF RL-LIM ON ADULT INCOME DATASET
In this subsection, we qualitatively analyze the local explanations provided by RL-LIM on the Adult Income dataset. Although RL-LIM is able to provide local explanations for each individual sepa- rately, we analyze its explanations in subgroup granularity for better visualization and understand- ing. Fig. represents the feature importance (derived by RL-LIM the local explanations) for

---

Under review as a conference paper at ICLR 2020
Datasets
Models
XGBoost
LightGBM
MLP
RF
(DT-APP)
Metrics
APR
R22
APR
R2
APR
R²
APR
R
Original
.096
1.0
.8254
1.0
.6778
1.0
.7621
1.0
RE-LIM
.8011
.8899
.8114
.9602
.7710
.9451
.7881
.8788
Adult (.6388)
LIME
.6211
.5099
.6031
.3798
.4700
.2511
.6166
.3833
SILO
8801
.869
8107
.9833
.7708
.9770
.7833
.8548
MAPLE
.7928
.994
8034
.9455
.7719
.9410
.7861
.8622
Original
.7133
1.0
.7299
1.0
.7205
1.0
.7274
1.0
RI-LIM
.7071
.9744
.7118
.6011
.7099
.9124
.7102
.9088
Weather (.5838)
LIME
.6179
.7783
.6159
.6913
.5651
.3417
.6209
.3534
SILO
.6991
.9680
.7522
.9452
.6997
.8864
.7042
.8398
MAPLE
.6973
.9755
.7556
9446
.6983
.8856
.6983
.8856
Table 2: Real-world classification dataset results. Overall prediction performance (metric: APR, higher is better) and fidelity (metrie: 172 score, higher better) on classification problems with shallow regression tree the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. APR global decision tree (DT) can be found below the data name.
Age
Meg.
captanan
WertClas
Relabonahic
Beeggest
O Kge
Mile
Marreta
Marnee and
1006
Mache are
bocto
hathehel
See marrraaog
Figure 4: Qualitative interpretability results. The analyses feature importance (derived by RL- LIM) for 5 types of subgroups in Adult Income dataset: (a) Age. (b) Gender, (c) Marital status, (d) Race, (e) Education. The color represents the feature importance for each subgroup.
five subgroups in predicting the annual income using XGBoost as the black-box model. We use ridge regression as the locally interpretable model and the absolute value of fitted coefficients as the estimated feature importance. As can be observed Fig. for age subgroups, capital gain seems much more important for mature people (older than 25) than young people (younger than 25). For education subgroups, capital gain/loss, occupation, and native countries are more critical for heduytedddcated people (Doctorate, Prof-school, and Masters graduates) than the others. We do not discover notable biases of black-box -bx models for gender, marital status, and race subgroups.
CONCLUSIONS
We propose a novel method for locally interpretable modeling of pre-trained black-box models. Our proposed method employs reinforcement learning to select a small number of valuable instances and use them to train a low-capacity locally interpretable model. The selection mechanism guided with a reward obtained from the similarity of predictions the locally interpretable model and the black- box model. Our approach near-matches the performance of black-box models and significantly outperforms alternative techniques terms overall prediction performance and fidelity metrics consistently across various datasets and black-box models.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806 00049, 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference Machine Learning, pp. 41-48. ACM, 2009.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint crXiv:130..3432
Adam Bloniarz, Ameet Talwalkar, Bin Yu, and Christopher Wu. Supervised neighborhoods for distributed nonparametrie regression. In Artificial Intelligence and Statistics, pp. 1450-1459, 2016.
Leo Breiman. Random forests. Machine Learning, 45(1):5-33 2001.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp- 785-794. ACM, 2016.
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80-89, Oct 2018.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7/www. dewpledeeled www.ddepppeaannnnooooooo.oo org
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531, 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung. Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-
driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning, pp. 2309-2313 2018.
Ulf Johansson, Cecilia Sönströd, Ulf Norinder, and Henrik Boström. Trade-off between accuracy and interpretability for predictive silico modeling. Future medicinal chemistry, 3(6):647-663,
2011.
Guolin Ke, Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems, pp. 2146-3155 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference Machine Learning, pp. 1885 1894, 2017.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Interpretable explorable approximations of black box models. arXiv preprint a7Xiv: 2017.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Faithful and customizable explanations black box models. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pp. 131-138. ACM, 2019.
David Legates and Gregory McCabe. Evaluating the use of "goodness-of-fit measures in hydrologic and hydroclimatic model validation. Water Resources Research, 35(1):233 1999.
Zachary C Lipton. The mythos of model interpretability. arXiv preprint arXiv: 0666666060999 2016.

---

Under review as a conference paper at ICLR 2020
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765 4774, 2017.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. Model agnostic supervised local explana- tions. In Advances in Neural Information Processing Systems, pp. 2515-2524, 2018.
Gregory Plumb, Maruan Al-Shedivat, Eric Xing, and Ameet Talwalkar. Regularizing black-box models for improved interpretability. arXiv preprint
2019.
Colin Raffel, Minh- Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck. Online and linear-ting attention by enforcing monotonic alignments. In International Conference Machine Learning, pp. 2837-2846, 2017.
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level train- ing with recurrent neural networks. arXiv preprint arXiv: 1 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pp. 4331-4340, 2018.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135-1144. ACM, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Cynthia Rudin. Please Stop Explaining Black Box Models for High arXiv:1811.10154, 2018.
Stakes Decisions.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Beerning-Volumee pp. 3145-3153, 2017.
Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41(3))647-66) 665, 2014.
Pang-Ning Tan. Introduction to Data Mining. Pearson Education India, 2018.
Miklós Virág and Tamás Nyitrai. Is there a trade-off between the predictive power and the inter- pretability of bankruptcy models? the case of the first hungarian bankruptcy prediction model. Acta Oeconomica, 64(4):419 440, 2014.
Tongzhou Wang, Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2019. Sy41ojC9tm
URL http::://openrevie..neett//
net
Ronald J Williams. Simple statistical gradient algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229-256, 1992.
Wojeiech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines-revised.
arXiv preprint arXiv:1505.00552 2015.
Xingxing Zhang and Mirella Lapata. Sentence simplification with deep reinforcement learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 584-594, 2017.
Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. "why should you trust my explanation?" understanding uncertainty in lime explanations. arXiv preprint ArXiv: 12991, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.
12

---

Under review as a conference paper at ICLR 2020
A
DATA STATISTICS
Problem
Data Name
# of samples
# of features
Label distribution
Blog
60,021
280
6.6 (0-0-22)
Regression
Facebook
603,713
7.2 (0-0-30)
News
39,644
59
3395.4 1584-1400-108000
Adult
48,842
108
11,687 (23.9%)
Classification
Weather
112,925
61
25,019 (22.2%)
Table Data Statistics (# represents the number). Label distributions: # of positive labels (positive label ratio) for classification problem, Mean (5%-50%%995% percentiles) for regression problem.
B HYPER-PARAMETER OF THE PREDICTIVE MODELS
In this paper, we use different predictive models. For each predictive model, the corresponding hyper-parameters used in the experiments are as follows:
XGBoost: booster gbtree, max depth- 6, learning rate 0.3, number of estimators 1000, depth 6, reg alpha-
LightGBM: booster gbdt, max depth None, learning rate 0.1, number of estimators 1000, min data in leaf 20
Random Forests: number of estimators - 1000, criterion gini, max depth None, warm start False
Multi-layer Perceptron:
Number of layers 4, hidden units [feature dimensions, feature di- mensions/2, feature dimensions/4, feature dimensions/8), activation function relu, early stoping
True with patient 10, batch size 256, maximum number of epochs 200, optimizer Adam
Ridge Regression: alpha
Regression Tree: max depth 3, criterion gini
Logistic Regression: solver - lbfgs, no regularization
Decision Tree: max depth criterion gini
We follow the default settings for the other hyper-parameters that mentioned here.
13

---

Under review as a conference paper at ICLR 2020
C PERFORMANCE METRICS
Mean Absolute Error (MAE):
1.
MAE (x -)>=)/(8xxxxxxxx))--- -1)ll
) (xx) - -yv -
k=1
Local MAE (LMAE)::
L
LMAE Ex (100(x))(x))) f 1x
Σ
| (xt) - -x)))))
R2 score (Legates McCabe, 1999):
R Ex (xf*(xx)) Po(x2) (x')11
E (x Fo(st (x E IIf Σ
Ex (x*) Ex If (x
If R 1, the predictions of the locally interpretable model perfectly match the predictions of the black-box model. On the other hand, if R² = 0, the locally interpretable model performs as similar as the constant mean value estimator. R < the locally interpretable model performs worse than the constant mean value estimator.
D LEARNING CURVES OF RL-LIM
Syn1
Syn dalasel
Syn3 dalasel
Rcwerds
RL-LMM Ead ellne
RLLIM
RL-M
Beselinc
Beseline
Iborst
Eorations
Ranatione
Figure 5: Learning curves of RL-LIM on three synthetic datasets. X-axis: The number of iter- ations on instance-wise weight estimator training, Y-axis: Rewards (LMAE of baseline (globally interpretable model)- LMAE of RL-LIMM,, higher the better.
14

---

Under review as a conference paper at ICLR 2020
E
ADDITIONAL RESULTS
E.1 REGRESSION WITH SHALLOW REGRESSION TREE AS THE LOCALLY INTERPRETABLE
MODEL
Datasets
Models
XGBoost
LightGBM
MLP
RF
(RT-MAE)
Metries
MAE
R²
MAE
R²
MAE
R²
MAE
R²
Original
5,131
1.0
4.965
1.0
4.939
1.0
5.203
1.0
RL-LIM
5,121
.8242
4.778
.8939
4.587
.6375
4.652
.8990
Blog (5.955)
LIME
11.80
.2658
13.22
.1483
7.396
-66001
19.61
-.116
SILO
5.149
.8035
4.818
.8166
4.649
.6177
4.715
.8774
MAPLE
5.329
.7991
5.024
.6600
4.609
.6339
5.016
.8201
Original
24.18
1.0
20.22
1.0
18.36
1.0
30.09
1.0
RL-LIM
21.82
.9077
21.35
.9944
18.56
.8832
22.44
.7236
Facebook (22.28)
LIME
36.69
.3278
44.21
.1809
40.85
-.1513
51.70
.2301
SILO
22.42
8655
22.33
.7235
19.57
8566
24.41
.6917
MAPLE
22.15
.8824
23.43
.5811
20.32
.8035
27.12
.3134
Original
2995
1.0
3140
1.0
2255
1.0
3378
1.0
RL-LIM
2938
.9822
2504
.4104
2226
.9166
2431
.2768
News (3093)
LIME
6272
-.6267
7737
-2.660
2390
.0013
9637
-7,075
SILO
2910
.1020
2854
.3461
2274
.8201
2874
.2278
MAPLE
2968
.9288
2846
.3631
2284
.8221
2888
.1872
Table 4: Real-world regression dataset results. Overall prediction performance (metric: MAE, lower better) and fidelity (metrie: R² score, higher is better) regression problems with shallow regres- sion tree as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. MAE of global regression tree (RT) can be found below the data name. Red represents performance that is worse than global regression tree and the negative R² scores. Bold represents the best results.
Datasets
Models
XGBoost
LightGBM
MLP
RF
RL-LIM
.7530
1.358
1.273
1.413
LIME
9.160
11.16
5.006
17.461
Blog
SILO
.8325
1.379
1.178
1.934
MAPLE
1.029
1.598
1.359
2.158
RL-LIM
7.240
6,867
5.596
15.77
LIME
31.52
37.75
30.58
45.58
Facebook
SILO
8.459
9.149
6.997
18.63
MAPLE
7.985
8.644
7.290
23.17
RL-LIM
389.0
1072
116.6
957.1
LIME
4455
6243
504.0
9969
News
SILO
496.7
1214
160.6
1175
MAPLE
440.7
1201
163.6
1196
Table 5: Fidelity results (Metric: LMAE, lower the better) on regression problems with shallow regression tree the locally interpretable model. Bold represents the best results.
15

---

Under review as a conference paper at ICLR 2020
E.2
CLASSIFICATION WITH RIDGE REGRESSION AS THE LOCALLY INTERPRETABLE MODEL
Datasets
Models
XGBoost
LightGBB
MLP
RF
(LR-APR)
Metrics
APR
R
APR
R
APR
R²
APR
R²
Original
.8096
1.0
.8254
1.0
.7678
1.0
.7621
1.0
RI-LIM
.7977
.9871
.8039
.9439
.7670
.9711
.7977
.9217
Adult (.7553)
LIME
.6803
.7195
.6805
.6259
.6957
.8310
.7057
.6759
SILO
.7912
.9500
.7884
.3011
.6555
.978
.7664
9140
MAPLE
.7947
.9840
.0111
.9866
.7683
.636
.7958
.9611
Original
.7133
.7299
1.0
.7205
1.0
.7274
1.0
RL-LIM
.7140
9979
.7290
.9801
.7212
.9555
.7331
.9450
Weather (.7009)
LIME
.6376
.7898
.6392
.6873
.6395
.5321
.6387
.5133
SILO
.7134
.9888
.7281
.9733
.7220
.9977
.7277
.0224
MAPLE
.7134
.9897
.7273
.9778
.7133
.9702
.7308
.9323
Table 6: Real-world classification dataset results. Overall prediction performance (metric: APR, higher is better) and fidelity (metric: It² score, higher is better) on classification problems with ridge regression as the locally interpretable model. 'Original' is the performance of the original black-box model that the models are approximating. APR of global logistic regression (LR) can be found below the data name. Red represents the results that worse than global logistic regression and the negative R² scores. Bold represents the best results.
E.3 REGRESSION WITH RIDGE REGRESSION AS THE LOCALLY INTERPRETABLE MODEL FIDELITY ANALYSIS TERMS OF LOCAL MAE (LMAE)
Datasets
Models
XGBoost
LightGBM
MLP
RF
RL-LIM
.8679
1.135
1.432
1.651
LIME
6.534
8.037
8.207
17.01
Blog
SILO
2.220
3.046
2.393
3.909
MAPLE
.9690
1.416
1.550
1.984
RL-LIM
6.394
21.29
8.217
33.64
LIME
32.57
33.70
27.38
48.03
Facebook
SILO
19.51
30.07
11.52
40.14
MAPLE
7.664
31.25
13.31
44.38
RL-LIM
436.9
1049
74.11
905.8
LIME
3317
4766
327.4
8828
News
SILO
657.2
1253
79.85
1345
MAPLE
500.5
1261
88.19
1157
Table 7: Fidelity results (Metric: LMAE, lower the better) on regression problems with ridge re- gression the locally interpretable model. Bold represents the best results.
16

---

Under review as a conference paper at ICLR 2020
E.4 COMPARISON TO DIFFERENTIABLE WEIGHTING BASELINES
In this subsection, we additionally compare RL-LIM to two baselines that have differentiable ob- jective for weighting: (1) straight-through estimator (STE) (Bengio al. 2013). (2) Learning to Reweight (1.2R) (Ren et al.) 2018). The experimental setup is the same with the synthetic experi- ments Section 4.1 and the performance metric is Average Weight Difference (AWD). As can be seen from Table g the first approach, STE, converges fast but to suboptimal AWD, whereas L2R overfits to the fidelity metric and cannot guide weighting of the training samples properly, eventually yielding poor AWD.
AWD Syn1
Syn2
Syn3
RL-LIM
0.1562
0.3325
0.3920
STE
0.1717
0.3601
0.4307
L2R
0.7532
0.7283
0.7506
Table 8: Average Weight Differences (AWD) of RL.LIM, STE, and 1.2R on three synthetic datasets. Bold represents the best results.
The sampler block in RL-LIM makes the optimization problem of the RL-LIM non-differentiable. The main motivation of using the sampler in RL-LIM is to encourage exploration in order to sys- tematically explore the extremely large action space. When we utilize traiigh-tthruugh estimator to make the entire process differentiable, the model converges faster but to suboptimal solution due to the under-expooration of the action space.
The main difference between L2R and RL-LIM is that L2R learns weights of the training samples that are the same across all validation and testing samples; on the other hand, RL-LIM uses a single instance-wist weight estimator model to learn different weights of the training samples for each probe samples. If we apply I.2R to this framework, we need separately apply L,2R for each probe sample. Therefore, there would be a high likelihood of overfitting to the probe dataset because it would only use one validation sample to learn the weights of the training samples.
E.5 SAMPLE COMPLEXITY ANALYSIS
In this subsection, we report the sample complexity analysis results of RL-LIM, comparing to: (1) randomly selected subsamples (Random), (2) STE-based weighting (STE). We vary the number of training samples (from 200 2000) and compute the AWD on three synthetic datasets.
Somple completttyyyyyyccoooooooooooyyy yno betu
RLUM
5.6
0.4
610
RL LI
Panzon
bizz
0.0
E48
RLAU
0.-
Raidon
150
Figure 6: AWD performances in terms of the number of training samples used to train three models (RI-LIM, STE, Random) Lower the better.
As can be seen in Fig. with randomly selected subsamples, only works well when we have an extremely small number of training samples due to a smaller chance of overfitting. For STE-based model, it converges fast; thus, it works better with smaller training samples; however, for larger training samples, it performs worse than RL-LIM due to under-exploration of the action space.
17

---

Under review as a conference paper at ICLR 2020
F ANALYSIS OF INSTANCE- WISE WEIGHTS DISTRIBUTIONS
In this section, we analyze the instance- wise weights of training samples visualizing the distributions of the instance-wise weights of training samples for the entire probe samples, and the dependence of the average instance-wis wise weights of training samples on the distance from the probe sample.
Instance-wise weight distribution analysis Syn1
Instance-wise weight distribution analysis Syn2
The number of training samples
175
samples
120
150
100
125
100
20
0-
0.0
0.3
0.0
0.1
02
0.3
0.4
0.5
0.6
0.7
Average weights
Average weights
(a)
(b)
weight distribution analysis Syn3
Instance-wise weight distribution analysis by distance
250
Syn1
The number training samples
Syn2
200
Syn3
150
100
0.0
0.1
02
0.3
0.4
0.5
0.6
0.7
0.8
0.2
0.4
0.6
0.8
10
Average weights
Distance data (percentile)
(c)
Figure 7: (a) Instance- wise weight distributions Syn1, (b) Instance- wise weight distributions Syn2. (c) Instance-wise weight distributions Syn33, (d) Average instance-wise weights of training samples in terms of distance from the probe sample (percentile)
As can be seen in Fig. (a)-c), the instance-wise weights have quite skewed distribution. Some samples (e.g. with average instance-wise weights above 0.5) are much more critical to interpreting the probe sample than many others (e.g. average instance- wise weights below 0.1)
In Fig. (d), there clear trend that the samples near the probe sample have higher average instance-wis weights, which shows that RL-LIM learns the meaningful distance metrics to measure the relevance while interpreting the probe samples. This trend consistent across all three synthetic datasets.
18


Reviews:
[{"review_id": "BJxA1JhCtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors aim to learn a locally interpretable model via the reinforcement learning approach, to address the fundamental challenge which is that the previous locally interpretable model has smaller representation capacity than black-box models, and causes under-fitting with conventional distillation techniques. Overall speaking, the paper is well organized, and the proposed approach is well tested, but in my opinion, there is a conceptual error.\n\nYou claimed your method is REINFORCEMENT LEARNING based, but the REINFORCEMENT LEARNING definition for your task is weird, or wrong. In section 3, you didn't give an explicit explanation for the state transition. With your given RL-like objective function, it seems that the state transition is from features to features. However, there is no specific correlated explanation in your paper on why you make such an assumption. Besides, the state transition in RL relies on decision making at each time step, while it has not reflected in your paper and code, namely, the state-transition independents on the decision making.\n\nTo sum up, I don’t think the proposed method is RL-based, it would be more appropriate to define it as a MAB problem, and this paper should solve this problem before publishing.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eu4ZJAtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of interpreting predictions of blackbox models. In particular, they study local interpretable models, which are used to study interpretability at the level of one or a few data points. A key challenge is that in order for local models to be interpretable, they need to be simple in form and therefore lower in capacity (i.e. linear); thus, if they are trained on entire datasets they will underfit.\n\nThe aim of this work is to address this issue by learning how to select representative samples from a dataset for training local linear models to reproduce predictions of black-boxes. In particular, they propose to use RL to learn to weight instances from datasets; RL is required as they make hard (i.e. non-differentiable) decisions to select a subset of the dataset. \n\nThis work is closely related to Ren et al. 2018 [1], which proposes to meta-learn how to weight samples in a batch so as to maximize performance on a validation set. By analogy, in this paper, the samples are taken from the entire dataset (i.e. the dataset is subsampled), while the validation loss is effectively an imitation loss formed by treating the black-box model predictions as a target. The data subsampling operation introduces the added complication of non-differentiability. The contribution of this paper is thus the use of RL for meta-learning how to subsample a larger dataset in order to maximize some validation loss. \n\t\t\n* Pros:\n\t* Considers an interesting dataset subsampling variant of the sample weighting meta-learning problem.\n\t* Novel application of meta-learning for improving locally-linear models.\n\t* Extensive quantitative evaluation shows that the method seems to perform better than baselines, though it might be that a differentiable approximation could do as well while being more sample efficient.\n\t* It is a nice result that the l1 penalty actually works well in reducing the number of samples chosen by the \n\t* I found the discussion and figures presented in 4.2 to be quite nice and informative.\n\n* Cons:\n\t* Given the lack of a differentiable approximation baseline, I am not entirely convinced that the use of RL is absolutely necessary/optimal.\n\t\t* I.e. if the weighting function is actually high-entropy, randomly sampling a (large) batch and weighting it might work just as well.  \n\t* Though there is discussion of the complexity of the overall method it would be nice to see a discussion and figures related to the sample efficiency of REINFORCE?\n\t\t* This would be strongest if given with a comparison to differentiable alternatives (mentioned above) as well.\n\t\t* This would help elucidate whether RL is optimal in this setting: fitting a linear model on more data might be cheaper learning to subsample with REINFORCE. \n\t* While the sample weighting function is fast at inference time, most of the overhead comes at training time. This function needs be updated in settings where the underlying dataset changes.\n\t* This is a minor issue, but this pushes the burden of interpretability further up to the black-box sample weighting function. While this interpretability problem is less critical, it still exists.\n\n* Other comments/requests:\n\t* While the use of RL is certainly motivated in order to solve the problem in an unbiased way, it would be nice to see a comparison to a differentiable approximation as a baseline? A few ideas:\n\t\t* Randomly sample a (possibly large batch) and learn to weight it (closely related to the straight through estimator)\n\t\t* Randomly sample a batch and apply [1]\n\t* Would be nice to show the sizes of datasets and how many samples end up being used for different values of lambda.\n\t* Would be nice to understand which samples are chosen and why. This is probably tricky to analyze, but it would be interesting to see if certain samples are often chosen, or if the weighting distribution has an interesting shape (i.e. is low or high-entropy).\n\nI’ve given a weak accept, conditioned on being provided more evidence regarding 1) comparisons to simple differentiable alternatives, 2) sample efficiency of the RL method, and 3) basic analysis of the weighting function. \n\n[1] Learning to Reweight Examples for Robust Deep Learning. Mengye Ren, Wenyuan Zeng, Bin Yang, Raquel Urtasun. https://arxiv.org/abs/1803.09050\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to find locally interpretable models, such that the local models are fit (w.r.t. the ground truth) and faithful (w.r.t. the global underlying black box model).
The contribution of the paper is that the local model is trained from a subset of points, selected via an optimized importance weight function. The difference compared to Ren et al. (cited) is that the IW function is non-differentiable and optimized using Reinforcement Learning. 

A first concern (Rev#1, Rev#2) regards the positioning of the paper w.r.t. RL, as the actual optimization method could be any black-box optimization method: one wants to find the IW that maximizes the faithfulness. The rebuttal makes a good job in explaining the impact of using a non-differentiable IW function.

A second concern (Rev#2) regards the interpretability of the IW underlying the local interpretable model. 

There is no doubt that the paper was considerably improved during the rebuttal period. However, the improvements raise additional questions (e.g. about selecting the IW depending on the distance to the probes). I encourage the authors to continue on this promising line of search. 
Decision: Reject