LIMIS: Locally Interpretable Modeling using Instance-wise Subsampling
Authors: Jinsung Yoon
Sercan O Arik
Tomas Pfister
Venue: Accepted by TMLR ()

Abstract: Understanding black-box machine learning models is crucial for their widespread adoption. Learning globally interpretable models is one approach, but achieving high performance with them is challenging. An alternative approach is to explain individual predictions using locally interpretable models. For locally interpretable modeling, various methods have been proposed and indeed commonly used, but they suffer from low fidelity, i.e. their explanations do not approximate the predictions well. In this paper, our goal is to push the state-of-the-art in high-fidelity locally interpretable modeling. We propose a novel framework, Locally Interpretable Modeling using Instance-wise Subsampling (LIMIS). LIMIS utilizes a policy gradient to select a small number of instances and distills the black-box model into a low-capacity locally interpretable model using those selected instances. Training is guided with a reward obtained directly by measuring the fidelity of the locally interpretable models. We show on multiple tabular datasets that LIMIS near-matches the prediction accuracy of black-box models, significantly outperforming state-of-the-art locally interpretable models in terms of fidelity and prediction accuracy. 

Published in Transactions on Machine Learning Research (09/2022)
LIMIS: Locally Interpretable Modeling using Instance-wise Subsampling
Jinsung Yoon, Sercan Ö. Arik, Tomas Pfister Google Cloud AI
(jinsungyoon, soarik, (pfister) (aaaleeer@goggee.oo
Reviewed on OpenReview: https: o/peerv//peereveeu net/ formm? id- S3eABAy8P3
Abstract
Understanding black-box machine learning models is crucial for their widespread adoption. Learning globally interpretable models is one approach, but achieving high performance with them is challenging. An alternative approach is explain individual predictions using locally interpretable models. For locally interpretable modeling, various methods have been proposed and indeed commonly used, but they suffer from low fidelity, i.e. their explanations do not approximate the predictions well. In this paper, our goal is to push the state-of-the-arr in high-fidelity locally interpretable modeling. We propose a novel framework, Locally Interpretable Modeling using Instance-wise Subsampling (LIMIS). LIMIS utilizes a policy gradient to select a small number of instances and distills the black-bos. model into a low-capacity locally interprotable model using those selected instances. Training is guided with reward obtained directly by measuring the fidelity of the locally interpretable models. We show multiple tabular datasets that LIMIS near-matches the prediction accuracy of black-box models, significantly outperforming state-of-thc-art locally interpretable models in terms of fidelity and prediction accuracy.
1 Introduction
In many real-world applications, machine learning is required to be interpretable doctors need to understand why a particular treatment is recommended, banks need to understand why loan is declined, and regulators need to investigate systems against potential fallacies (Rudin 2018). On the other hand, the machine learning models that have made the most significant impact via predictive accuracy improvements, such as deep neural networks (DNNs) and ensemble docision tree (DT) variants (Goodfellow et al.) 2016 He et al. 2016 Chen & Guestrin 2016 Ke et al., 2017), are "black-box' nature their decision making is based on complex non-linear interactions between many parameters that are difficult interpret. Many studies have suggested
trado-off between performance and interprotability (Virág & Nyitrai, 2014 Johansson et al. 2011 Lipton
2016). While globally interpretable models such as linear models or shallow Decision Trees (DIs) have simple explanations for the entire model behaviors, they generally yield significantly worse performance than black-box models.
One alternative approach locally interpretable modeling explainiig a single prediction individually instead of explaining the entire model (Ribeiro et al. 2016). globally interpretable model fits a single interpretable model to the entire data, while a locally interpretable model fits an interpretable model locally, i.e. for each vistance/sample individually, by distilling knowledge from black-box model around the observed sample. Locally interpretable models are useful for real-world Al deployments by providing succinct and human-like explanations via locally fitted models. They can be utilized to identify systematic failure cases (e.g. by seeking common trends in how failure cases depend on the inputs) (Mangalathu el al. 2020), detect biases (e.g. by quantifying the importance a particular feature) (ElShawi et al. (021), provide actionable feedback to improve a model (c.g. suggesting what training data collect) (Ribeiro et al. 2016), and for counterfactual analyses (c.g. by investigating the local model behavior around the observed data sample) (Grath al. 2018

---

Published in Transactions on Machine Learning Research (09/2022)
Test sample
Agc
Cup. gaia
Education
Geader
High
Labeled training
Locally-interpretalle model (DT)
Age
Cap- gain
Edocsiion
Geader
Iscome Berget)
Instance-wise weight estimator (DNN)
Weights
<550k
60
300000
M
x5506
0.1
Educoton
45
1009
Dattre
M
0.3
Weighted optimization
BBacheloos
Bachhiun
Edusttion <Bschelos & Age <33
21
Hub-schoo
F
-3506
>S50k
Age
5000000
Bachilors
M
>5966
2550K
<S5UK
Figure 1: LIMIS example for the income classification task. For each test sample, the most valuable training samples are chosen to fit. the locally-interpretable model (DT here), and it provides human-like explanations to the decision. More use-cases for
AI capabilities of LIMIS can be found in Sect. 0
Various methods have been proposed for locally interpretable modeling: Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al. 2016), Supervised Local modeling methods (SILO) (Bloniarz et al. 2016). and Model Agnostic Supervised Local Explanations (MAPLE) (Plumb et al. 2018). LIME in particular has gained siguificant popularity. Yet, the locally interpretable modeling problem is still far from as being solved. To be useful in practice, a locally interpretable model should have high fidelity, i.e, should approximate
the 'black-bax' model well (Plumb et al. 2019; Lakkaraju al. 2019 Recent studics have shown that
LIME incled often yields low fidelity MAlvarez-Melis & Jaakkola 2018 Zhang et al., 2019 Ribeiro et al.
2018 Lakkaraju et al. 2017); indeed, as we show in Sec. 5 in some cases, LIME's performance is even worse than simple globally interpretable models. The performance of other methods such as SILO and MAPLE are also far from the achievable limits. Overall, locally interpretable modeling while ensuring high fidelity across a wide range of cases is an everlasting challenging problem, and we propose that requires a substantially-novel design for the fitting paradigm. A fundamental challenge to fit a locally interpretable model is the representational capacity difference when applying distillation. Black-box models, such as DNNs ensemble DTTs, have much larger capacity compared to interpretable models. This can result in underfitting with conventional distillation techniques and consoquently suboptimal performance (Hinton et al.
2015 Wang
ct al. 2019
To address the fundamental challenges aforementioned above, we propose a novel instance-wist subsampling method to fit Locally Interpretable Models, named LIMIS, that is motivated by meta-learning (Ren et al., 2018). Fig. [ depicts LIMIS for the income classification task. LIMIS utilizes the instance-wise weight estimator to identify the importance of the training samples to explain the test sample. Then, it trains locally-interpretable model with weighted optimization to return the accurate prediction and corresponding local explanations. LIMIS efficiently tackles the distillation challenge by fitting the locally interpretable model with a small number of pustancee/sampllee that are determined to be most valuable to maximize the fidelity, Unlike alternative mothods that apply some supervised lcarning approaches to determine valuable instances, LIMIS learns an instance-wise weight estimator (modeled with DNN) directly using the fidelity metric for selection. Accurate determination of the most valuable instances allows the locally interpretable model to more effectively utilize its small representational capacity. At various regression and classification tasks, we demonstrate that LIMIS significantly outperforms alternatives. In most cases, the locally interpretable models obtained by LIMIS near-match the performance of the complex black-box models that they are trained to interpret. In addition, LIMIS offers the instance- based explainability via ranking of the most valuable training instances. We also show that the high-fidelity explanations can open new horizons for reliable counterfactual analysis, by understanding what input modification would change the outcome, which can be important for human-in-the-loop AI deployments (see Sec. 6.2
2
Related Work
Locally interpretable models: There are various approaches to interpret black-box models (Gilpin et al. 2018). One is to directly decompose the prediction into feature attributions, e.g. Shapley values (Strumbelj & Kononenko 2014) and their computationally-effceenn variants (Lundberg & Lee. 2017 Others based
2

---

Published in Transactions on Machine Learning Research (09/2022)
activation differences, e.g. DeepLIFT (Shrikumar et al. 2017 saliency maps using the gradient flows, e.g. CAM (Zhou et al. 2016) and Grad-CAM (Selvaraju et al. 2017). In this paper, we focus on the direction of locally interpretable modeling distilling a black-box model into an interpretable model for each instance in tabular domains. LIME (Ribeiro et al. 2016 is the most commonly used method for locally interpretable modeling in labular domains. LIME is based on modifying the input. feature values and learning from the impact of the modifications on the output. A fundamental challenge for LIME is the meaningful distance metric to determine neighborhoods, as simple metrics like Euclidean distance may yield poor fidelity, and the estimation is highly sensitive to normalization -elvasez-Melis & Jaakkola 2018 SILO (Bloniarz et al., 2016) proposed a nonparametric regression based on fitting small-scale local models which can be utilized for locally interpretable models similar to LIME. It determines the neighborhoods for each instance using tree-based ensembles it utilizes DT ensembles to determine the weights of training instances for each test instance and uses the weights to optimize a locally interpretable model. Note that these weights are independent of the locally interpretable modcls. MAPLE (Plumb et al.) 2018) further adds feature selection on top of SILO. SILO and MAPLE optimize the DT-based ensemble methods independently and this disjoint, optimization results in suboptimal performance. To fit a proper locally interpretable model, key problem is the selection of the appropriate training instances for each test instance. LIME uses Euclidean distances, whereas SILO and MAPLE use DT-based ensemble methods. Our proposed method, LIMIS, takes a very different approach: to efficiently explore the large search space, we directly optimize the instance-wise subsampler with the fidelity as the reward.
waig-ted training: Optimal weighting of training instances is paramount problem in machine learning. By hwwph//igg//wwwwwihhhhhnngg the high/low value instances, better performance can be obtained in certain sccnarios, such with noisy labels (Jiang et al. 2018). One approach for data weighting is utilizing influence functions (Koh & Liang, 2017), that are based on oracle access to gradients and Hessian-veclor products. Jointly-trained student-teachhrr method constitutes another approach (Jiang et al.] 2018 Bengio ct al. 2009) to learn a dlta-ddivenn curriculum. Using the feedback from the teacher, instance-wise weights are learned by the student model. Aligned with our motivations, meta learning is considered for data weighting in Ren el al. (2018). Their proposed method utilizes gradient descent-based meta learning, guided by a small validation set, to maximize the target performance. LIMIS utilizes data- weighted training for novel goal: interpretability. Unlike gradicnt bescent-basod meta lcarning. LIMIS uses policy gradient and integrates the
fidelity metric as the reward. Aforementioned works (Jiang et al.) 2018: Koh & Liang] 2017 Bengio et al. 2009 Ren et al. 20188) estimate the same ranking of training data for all instances. Instead, LIMIS yields an instance-wise ranking of training data, enabling efficient distillation of a black-box model prediction into a locally interpretable model. Yeh et al. (2018) can also provide instance-wiet wise ranking of training samples but for sample-based explainability. Differently, LIMTS utilizes instance-wise ranking with the objective of hocall--ntteppreabble modeling.
3 LIMIS Framework
Consider a training dataset D = (xx, w)) ~ P for a black-box model f, where x; E X are d-dimensional feature vectors and y: y' are the corresponding labels. We also assume a probe dataset DD {(x]) 1/) P, to evaluate the model performance to guide meta-learning as in Ren et al. (2018). Tf there is no explicit. probe datasct, it can be randomly split from the training datasct (D).
3.1 Training and inference
LIMIS is composed of: (i) Black-box model f:X y any machine learning model to be explained (e.g. DNN), (ii) Locally interpretable model G0 : X + y - an inherantlyinteepreaabli model by design (e.g. a shallow DT), (iii) Instance-wise weight estimation model he : X x X' x y + [0, 1] a function that outputs the instance-wise weights to fit the locally interpretable model, specifying for each instance how valuable is for training the locally interpretable model. It takes its input as the concatenation of a probe instance's feature, a training instance's feature, and a corresponding black-box model prediction. It can be a complex MI. model here DNN.
3

---

Published in Transactions on Machine Learning Research (09/2022)
Grey block: Trainable model
White block: Fixed model
Probe sample
Black-box model
weight estimatur
Instance-wise
XP
Thining ditaect (2)
Testing sample
Selection probubiiitiis
Instance-wise
weight estimator
Loss
Rounbrrsmen aqual
Sampler
Stage 0: Black-box model training
Selection probubilities
Ws
Selection vector
Weishted
Black box madel
x
Interpretable model
D.
0.6"
Wiigltsd optimizaaiinn
Interpretable model
Instanco-wise
Interpretable predictions
explanations
Stage I: Auxiliary dataset construction
Reward
(x9,9")
Stage Instance-wise weight estimator training
Stage 3: Interpretable inference
Figure 2: Block diagram of the proposed method. White blocks represent fixed (not learnable) models, and grey blocks represent trainable (lcarnable) models. Stage 0: Black-box model training. Stage 1: Auxiliary dataset construction. Stage 2: Instance-wise weight estimator training. Stage 3: Interpretable inference.
Our goal to construct an accurate locally interpretable model go such that the prodiction made by is similar to the prediction of the trained black-box model f i.e. the locally interpretable model has high fidelity. We use loss function, L 3 x y - R to quantify the fidelity of the locally interpretable model which measures the prediction differences between black-box model and locally interpretable model (e.g. in terms of mcan absolute error).
The locally interpretable model has a significantly lower representational capacity compared to the black-box model. This is the bottleneck that LIMIS aims to address. Ideally, to avoid underfitting, such low-capacity interpretable models should be learned with a minimal number of training instances that are most effective in capturing the model behavior. We propose an instance-wise weight. estimation model h& to output the likelihood of each training instance being used for fitting the locally interpretable model. Integrating this with the goal of training an accurate locally interpretable model yields the following objective:
(x²)) + AI xxx [ x, (x))] s.t. Do(rr) arg mingo Exxpx [x x, (x)) L. 90 (x),
min
ExP-Px [
hg
(1)
where À 0 is paraamee to control the number of training instances used to fit the locally interpretable model, and ha(x", x, fx)())) is the weight for each training pair (x. fxx)) and for the probe data x". Cg is the loss function to fit the locally interpretable model (here to minimize the mean squared error) between the pre- dicted values for regression and logits for classification. & and 0 are the trainable parameters, whereas f (the pre-trained black- box model) is fixed. The first term in the objective function Expepx [[[f (x"). "x(x)>>)))))) (xx)) is the fidelity metric, representing the prediction differences bet the black-box model and locally in- terpretable models. The sccond term in the objective function xxxxxx [xx,,, x, [x))) represents the expected number of selected training points to fit the locally interpretable model. Lastly, the constraint ensures that the locally interpretable model is derived from weighted optimization, where weights are the outputs of hg- Our formulation does not assume any constraint on go it can be any inherently interpretable model. In experiments, we use simple decision tree or regression model (with closed-form solution) so that the complexity of the constraint optimization is negligible. Note that we utilize a deep model for weight, optimization (h₂) but a simple interpretable model for explanation (ga). LIMIS encompasses 4 stages:
Stage 0 Black-box model training: Given the training set D, the black- box model f trained minimize loss function Ly (e.g. mean squared error for regression or cross-entropy for classífication), iee.,
4

---

Published in Transactions on Machine Learning Research (09/2022)
f* arg miny Nv Ei=1 Ly(f(x),,))) and retrieve the given pre-trained model as s"
ya). If there exists pre-trained black-box model, we can skip this stage
Stage Auxiliary dataset construction: Using the pre-trained black-box model f"; we create auxiliary training and probe datasets, as D = ((x,, I.). = 1, I,,,,N. N. (where 0s = f (xx)) and Dr = (x3,0f); l,... M} (where Jj = (x respectively. These auxiliary datasets (D, DP) are used for training the instance-wise weight estimation model and locally interpretable model.
If we want to understand the black-box models that are trained the given datasets, Stage 0 and Stage 1 are necessary because the main objective is to understand the local dynamics of the black-box models' decision boundary. On the other hand, if we want to understand the local dynamics of the given datasets, we can skip Stage 0 and I and directly utilize the given dataset for Stage 2 and 3.
Stage 2 Instance-wiss wise weight estimator training: LIMIS employs an instance-wise weight estimator output selection probabílitics that yield the selection weights, and the selection weights determine the fitted local interpretable model via weighted optimization. We train the instance-wise weight, estimator using the auxiliary datasets (D, DP). The search space for all sample weights would be very large, and for efficient search, proper exploration is crucial. To this end, we consider probabilistic selection with sampler block that is based on the output of the instance-war wise weight estimator - hs (xf,, x,, dt) represents the probability that (x; (t) is selected to train a locally interpretable model for the probe instance xy. Let the binary vector c(x²) E [0,1]* 1 represent the selection vector, such that (x,, 0:) is selected for x when cxx3))= =1
Now, we convert the intractable optimization problem in Eq. 4 with the following approximations:
(i) The sample mean is used as an approximation of the first term of the objective function:
M
Ex [x
(x*))
1
Σ
(()((x)),
(0(e)) (xf)))
M
(ii) The second term of the objective function, which represents the average selection probability, is approxi- mated as the average number of selected instances:
M N
1
Xx x x,
les
(xf)l
MN
==
(iii) The objective of the constraint term is approximated using the sample mean of the training loss as
1 N
Ex-Py [ x, (x)) (x(f"(x),
= N
Σ [xx3)) Lg
(xx) 90(x:))
The converted tractable optimization problem becomes:
1 [C(S"(")) (x)) Doseg) (xg)) + x ΣN (x,l]
min he
s.l. F0x2 arg ming 1/2 Ei (xxx)) Lg (x)) go (x:)) where (x(x)) Ber(ho (Na((xx, Xi, l (x:)))
The sampler block yields non- non-differential objective the optimization is over c(x²) E (0, 13 weighted instances, and we cannot use conventional gradient descent-based optimization to solve the above optimization problem. Motivated by its successful applications (Ranzato et al. 2015 Zaremba & Sutskever 2015 Zhang
& Lapata 2017), we adapt the policy-gradient based REINFORCE algorithm (Williams) 1992) such that the selection action is rewarded by its impact on performance. We consider the loss function
1 M
1(0) = M Σ (x00) (x +
N Σ cxx")
States the features of input instances, actious the sclection vectors from he (policy) that selects the most valuable samples, and reward the fidelity the locally interpretable model compared the black box model which depends on the input features (state) and the selection vector (action).
5

---

Published in Transactions on Machine Learning Research (09/2022)
the reward given the state and action for the selection policy
Correspondingly, Pa(xx c(x))) is the probability mass function for c(x²) given ha(-):
N
Polxy, c(xg))
II
fxx
(1 (1-ha(xe)) (xx
To apply the REINFORCE algorithm, directly compute its gradient with respect to Φi
1 M [x(f 90zg) +
Vollo)
(x(x))]] V log (xf,,
Bringing all this together, we update the parameters of the -sstacce-wise weight estimator with the following steps (bi-level optimization) repeated until convergence:
(i) Estimate instance-wise weights wx(x)) = has (xf. x;, fi) and instance selection vector e, cx(x)) Ber(uxxx]))
for each training and probe instance in mini-bater (Nms is the number of samples in a mini batch).
(ii) Optimize the locally interpretable model with the selection for each probe instance:
Nuch
Bo(x7) arg min
Σ (xg()
(3)
(iii) Update the instance-ve wise weight estimation model (where a > 0 is a learning rate):
M
N
a M
(x)( (x
la(x5))
V log P& (x),()xxc(x
(4)
Pseudo-code of the LIMIS training is in Algorithm. We stop training LIMIS algorithm if there are no fidelity improvements. Hyper-parameters are optimized to maximize the validation fidelity.
Algorithm LIMIS Training
Input: Training data (D), probe data (DP), black-box model (f*)
1: Initialize ha-
Construct auxiliary data (D, (D::)): D {(x,, fxx)) Do (())))
3: while hg is not converged do
4: Estimate wxxxx) = hx x;, 9:)
5: Sample (xx)) Ber w, (w;()))
6: Optimize locally interpretable models with es (x²)) using Eq. (3)
Update he using Eq. (4)
end while
Output: Trained instance-wise weight estimator (h²)
Stage Interpretable inference: Unlike training, we use fized instance-wise weight estimator without the sampler. Note that we use the probabilistic selection for encouraging the exploration during the training. At inference time, exploration is no longer needed; it would be better to minimize the randomness to maximize the fidelity of the locally interpretable models. Given the test instance x', obtain the selection probabilities from the instance-wise weight estimator, and using these the weights, we fit the locally interpretable model via weighted optimization. The outputs of the fitted model are the instance-wiss predictions and the corrosponding explanations (c.g. coofficients for a lincar model). Pseudo-codo of the LIMIS inference in Algorithm.
2Other desired properties, such as robustness explanations against. input perturbations, can be further added the reward the flexibility constitutes the major advantages.

---

Published in Transactions on Machine Learning Research (09/2022)
Algorithm 2 LIMIS Inference
Input: Training data (D), test sample (x'); trained instanco-wisc weight estimator (h²)
Estimate wx(xx') = h3 (Xx, x,, Ds)
Optimize locally interpretable model using instance-wise weights wy(xx) via weighted optimization: (x(x))
3: (0(x)) = arg ming, Ei=1 wx(xx)
Output: Predictions ( (x')), explanations (Doxe)) and instance- weights {xx(x'))
3.2 Computational cost
As a representative and commonly used example, consider a linear ridge regression (RR) model as the locally interpretable model, which has a computational complexity of O(d + (dd)) to fit, where d is the number of features and N the number of training instances. When N >> d (which is oflen the case in practice), the training computational complexity is approximated as (dd TTn 2018).
Training: Given pre-trained black-box model, Stage 1 involves running inference N times and the total complexity is determined by the black-box model. Unless the black-box model is very complex, the computational cost of Stage 1 is much smaller than Stage 2. At Stage 2, we iteratively train the instance- wise weight estimator and fit the locally interpretable model using weighted optimization. Therefore, the computational complexity is Odd NT) where N, is the number of iterations (lypically N₁ < 10, 000 until convergence). Thus, the training complexity scales roughly linearly with the number of training instances.
Interpretable inference: To infer with the locally interpretable model, we need to fit the locally interpretable model after obtaining the instance-wise weights from the trained instance-wis weight estimator. For each testing instance, the computational complexity is O(d
Experimental results on the computational cost for both training and inference can be found in Sect 5.3
4 Synthetic Data Experiments
Evaluations of explanation quality are challenging on real-world datasets due to the absence of ground-truth explanations. Therefore, we initially perform experiments synthetic datascts with known ground-truth explanations lo directly evaluate how well the locally interpretable models can recover the underlying reasoning behind outputs.
We construct three synthetic datasets that have different local behaviors in different input regimes. The 11-dimensional input features X are sampled from N/ I) and Y are determined as follows:
Synl: Y X1 2Xa X10 < <0&
X3 2X4 if X 2 0,
Syn2: Y X1 + 2X, if X10 XI < 1 & Y = X3 + 2X, if X10 He X11 2 1,
Syn3: Y = X; + 2X, if X10 + (Xn))² < 0 & Y = X3 + 2X, if X10 + (X11)3 2 2 0.
Y directly dependent to XI,..., X, and not directly dependent to X10. X11. However, X10, X11 determine how Y are dependent on X1,...,X; X,, For instance, Syn1 dataset, Y is directly dependent with X1, X2 if X is nogative. If X10 is positive, is directly dependent with X3, X, but independent of X2, X2. Additional results with non-linear feature-label relationships can be found in Appendix. H.7
We directly use the ground truth function as the black box model instead of a fitted nonlinear black-box model to solely focus on LIMIS performance, decoupling from the nonlincar black-box model fitting performance. We quantify how well locally interpretable modeling can capture the underlying local function behavior using the Absolute Weight Difference (AWD) metrie: AWD = ||w - ww], where w is the ground truth linear coefficients to generate Y given X and w the estimated coefficient from the lincar locally interpretable model (RR our experiments). To make the experiments consistent and robust, we use the probe sample as the criteria to determine the ground-truth local dynamics (w). We report the results over 10 independent.
7

---

Published in Transactions on Machine Learning Research (09/2022)
runs with 2,000 samples per each synthetic dataset. Additional results can be found in the Appendix. and
G
4.1
Recovering local function behavior
Syn1 Dataset
Syn2 Dataset
Syn3 Dataset
LIMIS
0.
LIMIS
LIMIS
0.4
SILO
07
MAPLE
MAPLE
06
MAPLE
02
AWD
05
04
01
02
03
80
22
42
Distance from the boundary (parcertile)
Distance from the boundary (percantile)
Distance from the boundary (percentill)
Figure 3: Mean AWD (aggregated per uniformly divided x-axis bin) with 95% confidence intervals (of 10 independent runs) on three synthetic datasets (y-axis) vs. the percentile distance from the boundary where the local function behavior change (x-axis), e.g. X10 = for Synl. We exclude LIME due to its poor performance (its AWD higher than 1.6 in all cases distance regimes for all datasets). LIME results and the scatter plots of LIMIS can be found in the appendix.
We compare LIMIS to LIME {Ribeiro et al. 2016), SILO (Bloniarz et al. 2016). and MAPLE (Plumb et al, 2018). Fig. 3 shows that LIMIS significantly outperforms other methods in discovering the local function behavior on all three datasets, in different regimes. Even the decision boundaries are non-linear (Syn2 and Syn3), LIMIS can efficiently learn them, beyond the capabilities of the linear RR model. LIME fails to recover the local function behavior it the Euclidean distance and cannot distinguish the special properties of the features. SILO and MAPLE only use the relevant variables for the predictions; thus, it is difficult for them to discover the decision boundary that depends on other variables, independent of the predictions.
4.2
The impact of the number of selected instances
LMAE Syn1
LMAE Syn2 & Syn3
Avc: Sewoccion Prob
110
Syn1
Syn3 Syn3
Syn1
85-
Syn2
0.41
3.00
Syn3
LMAE
24-
UMAC
200
0.00
0.56
09
0.2
04
2.1
25
13
355
0.2
02
2.4
06
0.3
1.0
88 02
04
0.
10
2.
A
Figure 4: Fidelity (in LMAE) and average selection probability of training samples (y-axis) vs. À (x-axis).
Optimal distillation in LIMIS is enabled by using a small subset of training instances to fit the low-capacity locally interpretable model. The number of selected instances controlled by X if X is high/low, LIMIS penalizes more/less, thus less/more instances are selected to fit the locally interpretable model. We analyze the efficacy of in controlling the likelihood of selection and the fidelity. Fig. 4 (left and middle) demonstrates the clear relationship between À and the fidelity. If À is too large, LIMIS selects insufficient number of instances; thus, the fitted locally interpretable model is loss accurate (due to underfitting). If À is too small, LIMIS is not sufficiently encouraged lo select the most relevant instances related to the local dynamics to fit. the ridge regression model, thus is more prone to learning feature relationships that may generalize poorly. To achieve the optimal X, we conduct vals-vaaiootion and select with the best validation fidelity. Fig. [ (right) shows the average selection probability of the training instances for each X. As increases, the average selection probabilities decrease due the higher penalty the number of selected instances. Even using
8

---

Published in Transactions on Machine Learning Research (09/2022)
small portion of training instances, LIMIS accurately distill the predictions into locally interpretable models, which is crucial to understand the predictions using the most relevant. instances.
4.3
Comparison to differentiable baselines
Table 1: AWD comparisons on three synthetic datasets with different number of train samples (N).
Number train samples
=
500
N = 1000
2000
Datasets
Synl
Syu2
Syn3
Avg.
Synl
Syn2
Syn3
Avg.
Synl
Syu2
Syn3
Avg.
LIMIS
.5531
.5869
.6512
.5971
.2129
.4289
.5527
.3982
.1562
.3325
.3920
.2936
Gumbel-softnax
.4177
.5017
.5953
.5049
.2712
.4511
.5405
.4209
.1698
.3655
.4217
.3190
STE
.4281
.4911
.6001
.5074
.2688
.4007
.5372
.1566
.1717
.3601
.4007
.3208
L2R
.6758
.6607
.6003
.6756
.0689
.6412
.6217
.6339
.7332
.7283
.7506
.7440
We compare LIMIS to three basclines that have differentiable objectives for data weighting in Table | (1)
Gumbel-softmax (Jang al. 2016 3 (2) straight-through estimator (STE) (Bengio et al. 2013). (3) Learning
to Reweight (L2R) (Ren et al. 2018).
LIMIS utilizes the sampling procedure; thus, the objective (loss) function is non-differentiable. In that case, we cannot train the model in an end-to-end way using the stochastic gradient descent (SGD). Instead, there are multiple ways to train with non-differentiable objectives. In the LIMIS framework, we use the REINFORCE algorithm to train the model with nonfiifferentiable objective. Gumbel-softmax is another approximation method to convert non-differentiable sampling procedure the differentiable softmax outputs. STE is replacing the sampling procedure with direct weighted optimization. Both STE and Gumbel-softmax baselines are differentiable approaches and we can optimize the models in an end-to-end way via SGD.
We observe that Gumbel-softmax and STE converge faster but to suboptimal solution, due to under- exploration. L2R overfits to the fidelity metric and cannot guide weighting of the instances accurately, yielding poor AWD. Because L2R learns the same weights across all instances, whereas LIMIS uses an instance-wise weight estimator to learn instance-wise weights separately for each probe instance. In Table Gumbel-softmax and STE models outperform LIMIS only if N 500 (in the regime of extremely small number of training instances), given their favorable inductive bias with gradient-descent based optimization (that also yields faster convergence). However, with N 1000, 2000, they underperform LIMTS due to the under-exploration. More specifically, the average performance improvements with LIMIS is 4.2%, 6.4% (with respect to N = 1000, 2000) in comparison with the best alternative. As seen in Fig, g the performance gap between LIMIS and alternatives increases as the number of training samples increases.
5 Real-world Data Experiments
We next study LIMIS on 3 real-world regression datasets: (1) Blog Feedback, (2) Facebook Comment, (3) News Popularity; and 2 real-world classification datasets: (4) Adult Income, (5) Weather. We use raw data after normalizing each feature to be in [0, 1], using standard Min-Max scaler and apply one-hot encoding to categorical features. We focus on black-box models that are shown to yield strong performance on target tasks. We implement the instance-wise weight estimator as an MLP with tanh activation. Its hyperparameters are optimized using cross-validation (5-layer MLP with 100 hidden units performs reasonably-well across all datasets). Model details the data and hyperparameters can be found in the Appendix. D and A
5.1 Performance comparisons
We evaluate the performance on disjoint testing sets D = ((xx vi)) ~ P and report the results over 10 independent runs. For fidelity, we compare the outputs (predicted values for regression and logits for classification) of the locally interpretable models and the black-box model, using Nash-Sutcliffe Elficienccy 1970 For the prediction performance, we use Mean Absolute Error (MAE) for
(NSE) Nash & Sutcliffe
We set Gumbel-softmax temperature as 0.5; we do use temperature annealing due the training instability.

---

Published in Transactions on Machine Learning Research (09/2022)
regression and Average Precision Recall (APR) for classification. Details on the metrics can be found in Appendix.
Table 2: Fidelity (metric: NSE, higher is better) and prediction performance (metric: MAE, lower better APR, higher the better) regression/ classification datasets, using RR/DT as the locally interpretable model while explaining the black box models: XGBoost (Chen & Guestrin 2016). LightGBM (Ke et al. 2017j, Random Forests (RF) (Breiman, 2001) and Multi-layer Perceptron (MLP). "Original' represents the performance of the original black-box model that the locall-innteppreabble modeling is applied on. We also model under the data name. Red: performance worse than globally-interpretable RR/DT and the negative NSE. Bold: best. results.
show the performance of RR/ DT (in terms of MAE/APR) as a
Regression Datasets
Models
XGBoost
LightGBM
MLP
RF
(Ridge Regression)
Metrics
MAE
NSE
MAE
NSE
MAE
NSE
MAE
NSE
Original
5.131
1.0
4.965
1.0
4.893
1.0
5.203
1.0
LIMIS
5.289
.8679
4.971
.9069
4.994
.7177
4.993
.8573
Blog (8.420)
LIME
9.421
.3400
10.243
.3019
10.936
-,2723
19.222
2143
SILO
6.261
.0005
6.040
.2899
5.413
.4274
6.610
.4500
MАPLE
5.307
.8248
4.981
.8972
5.012
.5624
5.058
.8471
Original
24.18
1.0
20.22
1.0
18.36
1.0
30.09
1.0
LIMIS
22.92
.7071
24.84
.4268
20.23
.5495
22.65
.4360
Facebook (21.64)
LIME
35.20
.205
38.19
.2159
38.82
.2463
51.77
.1797
SILO
31.41
-3405
39.10
-1.994
22.35
.3307
42.05
7929
MАPLE
23.28
68803
41.86
-3.233
24.77
-.1721
44.75
-1.078
Original
2995
1.0
3140
1.0
2255
1.0
3378
1.0
LIMIS
2958
.7534
2957
.5936
2260
.9761
2396
.6523
News
LIME
5141
-22477
6301
-2.008
2289
.5030
9435
-7.477
(.2989)
SILO
3069
.4447
3006
.4025
2257
.9617
3251
.8866
MAPLE
2967
7010
3005
.3963
2259
.9534
3060
.5901
Classification Datasets
Models
XGBoost
LightGBM
MLP
RF
(Decision Tree)
Metrics
APR
NSE
APR
NSE
APR
NSE
APR
NSE
Original
.8966
1.0
.8254
1.0
7678
1.0
.7621
1.0
LIMIS
.8011
.9889
.8114
.9602
.7710
.9451
.7881
.8788
Adult (.6388)
LIME
.6211
.5099
.6031
.3798
.2700
.2511
.6166
.3833
SILO
8001
.9869
.8107
.9583
.7708
.9470
.7833
.8548
MAPLE
.7928
.9794
.8034
.9405
.7719
.9410
.7861
.6222
Original
.7133
1.0
.7299
1.0
.7205
1.0
.7274
1.0
LIMIS
.7071
.9734
.7118
.9601
.7099
.9124
.7102
.9008
Weather (.5838)
LIME
.6179
.7783
.6159
.6913
.6511
.3417
.6209
.3534
SILO
.6991
.9680
.7052
.9452
.6997
.8844
.7042
.8988
MAPLE
.6973
.9675
.7556
.9446
.6983
.8856
.6983
.8856
Table shows that for regression tasks, the performance of globally interpretable RR (trained on the entire dataset from scratch) is much worse than complex black-box models, underlining the importance of non-linear modeling. Locally interpretable modeling with LIME, SILO and MAPLE yield significant performance degradation compared to the original black-box model. In some cases (e.g. on Facebook), the performance of previous work is even worse than the globally interpretable RR, undermining the use of locally interpretable modeling. In contrast, LIMIS achieves consistently high prediction performance and significantly outperlorms RR. Table 2 also compares the fidelity in terms of NSE, We observe that NSE is negative for some cases (e.g. LIME on Facebook data), implying that output of the locally interpretable model is even worse than the constant mean value estimator. On the other hand, LIMIS achieves high NSE consistently across all datasets with all blx models. Table also shows the performance on classification tasks using shallow regression
10

---

Published in Transactions on Machine Learning Research (09/2022)
DIs the locally interpretable model (Regression DIs model outputs logits for classification.). Among the locally interpretable models, LIMTS often achieves the best. APR and NSE, underlining its strength in distilling the predictions of the black-box model accurately. In some cases, the benchmarks (especially LIME) yield worse prodiction performance than the globally interpretable model, DT. Additional results can be found in the Appendix G
5.2 Local generalization of explanations
For locally interpretability modeling, local generalization of explanations is very important, as one expect a similar behavior around a meaningful vicinity of sample. To quantify the local generalization of explanations, we include evaluations with neighborhood metrics (Plumb al.) 2019), which give insights on the explanation quality at nearby points. We show the results two regression datasets (Blog and Facebook) with two black-box models (XGBoost and Ligh and evaluate them in terms of neighborhood LMAE (Local MAE) and pointwise LMAE. Here, the difference would be a mcasure of local generalization, i.c. how reliable the explanations are against imput changes. Neighborhood LMAE is defined as Ex'~Px "tt=>Px.~NN(O... -=~((0 (l + n) f + +)]li where set 0.1 the neighborhood vicinity (with standard normalization for the imputs). Pointwise LMAE is defined as Ex 1190)x)) (x') fx'))x Details the metrics can be found in Appendix. B Additional results can be also found in Appendix. H.4
Table 3: Prediction performance (metric: neighborhood LMAE and pointwise LMAE, lower is better) on regression datasets, using RR as the locally interpretable model while explaining the black box models:
XGBoost
Chen & Guestrin 2016) and LightGBM (ee el. al. 2017 Bold: best results.
Datasets
Models
XGBoost
Light
(RR)
Metrics
Neighbor LMAE
Pointwise LMAE
Diff
Neighbor LMAE
Poinfuris LMAE
Diff
LIMIS
.8894
.8679
2.48%
1.217
1.135
7.22%
Blog
LIME
6.872
6.534
5.17%
8.233
8.037
2.14%
SILO
2.368
2.220
6.68%
3.119
3.046
2.40%
MAPLE
1.007
.9600
3.96%
1.422
1.416
1.87%
LIMIS
6.533
6.394
2.18%
8.250
8.217
0.11%
Facebook
LIME
32.82
32.57
0.77%
34.85
33.70
3.31%
SILO
19.82
19.51
1.60%
30.60
30.07
1.79%
MAPLE
8.189
7.664
6.86%
31.32
31.25
0.25%
As can be seen in Table 3 LIMI''s superior performance is still apparent in neighborhood MAE. For instance, LIMIS achieves 25.56 in neighborhood metric (with MAE) which is better than the results with MAPLE (42.21) and LIME (39.51) using Facebook data and LightGBM model (over independent runs). Note that the differences between pointwise and neighborhood fidelity metrics with LIMIS are negligible across other datasets and black-box models. This shows that the performance of LIMIS is locally generalizable and reliable, which is the main objective of locally interpretable modeling.
5.3 Computational time
We quantify the computational time the largest experimented dataset, Facebook Comments, that consists 600,000 samples. On single NVIDIA V100 GPU (without any hardware optimizations), LIMIS yields a training time of less than 5 hours (including Stage 1, 2 and 3) and an interpretable inference time of less than 10 seconds per testing instance. On the other hand, LIME results much longer interpretable inference time, around 30 seconds per a testing instance, due to acquiring large number of black- box model predictions for the input perturbations, while SILO and MAPLE show similar computational time with LIMIS.
6 LIMIS Explainability Use Cases
In this section, we showcase explainability use-cases of LIMIS for iuman-in-the-loop AI deployments. LIMIS distill complex black-box models into explainable surrogate models, such shallow DTs or linear
11

---

Published in Transactions on Machine Learning Research (09/2022)
regression. These surrogate models are explainable and can bring significant value to many applications where exact and concise input-ouiput mapping would be needed. As the fidelity of LIMIS is very high, the users would have high trust in the surrogate models. Still, the outputs of the LIMIS should not be treated as the ground-truth interpretations (local dynamics).
6.1 Sample-wise feature importance
Discovering the importance of features for a prediction is one of the most commonly-usee explanation tasks, Locally interpretable modcling can enable this capability by using a lincar model as the surrogate, since the coelficients of linear models can directly tell how features combined for a prediction.
To highlight this capability, consider LIMIS with RR UCI Adult Income, shown in Fig. Here, we use XGBoost as the black-box model along with the locally interpretable RR. We focus visualizing the feature importance (the absolute weights of the fitted locally interpretable RR model) for various important subsots, to analyze common decision making drivers for them. Specifically, we focus on 5 subgroups, divided based on (a) Age, (b) Gender, (c) Marital status, (d) Race, (e) Education.
Gender
Marital status
Racs
Eduration
Age
Iniggt
ElucationNum
CapitalGain
Capiolloss
JoursPerWieek
WorkClass
Education
WartalStatus
Decupation
Raco
38-c-Age-BS 65
Femala
Malle
Me meereesd-a sproere
Merreed spousse
Martied spoouco abooent
haever rever-mmarreed
Supar
Widowod
Nerwe Patan Pacc Paclsedd tolledoe
Black
10th
12th
sth eth
ath
Roso acm
Aasoo voc
Bechrolons
Decterste
HS-gu
Preschool
schoo)
Some callage
to
Figure 5: Feature importance (denoted with the colors) discovered by LIMTS the absolute weights of the fitted locally interpretable RR model, UCI Adult Income, for 5 types of subgroups: (a) Age, (b) Gender, (c) Marital status, (d) Race, (c) Education.
We observe that for age-basee based subgroups, more importance is attributed to capital gain for older people (age > 25) compared to young people (age < 25), as intuitively expected given that older people often have more savings [Wolff 2015). For education-hased subgroups, capital attributed be more important for highly-oducated people (Doctorate, Prof-school, and Masters graduates), also intuitively expected the income gaps lend to get wider with education levels (Nordin al. 2010 Sullivan & Wolla 2017). LIMIS can also be used for dependency assessments for black-box models by analyzing the importance of sensitive attribute features locally to see whether any would play a significant role in decision making. For this example, some attributes (such as gender, marital status and race) are not observed to have high importance for most subgroups.
occupation, and native countries are
These exemplify how local explanations from LIMIS can bring value users, particularly with a lincar interpretable model where the learned weights can readily provide insights on how features affect the predictions. We provide more results in the Appendix. G.7
12

---

Published in Transactions on Machine Learning Research (09/2022)
6.2 Suggesting actionable input modifications to alter decisions
In many applications, it is desired to understand what it would take to alter the decision of model. For example, after rejection a loan application, the applications would want to understand what they can change to get it accepted, or after a disease diagnosis, doctors and patients would want to understand the suggestions on what. can be changed about the patients health to reverse the diagnosis prediction.
The fundamental benefit of locally interpretable modeling is that it allows understanding the how features are affecting the prediction for an instance, via interpretable model that yields the precise relationship. High fidelity of LIMIS with simple interpretable models enable this capability effectively.
No
Key characteristics
Prediction
Suggestion for >$50K income
1
Education: High-school, No capital gain
<
Get Masters & increase capital gains 6K
2
No capital gain, Hours per week: 40
$50K
3
Age: 33, Education year: 13, Married
$50K
Age: 14, Job: Craft-repair
<S50K
Increase capital gain by 6K
5
Job: Local-gov, Education: HS, Hours week: 40
<55K
Change job to Federal-gov
6
Capital loss: 23K, Job: Sales, Education: College
<55K
Decrease the capital loss to 9K
7
Hours per week: 26, Job: Sales
$50K
Change job to Tech support
8
Capital gain: 15K, Masters, Age:
$50K
Increase the capital gain to 10K
9 Capital loss: 17K, Hours per week: 40
$50K
Reduce the capital loss to 11K
10
Age: 38, Occupation: Exec managerial
<55K
Table 4: For ten individuals explanations given by LIMIS using shallow DT on UCI Adult dataset are shown. The individual characteristics are based on the DT and the suggestions are obtained with the goal of making the locally interpretable model prediction as >$5KK, by inspecting the fitted DT.
consider LIMIS with shallow DTs (with depth of 3) interpretable models, the UCI Adult Income dataset. LIMIS is first trained on the entire training data, and then, for some test instance, LIMIS is used to extract local explanations on the predictions of black-box model, XGBoost. In addition, we consider the question of 'what's 's the suggested minimum change to alter the decision?' and utilize the fitted shallow DT for this purpose. Essentially, our approach to find the modification suggestion relies on traversing the DT upwards from the leaf, and finding the nearest. (in terms of being closest in edge distance) node condition that would yield the opposite prediction. We specifically focus suggestions to increase the income prediction from low (<550K) to high income (>$50K)
For this demonstration, shown in Table.
In most cases, we observe that the suggestions are consistent with the expectations (Nordin et al. 2010 Sullivan & Wolla 2017 Wolff 2015). For example, better investment outcomes, higher paying jobs and additional education are among common suggestions. When the inputs are modified with these changes, the black-box model predictions change from <$50K to >$5K,, for all cases exemplified here, underlining the accuracy of the suggestions.
7 Conclusions
We propose novel method for locally interpretable modeling of pre-trained black-box models, called LIMIS. LIMIS sclects a small number of valuable instances and uses them to train low-capacity locally interpretable model. The selection mechanism is guided with a reward obtained from the similarity of predictions of the locally interpretable model and the black-box model, defined as fidelity. LIMIS near-matches the performance of black-box models, and significantly outperforms alternatives, consistently across various datascts and for various black-box models. We demonstrate the high-fidelity explanations provided by LIMIS can be highly useful to gain insights about the task and to understand what would modify the model's outcome.
13

---

Published in Transactions on Machine Learning Research (09/2022)
8 Broader Impact
Interprctability is critical, to increase the reach of AT to many more use cases compared to its reach today, in a reliable way, by showing rationale behind decisions, eliminating biases, improving fairness, enabling detection of systematic failure cases, and providing actionable feedback to improve models (Rudin) 2018). We introduce a novel method, LIMTS, that tackles interpretability via instance-wise weighted training to provide local explanations. LIMIS provides highly faithful, and easy-to-digest explanations to humans. Applications of L.MTS can span understanding instance-wise local dynamics, building trust. by explaining the constituent. components behind the decisions and enabling actionable insights such manipulating outcomes. For sccnarios such as medical treatment planning, where the imput variables can be controlled based on the leedback from the output responses, interpretable local dynamics can be highly valuable for manipulating outcomes (Saria et al. 2018).
As one limitation, the proposed instance-wise weight estimator is indoed a black-box model and difficult interpret without post-hoc interpretable methods. However, the main value proposition of LIMIS that the final output from the locally interpretable model, is fully interpretable and the users utilize the final output for understanding the rationale behind the local decision making process of the black-box model. With our experiments, we show that the fidelity metrics of the locally interpretable models of LIMIS are high, in other words, for each sample, they approximate the black-box model functionality very well.
Broadly, there are many different forms of explainable AI approaches, from single interpretable models to post-hoc methods for complex black-ban models. LIMIS is not an alternative all of them, but it specifically provides the locally interpretable modeling capability, which has numerous impactful use cases in real-world AI deployments, including Finance, Hcalthcare, Policy, Law, Recruiting, Physical Sciences, and Retail.
We demonstrate that LIMTS provides the local interpretability capabilities similar to other notable methods like LIME, while achieving much higher fidelity, as strong quantitative evidence for its utility, Still, the outputs of the LIMIS framework should not be treated as the ground truth interpretation of the black-box models. Large-scale human subject evaluations can further add confidence in the capabilities of LIMTS and the quality of its explanations leave this important aspect future work.
References
David Alvarez-Melis and Tommi Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.0804 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML. ACM, 2009.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013.
Adam Bloniarz, Amect Talwalkar, Bin Yu, and Christopher Wu. Supervised neighborhoods for distributed nonparametric regression. In AISTATS, 2016.
Lco Breiman. Random forests. Machine Learning, 45(1):5-32, 2001.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD. ACM, 2016.
Radwa ElShawi, Youssef Sherif, Mouaz Al-Mallah, and Sherif Sakr. Interpretability in healthcare: A comparative study of local machine learning interpretability techniques. Computational Intelligence, 37(4): 1633-1650, 2021.
L. Н. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An overview of interpretability of machine learning. In IEEE International Conference on Dala Science and Advanced Analytics, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning. MIT Press, 2016. deeplearmingbook.orgg
http:// wwW
14

---

Published in Transactions on Machine Learning Research (09/2022)
Rory Mc Grath, Luca Costabello, Chan Le Van, Paul Sweeney, Farbod Kamiab, Zhao Shen, and Freddy Lecue. Interpretable credit application predictions with counterfactual explanations. ar Xiu preprint arXiv:1811.05245, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1508.02533 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-soflmax. In ICLR, 2016.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, 2018.
Ulf Johansson, Cecilia Sönströd, Ulf Norinder, and Henrik Boström. Trade-off between accuracy and interprctability for predictive silico modeling. Future medicinal chemistry, 367)6647-633, 2011.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In NIPS, 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. Interpretable explorable approxima- tions of black box models. arXiv preprint ar arXf::107.00155. 2017.
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovee. Faithful and customizable explanations black box models. In AAAI/ACM Conference on AI, Ethics, and Society. ACM, 2019.
Zachary C Lipton. The mythos of model interpretability. arXiv preprint arXiv:1666.03:: 03490. 2016.
Scott M Lundberg and Su-ln Lee. A unified approach to interpreting model predictions. In NIPS, 2017.
Sujith Mangalathu, Seong-Hoon Hwang, and Jong-Su Jeon. Failure mode and effects analysis of rc members based on machine-learring--bsse based shapley additive explanations (shap) approach. Engineering Structures, 109:1, 2020.
J Eamonn Nash and Jonh V Sutcliffe. River flow forecasting through conceptual models part iâAfa discussion of principles. Journal of hydrology, 10(3):282-25) 290, 1970.
Martin Nordin, Inga Persson, and Dan-Olof Rooth. Aooocuppona mismatch: Is there an income penalty? Economies of Education Review, 29(0):1047-1059, 2010. ISSN 0272-7757.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. Model agnostic supervised local explanations, In NIPS, 2018.
Gregory Plumb, Maruan Al-Shodivat, Eric Xing, and Amoct Talwalkar. Regularizing black box models for improved interpretability. arXiv preprint arXio:1902. 067007777 2019.
Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. arXiv preprint 151 2015.
Mengye Ren, Wenyuan Zeng. Bin Yang, and Raquel Urtasun. Learning reweight examples for robust deep learning. In ICML, 2018
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In KDD. ACM, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision mnnel-agnostic explanations. In AAAT, 2018.
15

---

Published in Transactions on Machine Learning Research (09/2022)
Cynthia Rudin. Please Stop Explaining Black Box Models for High Stakes Decisions. arXtv:1811.10154 2018.
Suchi Saria, Atul Butte, and Aziz Sheikh. Better medicine through machine learning: What's real, and what's artificial? PLoS medicine, 15112))e1002221, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishma Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, 2017.
Adam Stooke and Pieter Abbeel. Accelerated methods for deep reinforcement learning. or preprint WrXiv:1803.02811 2018.
Erik Strumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41(3):647 665, 2014.
Jessica Sullivan and Scott Wolla. Education, income, and wealth. Page One Economics Newsletter, pp. 1, 2017. URL https://EconPappers rewec.
Pang-Ning Tan. Introduction Data Mining. Pearson Education India, 2018
Miklós Virág and Tamás Nyitrai. Is there a trade-off between the predictive power and the interpretability of bankruptey models? the case of the first hungarian bankruptcy prediction model. Acta Oeconomica, 64(4): 419-440, 2014.
Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A. Efros. Dataset distillation, 2019. URL http:///opeaerereeeeeeeccoooooo/
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229 256, 1992.
Edward N. Wolff 177Wealth Accumulation by Age Cohort: The Role of Savings, Capital Gains, and Intergenerational Transfers. In Inheriting Wealth America: Future Boom or Bust? Oxford University doi: 10. (0..109//aaa ////////rE0000///7/1313333555333 003. 0005
Press, 02 2015. ISBN 9780199353958.
URL https;
//doi.org/ 10///10..1 1093/ www.:0009///aprrofoooo :ttp://ffo..//000003333333333330
Chih-Kuan Yeh, Joon Kim, Tan En-Hsu Yen, and Pradeep Ravikumar. Representer point selection for explaining deep neural networks. In Advances Neural Information Processing Systems, pp. 9291-9301, 2018.
Wojciech Zaremba and Πlya Sutskever. Reinforcement learning neural turing machines-revised. arXie preprint arXiv:1505.0055 2015.
Xingxing Zhang and Mirella Lapata. Sentence simplification with deep reinforcement learning, In Empirical Methods in Natural Language Processing, 2017.
Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. Why should you trust my explanation? understanding uncertainty in lime explanations. ar preprint arXio: 12991, 2019.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
16

---

Published in Transactions on Machine Learning Research (09/2022)
A Hyper-parameters of the predictive models
In this paper, we use different predictive models. For each predictive model, the corresponding hyper- parameters used in the experiments as follows:
XGBoost (Chen & Guestring 2016): booster gbtree, max depth 6, learning rate 0.3, number of estimators 1000, max depth- 6, reg alpha-
LightGBM Ke et al. 2017): booster gbdt, max depth None, learning rate 0.1, number of estimators 1000, min data in leaf 20
Random Forcsts (RF) None, warm start False
Breiman
2001 number of estimators 1000, criterion gini, max depth
Multi-layer Perceptron (MLP); Number of layers 4, hidden units ffeature dimensions, feature
dimensions/2, leature dimensions/4 feature dimensions/8], activation function - ReLU, early stopping
True with patience 10, batch size 256, maximum number of epochs 200, optimizer Adam
Ridge Regression: alpha-1
Regression DT: max depth 3, criterion gini
Logistic Regression: solver lbfgs, regularization
Classification DT: max depth 3, criterion gini
We follow the default settings for the other hyper-parameters that not mentioned here.
B Performance metrics
Mean Absolute Error (MAE):
MAE = E (xt " (10001)))))))))))))))))))) ->'ll
110002; (xx) sill,,
Local MAE (LMAE):
LMAE = -E Px (1x)x))(x²)) -f*(x")ll
( (x4)))
NSE
Nash & Sutcliffe 1970
E 115* S0(e (xbi1 Ex [f*((x)) xp If (*' 1 || - 11
Ex (If (x') - )0())) (x') )1
NSE =1
If INEE- = 1, the predictions of the locally interpretable model perfectly match the predictions the black-box model. On the other hand, if NSE 0, the locally interpretable model performs similar as the constant mean value estimator. NSE the locally interpretable model performs worse than the constant mean value istimor
17

---

Published in Transactions on Machine Learning Research (09/2022)
C
Implementations of benchmark models
In this paper, we use different benchmark models. Implementations of those models can be found in the below links.
LIME: httpss//gthuuu.ooo/arror///////1 Ribeiro al.
2016
SILO:
http:://gthhub.cc///////m//////
Bloniarz et al.
2016
MAPLE: http:///ggaaab..co//////m////// Plumb et al.
2018
D
Data statistics
Table 5: Data Statistics of 5 real-world datasets. Tabel distributions: Number of positive labels (positive label ratio) for classification problem, and label mean (5%-50%-95%% percentiles) for regression problem.
Problem
Data name
Number of samples
Dimensions
Label distribution
Blog Feedback
60,021
280
6.6 (0-0-22)
Regression
Facebook Comment
603,713
54
7.2 (0-0-30)
News Popularity
39,644
59
3395.4 (684-1440-10800)
Adult Income
48,842
108
11,687 (23.9%)
Classification
Weather
112,925
25,019 (22.2%)
E
Learning curves of LIMIS
Syn1 datasat
SynZ datasot
Syn3 dataset
07
06.
Rewards
UMIS Baseline
Hewardss
LIMIS Baseline
Rewards
UMMS Baseline
250
300
150
10.0
1253
1300
1756
2000
254
300
750
2008 Iterations
1250
2500
3750
200
221
300
755
2008 Berations
1220
1500
3730
2000
Berations
Figure 6: Learning curves of LIMIS three synthetic datasets. X-axis: instance-wise weight, estimator training, Y-axis: Rewards (LMAE of baseline (globally interpretable model). LMAE of LIMIS), higher is better.
The number of iterations
F Instance-wise weight distributions for synthetic datasets
Fig. (c)-(c) show that the instance-wise weights have quite skewed distribution. Some samples (e.g. with average instance-wise weights above 0.5) much more critical to interpreting the probe sample than many others (e.g. average instance-wise weights below 0.1).
Furthermore, we analyze the instance-wise weights of training samples, and Fig. g shows that the training samples near the probe sample get higher weights L.IIS learns the meaningful distance metrics to measure the relevance while interpreting the probe samples.
18

---

Published in Transactions on Machine Learning Research (09/2022)
Instance wise weight distribution analysio Sym1
we gin distribution analysis -5y0
weight distribution analysis Syn3
200
The number of ttaikin caneees
120
225
200
8
300
10
0.2
04
0.1
0.5
0.4
00
02
0.3
0.5
0.7
Aediage weights
Ascrage weihts
Aucrage We gifs
Figure 7: Instance-wise weight distributions for (a) Synl, (b) Syn2, and (c) Syn3 datasets.
Instance-wise weight distribution analysis by distance
0.5
Syn1
Syn2
0.4
Syn3
Averago weights
0.2
0.1
0.0
02
0.4
0.6
0.8
1.0
Distance from data (percentile)
Figure 8: Average instance-wise weights vs. distance from the probe sample.
G Additional results
G.1 Sample complexity analyses with differentiable baselines
Sample Complesity Analyis Syn1
Sample Complexity Analysis Syn2
Sample Complexity Analysis Syn'
LIMIS STE
0.55
Random
LIMIS
077
Random
850
0.5
02
835
Random
500
750
3000
1250
1500
1750
2000
250
500
750
1000
3250
1500
1750
2000
250
500
750
1200
1250
1500
1750
2000
traning somples
trainng semples
stawing samples
Figure 9: AWD performances in terms of the number of training samples used to train three models: LIMIS, STE and Random.
G.2 Which training samples selected by LIMIS, MAPLE and LIME?
LIMIS, MAPLE and LIME select subset of training samples to construct lncelll-itterpretable models. The training samples selected by LIME are the ones closest to the point to explain. MAPLE utilizes random forest model (trained to predict black-box model outputs) to select the subset of training samples. In this subsection, we quantitatively analyze which samples are chosen by LIMIS, MAPLE and LIME.
Due to the lack of ground truth for idcal training sample selection real-world datasets, we use synthetic datasets to demonstrate this experiment. Note that for each synthetic data, ideal training sample selections are explicitly determined by X and Xu (see the definitions of Synl to Syn3). Therefore, quantitatively
19

---

Published in Transactions on Machine Learning Research (09/2022)
evaluate the performances in terms of AUC comparing between selected training samples and ideal training sample selection.
Table 6: Evaluation correctly selected training samples by LIMIS, LIME, and MAPLE in terms of AUC. Bold represents the best.
Models Datasets
Synl
Syn2
Syn3
LIMIS
0.7837
0.6892
0.6935
LIME 0.5253
0.5017
0.5202
MAPLE 0.6723
0.5844
0.5452
As can be seen in Table 6 the average performance of correctly chosen samples on Synl to 3 are 0.7218, 0.5157, 0.6006 using LIMIS, LIME, and MAPLE, indicating the superiority LIMIS.
G.3 Additional ablation study Optimization
To better motivate our method, we perform ablation studies, demonstrating that the proposed complex objective can be efficiently addressed with policy-gradient based RL where the gradient has a closed-form expression. The inner optimization used for fitting the surrogate explainable model. We explain that for simple surrogate models such as ridge regression, the fitting has a closed form expression and the overall computational complexity negligible indeed, yielding similar training time compared to the alternative methods. Note that policy-gradient is only utilized for the outer-optimization.
Table Average Weight Difference (AWD) comparisons on three synthetic datasets with different number of train samples (N). Training time is computed single K80 GPU until the model convergence (i.c., more validation fidelity improvements).
Training samples
N 1000
N 2000
Optimization
Average performance
Training time
Average performance
Training time
Bi-level
LIMIS
0.3982
49 mins
0.2936
92 mins
Gumnbel-softmax
0.4209
38 mins
0.3190
71 mins
Singlo-level
STE
0.4156
39 mins
0.3208
73 mins
LIME
1.6372
17 mins
1.5633
mins
Two-stage single-level
SILO
0.6983
30 mins
0,6561
44 mins
MAPLE
0.6217
mins
0.5890
104 mins
Table 12 compares our proposed method LIMIS to other methods (Gumbel-softmax and STE) which utilize single- level optimization (i.e. direct back-propagation) LIMIS with bi-level optimization achieves better performance (lower AWD) with small increase in computational complexity. In addition, compared to other baselines (LIME, SILO, and MAPLE) which utilize two-stage optimization (where each stage is single-level), the proposed bi-level optimization in LIMIS shows significantly better performance with similar complexity.
20

---

Published in Transactions on Machine Learning Research (09/2022)
G.4 Regression with shallow regression DT as the locally interpretable model
Table 8: Overall prediction performance (metric: MAE, lower better) and fidelity (metric: NSE, higher is better) on real-world regression datasets, using shallow Regression DT the locally interpretable model while explaining the black box models: XGBoost, LightGBM, MLP and RF. "Original" represents the performance of the original black-box model, that the locally-interpretable modeling is applied on. We also show the performance of shallow regression RDT as globally-interpretable model (reported the performance (in terms of MAE) under the data name). Red represents performance that worse than globally-interpretable shallow regression DT and the negative NSE. Bold represents the best results.
Datasets
Models
XGBoost
LightGBM
MLP
RF
(RDT)
Metries
MAE
NSE
MAE
NSE
MAE
NSE
MAE
NSE
Original
5.131
1.0
4.965
1.0
4.939
1.0
5.203
1.0
LIMIS
5.121
.8242
4.778
.8939
4.587
.6375
4.652
.8990
Blog (5.955)
LIME
11.80
.6658
13.22
.1483
7.396
-.6201
19.61
-.4116
SILO
5.149
.8035
4.818
.8816
4.649
.6177
4.715
.8744
MAPLE
5.329
.7991
5.024
.8660
4.609
.6339
5.016
.8201
Original
24.18
1.0
20.22
1.0
18.36
1.0
30.09
1.0
LIMIS
21.82
.9307
21.35
.9194
18.56
.8832
22.44
.7236
Facebook (22.28)
LIME
36.69
.3278
44.21
.1809
40.85
-.1513
51.70
.2001
SILO
22.42
.8555
22.33
.7235
19.57
.8566
24.41
.6917
MAPLE
22.15
.8244
23.43
.8581
20.32
.8035
27.12
.3144
Original
2995
1.0
3140
1.0
2255
1.0
3378
1.0
LIMIS
2938
.9382
2504
.4104
2226
.9016
2431
.2768
News (3093)
LIME
6272
-,6267
7737
-2.960
2390
.0013
9637
-7.075
SILO
2910
.1020
2854
.3461
2274
.8201
2874
.2278
MAPLE
2968
.9288
2846
.3631
2284
.8021
2888
.1872
Table Fidclity results (metric: LMAE, lower better) on regression problems with shallow regression DT as the locally interpretable model. Bold represents the best results.
Datasets
Models
XGBoost
LightGBM
MLP
RF
LIMIS
.7530
1.358
1.273
1.413
LIME
9.160
11.16
5.006
17.461
Blog
SILO
.8325
1.379
1.178
1.934
MAPLE
1.029
1.598
1.359
2.158
LIMIS
7.240
6.867
5.596
15.77
LIME
31.52
37.75
30.58
45.58
Facebook
SILO
8.459
9.149
6.997
18.63
MAPLE
7.985
8.644
7.290
23.17 17
LIMIS
389.0
1072
116.6
957.1 11
LIME
4455
6243
504.0
9969
News
SILO
496.7
1214
160.6
1175
MAPLE
440.7
1201
163.6
1196
21

---

Published in Transactions on Machine Learning Research (09/2022)
G.5 Regression with RR as the locally interpretable model Fidelity analysis in Local MAE
Table 10: Fidelity results (metric: LMAE, lower is better) on regression problems with ridge regression as the locally interpretable model. Bold represents the best results.
Datasets
Models
XGBoost
LBg
MLP
RF
LIMIS
.8679
1.135
1.432
1.651
LIME
6.534
8.037
8.207
17.01
Blog
SILO
2.220
3.046
2.393
3.909
MAPLE
.9690
1.416
1.550
1.984
LIMIS
6.394
21.29
8.217
33.64
LIME
32.57
33.70
27.38
48.03
Facebook
SILO
19.51
30.07
11.52
40.14
MAPLE
7.664
31.25
13.31
44.38
LIMIS
436.9
1049
74.11
905.8
LIME
3317
4766
327.4
8828
News
SILO
657.2
1253
79.85
1345
MAPLE
500.5
1261
88.19
1157
22

---

Published in Transactions on Machine Learning Research (09/2022)
G.6 Classification with RR the locally interpretable model
Table Overall prediction performance (metric: APR, higher better) and fidelity (metric: NSE, higher is better) on real-world classification datasets, using RR as the locally interpretable model while explaining the black box models: XGBoost, LightGBM, MLP and RF. 'Original' represents the performance of the original blxck-box model, that the locally-interpretablee modeling is applied We also show the performance of Logistic Regression (LR) as giobally-interpretaal model (reported the performance (in terms of APR) under the data name). Red represents performance that is worse than globally-interprelabll model logistic regression and the negative NSE. Bold represents the best results.
Datasets
Models
XGBoost
LightGBM
MLP
RF
Metrics
APR
NSE
APR
NSE
APR
NSE
APR
NSE
Original
.8966
1.0
8254
1.0
.7778
1.0
.7621
1.0
LIMIS
.7977
.9871
.8039
.9439
.7670
.9791
.7977
.9217
Adult
LIME
.6803
.7195
6805
.6259
.9577
.8310
.7057
.6759
(.7553)
SILO
.7912
.9500
.7884
.9301
.7655
.9778
.7664
.9140
MAPLE
.7947
.9400
.8011
.9386
.7683
.9636
.7958
.8961
Original
.7133
1.0
.7299
1.0
.7205
1.0
.7274
1.0
LIMIS
.7140
.9799
.7290
.9801
.7212
.9755
.7311
.9450
Weather (.7009)
LIME
.6376
.7898
.6392
.6873
.395
.5321
.6387
.4513
SILO
.7134
.9888
.7281
.9773
.7220
.9797
.7277
.9024
MAPLE
.7134
.9897
.7273
.9778
.7213
.9022
.7308
.9223
G.7 Qualitative analysis: LIMIS interpretation
Rein
Rair rall
Wine Speed 3pm
Humrdity 3om
Phessure Jpm
Temperature 3pm
MinTemp
MoxTemp
Reinfell
WindCustSpeed
Windspredeem
WnddppadBBmm
Femi
Humistyy 9pm
Pressun9emm
fressure3pm
3pm
Re sooday
WindGusiDir
Wind be 3pm
No Boin Toalyy Rain Tosky
Ra Fall ic 1
sa aal 5
Rain Fall
10
20
30
< Hummdiy 3pm 20"
20 <- Hummoitt Som 50
50 <- Humidry 3pm
pressur 5pm 580
1910
5000 co Pros Prosure 3pm
10
10 rencersturs 20
3pm
Wind Spase 3pr <- 10 W vo Spoed 3pm
Tampe dtrs 3pr sc-
20 <n Wi Syoed Sym
30 Wind Speed
Humidtn Spm
20 e- Tempee Som
Figure 10: Discovered feature importance (denoted with the colors) by LIMIS on Weather dataset. for 6
types of subgroups: (1) Rain, (2) Rain fall, (3) Wind speed 3pm, (4) Humidity 3pm, (5) Pressure 3pm, (6) Temperature pm.
23

---

Published in Transactions on Machine Learning Research (09/2022)
(Rain fallez 11 (Mird quedee: ('erperstiee
yiole fall 1-)4-
(tvind speei 2544 (lompeoture
Narrtall
pind
WhoS poec9am
WineS Spe
HumisitySam
Hum dity 2pm
Pressureaam
PressureJpm
Tompdam
Tempslm
ReinTrsday
WindDrraam
Windtrr3pm
810
810
Figure 11: Discovered feature importance Weather data for: rain fall S 1, wind speed (at 3pm) S and temperature (at 3pm) >> 30 (left), and 'rain fall >> 15, wind speed (at 3pm) >> 25, and temperature (3pm) < 10 (right).
this section, we qualitatively analyze the explanations provided by LIMIS. Although LIMIS can provide local explanations for each instance separately, we consider the explanations in subgroup granularity for better visualization and understanding. On Weather dataset, Fig. I0 shows the feature importance (discovered by LIMIS) for six subgroups in predicting whether it will rain tomorrow, using XGBoost as the black-box model. We use RR the locally interpretable model and the absolute value of fitted coefficients are used as the estimated feature importance. For rain fall subgroups, humidity and wind gust speed seem more important for heavy rain (rain fall 2 5) than light rain (rain fall < 5). For temperature subgroups, rainfall, wind gust speed and humidity are more important for cold days (temperature (at 3pm) < 10) than warm day (temperature (at 3pm) 20). In general, for heavy rain, fast wind speed, low pressure, and low temperature subgroups, humidity, wind gust speed and rain fall variables are more important for prediction. Fig. [ shows the feature importance (discovered by LIMIS) for two subgroups. We observe the clear difference of the impact of afternoon humidity and wind gust speed, This underlines how LIMIS can shed light on the samples with distinct characteristics. Additional use cases for human-in-the-llop Al capabilities can be found in the Sect.
instances that clearly reflect different climate characteristics.
24

---

Published in Transactions on Machine Learning Research (09/2022)
H Additional Analyses
H.1 Synthetic data experiments with LIME
In Fig. 3 we exclude the performance of LIME bascline due to highlighting the performance improvements
from best alternative methods. Fig. 12
we include the results with LIME baselines for completeness of
the synthetic data experiments.
Syn1 Dataset
Syn2 Dataset
Syn3 Dataset
1.0
LIMI
SILO
SILO
SILD
MAPLE
MAPLE
MAPLE
LIME
06
LIME
LIME
626
04
02
Distance from the boundary (percentile)
Distance from the boundary (percentile)
Distance from the boundary (percentile)
Figure 12: Mean AWD (aggregated per uniformly divided x-axis bin) with 95% confidence intervals (of 10 independent runs) on three synthetic datasets (y-axis) vs. the percentile distance from the boundary where the local function behavior change (x-axis) with LIME as an additional baseline.
H.2 Scatter plots for synthetic data experiments
In Fig. [] we report the average AWD performances after aggregating the AWD values per uniformly divided x-axis bins. this subsection, we report the scatter plots between distance from the boundary (x-axis) and LIMIS's AWD for each sample (y-axis) across 3 different synthctic datasets.
Syn1 Dataset
Syn2 Dataset
Syn3 Dataset
LIMIS
10
UMMI
0.8
UMIS
66
07
06
AND
0.6
0.
0.4
0.3
02
02
01
Distance from the boundary
Distance from the boundary
Distance from the boundary
Figure 13: Scatter plots of LIMIS's AWD across distance from the boundary in terms of percentiles.
H.3 Additional ablation study: Sampling at inference
As explained Sect. 3 we only the sampling training time to encourage exploration. At inference time, use the weighted optimization using the outputs of the instance-wise weight estimator. In this subsection, we analyze the impact of weighted optimization in comparison to the sampling at inference procedure.
We use XGBoost as the black-box model and RR as the locally interpretable models. As can be seen in Table. 12 sampling at inference consistently shows worse performances than weighted optimization (that is proposed in the LIMIS framework), but overall the differences are small.
H.4 Local generalization of explanations: Additional datasets
In Table. 3 we provide the results of local generalization of explanations via Noighbor LMAE with Blog and Facebook datasets. In Table, 13] we also provide the Neighbor LMAE results for other real-world datasets: News, Adult, and Weather.
25

---

Published in Transactions on Machine Learning Research (09/2022)
Table 12: Additional ablation studies on real-world datasets with weighted optimization vs. sampling at inference. Metrics are LMAE for both regression and classification datasets.
Regression datasets
Classification datasets
Datasets (Metrics: LMAE)
Blog
Facebook
News
Adult
Weather
Weighted optimization at inference
.8679
6.394
436.9
.1997
.1129
Sampling at inference
.8821
6.671
455.2
.1465
.1157
Difference (%)
1.6%
4.3%
4.2%
4.9%
2.5%
Table 13: Prediction performance (metric: weighborhood LMAE and pointwise LMAE, lower is better) on aegression/classiiin /classification datasets using RR as the locally interpretable model while explaining the black box models: XGBoost
regression/
Chen & Guestrin 2016) and LightGBM
(Ke et al. 2017
Datasets
Models
XGBoost
LightGBM
(RR)
Metrics
Neighbor LMAE
Pointwise LMAE
Diff
Neighbor LMAE
Pointwise LMAE
Diff
LIMIS
442.9
436.9
1.38%
1064
1049
1.47%
News
LIME
3550
3317
7.05%
5161
4766
8.30%
SILO
667.5
657.2
1.57%
1258
1253
0.4%
MAPLE
501.5
500.5
0.19%
1310
3.9%
LIMIS
.4022
.1397
0.39%
.1297
.1288
0.76%
Adult
LIME
3162
.3177
1.46%
.3522
.2975
5.96%
SILO
.6644
.1622
2.59%
.1474
.1432
2.95%
MAPLE
.717
.1599
7.42%
.1428
.1407
1.50%
LIMIS
.2166
.1129
7.7%
.1357
.1291
5.14%
Weather
LIME
3093
.7500
12.5%
.3037
.2933
3.55%
SILO
.1299
.1257
0.21%
.1410
.1388
1.61%
MAPLE
.1244
.1232
5.10%
.1437
.1371
4.39%
H.5 Training / Inference time for real-world datasets
In this subsection, we demonstrate the training and inference times of LIMIS on real-world datasets. For training time, we exclude Stage 1, black-box model training, to solely focus on LIMIS-specific instance-wise weight estimator training time. We use a single NVIDIA V100 GPU to train and infer the LIMIS framework.
Table 14: Runtime analyses on real-world datasets. Inference time is computed per one testing sample. Training time is computed until the model convergence (i.e., no more validation fidelity improvements)
Regression datasets
Classification datasets
Datasets
Blog
Facebook
News
Adult
Weather
Number samples
60,021
603,713
39,644
48,842
112,925
Dimensions
380
54
59
108
61
Training time
56 mins
3 hours 27 mins
49 mins
21 mins
1 hour 17 mins
Inference time
1.7 secs
1.2 secs
1.1 secs
0.8 secs
0.7 secs
H.6 Convergence plots for real-world datasets
Fig. 0 shows the convergence plots for 3 synthetic datasots. In this subscction, we additionally show the convergence plots for 3 real-world datasets (Adult, Weather, and Blog Feedback). these experiments, we XGBoost as the black-box model and RR the locally interpretable model.
26

---

Published in Transactions on Machine Learning Research (09/2022)
Fig. 14 shows that the convergence of LIMIS is stable for the real-world datasets. The convergence is often observed to be around 1000 iterations. Note that for LIMIS training, we use large batch sizes to reduce the noise in the gradients which would be critical for fast and stable convergence (Stooke & Abbeel 2018).
Blog Feedback dataset
Weather dataset
Rewards Fidelity Improve
0.00
Rewards -Fideiity Improve
Marth
0.03
0.07
0.08
0.04
0.05
0.03
0.0
0.2
0.01
250
300
750
1200
2250
1300
1750
2002
200
500
750
3003
1250
15000
1750
2000
259
300
1002
1250
1500
1750
2008
Iterations
Rerations
Iterations
Figure 14: Learning curves of LIMIS three real-world datasots. X-axis: The number of iterations instance-wise weight estimator training, Y-axis: Rewards (Fidelity improvements). higher is better.
We also include the convergence plots for STE and Gumbel-softmax variants in Fig. The convergence of the alternatives seems a bit faster but overall the convergence trends seem mostly similar across LIMIS, STE, and Sofmel-softmax.
Blog Feedback dataset
Adult dataset
Weather dataset
Rewards Fidelity Improve
Rewards Fidelity Improve
0.07
Rewards Fidelity Improve
0.00
0.00
0.3
6.03
STE
0.02
STE
STE
Gumbel-soffe softmax
Gumbel-softman
Gumbel-softmax
0.1
1000
1590
2003
US5
1750
250
300
1002
1250
1500
1750
2008
Iterations
Etrraiinn
lterations
Figure 15: Learning curves of STE and Gumbel-softmax on three real-world datascts. X-axis: The number of iterations on instance-wet wise weight estimator training, Y-axis: Rewards (Fidelity improvements).
27

---

Published in Transactions on Machine Learning Research (09/2022)
H.7 Synthetic data with non-linear feature-label relationships
Three synthetic datasets used in the main manuscript have piece-wise linear leature-label relationship. In this subsection, we generalize the results to another synthetic data, with non-linear feature-label relationships. More specifically, we construct Syn4 datasct as follows:
Y [(x) = sin(Xi) + 2cos( - - -
X are sampled from Uniform(-2 -1, 1) distribution (with 11 dimensions); we set the ground truth local dynamics as the first order coefficients of the Taylor expansion:
(x) = [cos(<r)) -2sin(x2), -33 exp(( 0,...
Then, we utilize LIMIS, LIME, SILO, and MAPLE to recover the ground truth local dynamics with ridge regression as the locally interpretable model. We use the AWD (defined for the first-order coefficients of Taylor expansions as the ground truth explanation) to evaluate the performances the interpretations. As can be seen Table 15 the performance of LIMIS is significantly better than other alternatives.
Datasets / Methods
LIMIS
LIME
SILO
MAPLE
Syn4
0.2508
0.5549
0.3411
0.3254
Table 15: AWD values (the lower, the better) for synthetic data Syn4 whose feature-label relationships are non-linear,
28


Reviews:
[{"review_id": "djarL8UL7w", "reviewer": "Reviewer_2fzZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n[Strengths]\n- The paper combines ideas from sub-sample selection and local model interpretability in a compelling way\n- For a purely technical perspective the approach seems sound and correct. The use of REINFORCE for sampling discrete sample weights seems reasonable (and perhaps unavoidable). My only question/comment in this regard is that it's not clear to me why the training phase samples discrete {0,1} weights for fitting the locally-interpretable models (Algo 1: 5) whereas at test time the dense [0,1] weights are used instead (Algo 2: 3). The reason for this discrepancy does not seem to be discussed in the paper. \n- From a purely quantitative standpoint, the method seems to be perform well in terms of local functional approximation compared to other methods\n- The paper overall is well written, easy to follow, and clear in its argumentation and derivations\n\n[Weaknesses]\n- The method puts a heavy burden of interpretability on the instance-wise weight estimation model, which raises some issues (see below for details). The instance-wise weight estimation model (which is itself a deep learning model) seems key to achieve strong local fidelity but at the same time can potentially transfer the 'black-box-ness' from one  place to the other. In other words, the complex and highly-nonlinear interaction between sample selection of locally-fitted model might be hard to explain to a user, and its effect on the is overall interpretability of the approach is unclear.\n- The experiments leave many unanswered questions and could use more explanation, better baselines, and more analysis. In particular:\n    - The hand-crafted datasets (the 'black-box models' in this case) seem to be perfect fits for this method: piecewise linear functions over some linear and some non-linear regions. Clearly, any method with any reasonable 'bandwidth'/sample selection and locally-fit linear models should be able to recover the functions. So the results of Section 4.1 don't provide much information, and SILO and MAPLE seem like poor choices of baselines for this task (there's a very extensive literature of locally-linear interpretability models - why not pick one of those?). In addition, the reasons for LIME failing in this task are not very clear to me.\n     - In Section 4.2, the paper states \"if λ is too small, LIMIS selects too many instances and deteriorates fidelity (due to overfitting)\". Is this for a ridge regression local model? How can a linear regression model overfit? Is this Kernel RR?. Furthermore, the results in Fig  4 are, again, not very insightful, except to show that this method has a heavy dependence on the λ parameter, and its impact of the interpretability itself might be strong.\n     - The results of 4.3 are perhaps the most informative of this Section, and highlight that that ideal use cases of this method are on large-N scenarios\n- I am not sure what's the point of Section 6: LIMIS Explainability Use Cases. It doesn't seem to specifically discuss use cases of this method (especially when compared to other methods). Figure 5 is poorly explained, and its interpretation requires heavy hand-holding. The examples in 6.2 rely on a method to generate counterfactuals (shortest path from prediction to desired outcome) that warrants further motivation. At any rate, these are two experiments where some form of human validation would have been useful. In 2022, an interpretability paper without an attempt to evaluate the method in-context with human subjects does a disservice to itself. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new approach to local model interpretability based on an instance-wise subsampling method. At the core of the method -- and its main difference compared to other local interpretability methods -- is that it fits an instance-wise weight estimator to identify the importance of the training samples to explain a test sample. A locally-interpretable model is then trained on this subset of examples, obtaining the explanation. The method is validated on various synthetic and simple ML benchmark datasets, showing strong performance in terms of fidelity/robustness quantitative metrics when compared to other similar interpretability methods.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\n- Provide explanation for the discrepancy between the hard (sampled) weights during training of the instance-wise weight estimation model, compared to the use of soft (non-sampled) weights during test time. [Critical]\n- Add a broader impact statement with disclaimers and discussion of its limitations for use in real applications [Critical]\n- Validation with human subjects [Nice to Have].\n- Remove or re-frame Section 6 as it currently seems to serve to specific purpose [Nice to Have]\n- A more detail runtime analysis of training vs testing of the model in the all the benchmark datasets not just Facebook. In particular: effect of lambda and convergence criterion for Algorithm 1. [Critical]\n- A deeper qualitative analysis of the instance-wise weight estimator in the benchmark datasets. It would be interesting to see what kind of examples it up/down-weights in which situations. What can we learn about the model from the lens of instance-wise importance? [Nice to Have] \n- Typos: \"they should\" -> \"it should\" in Ph 2 \n- In-text citations should use \\citet instead of \\citep ", "broader_impact_concerns": "\nThere is no broader impact statement. But it is my opinion that **any paper proposing a interpretability method for ML should come with disclaimers of its limitations**. This is particularly important given the widespread mis-use and mis-understanding of explainable AI methods (see, e.g. Kaur et al \"Interpreting Interpretability: Understanding Data Scientists’ Use of Interpretability Tools for Machine Learning\", CHI 2020). ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gNnJmR7MAD", "reviewer": "Reviewer_ny2W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nIn terms of significance, the problem of developing interpretable predictions is important, since people usually need to explain their decision-making process to stakeholders.\n\nIn terms of quality, the paper provides good evidence to support that its locally interpretable predictor makes good predictions and provides explanability. I do have questions about the specifics of some of the experiments (please see the Weaknesses section).\n- Table 2 shows that the proposed method has higher fidelity than existing baselines to train locally interpretable predictors. It is good to know that baseline techniques on locally interpretable predictions are actually performing worse than the ``sanity check'' of training one globally interpretable model, while the proposed LIMIS is consistently improving upon it.\n- The paper provides estimates on computational complexity (Section 3.2) and actual runtimes (Section 5.3), which are useful information to decide whether to run a technique or not.\n- It’s good to have the synthetic experiments in which the ground truth is known and can be compared against.\n- As somebody new to the local interpretability literature, I greatly appreciate Section 6, which is about explainability use cases.\t\n\n# Weaknesses\nIn terms of clarity, the paper is not clear in its discussion of Stage 2, and some experiments are missing important experimental details.\n- In the methodological section, the presentation of Equation 1 and the discussion in Stage 2 ultimately talk about different optimization problems. It’s unclear what is the final optimization problem that Stage 2 arrives at after its proposed series of approximation. As a result, I don’t understand to what optimization problem is policy-gradient applied to. To me, Equations 2 and 3 are stochastic gradient descent applied to the $l(\\phi)$ objective – I am willing to be convinced otherwise. I also don’t understand the sentence “The sampler block yields a non-differentiable objective …’’, which is tied to the lack of a clearly spelled-out optimization problem.\n- Regarding the experiments, I have some questions about experimental details that I could not find in the main text.\n  - What do the differentiable baselines in Section 4.3 look like? This is related to the point above about the lack of a clearly spelled-out optimization problem in Stage 2.\n  - What does $X_{10}$ mean in the context of Syn1 through Syn3? The data-generating process involves objects like $X_1, X_2$ but there is no explanation of $X_{10}$.\n  - How was figure 3 generated? How is the mean AWD computed across 10 runs? Is it taken by averaging the AWD for individual test points, or by aggregating test points whose distance from the boundary lives in a small bin?\n    - Relatedly, why is figure 3 a line plot and not a scatter plot? If I understand distance from the boundary and AWD correctly, for each independent run, there should be M (the number of test points) scatter plot points, where the x-axis is the distance of that test point from the boundary, and the y-axis is the AWD for that test point. Perhaps there is some post-processing of the individual scatter plots, since Figure 3 reports results over 10 independent runs – if so, it would be good to discuss.\n  - How was Figure 5 generated? There is no discussion of how the feature importance was constructed. Is the feature importance constructed using just the instance-wise weight estimator, or is it constructed using the local predictor as well? What is the exact definition of feature importance?  \n  - How is the difference between neighbor MAE and pointwise MAE is a measure of robustness? The neighborhood metric is not discussed in either main text or appendix. Since I don’t understand what the neighborhood metric is, I don’t understand how the difference between the two MAEs is a measure of robustness. I am willing to be convinced, but I would need to see the argument clearly spelled out.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes (and verifies the performance of) a method to train locally interpretable predictors for both classification and regression. The locally interpretable predictor mimics the predictions of highly accurate but not interpretable predictors i.e., black-box predictors. Examples of globally interpretable predictors include a linear model (with linear features) and decision trees of small depth. A locally interpretable predictor’s evaluation at a particular test point is equal to the evaluation of a globally interpretable predictor, but for different test points, the globally interpretable predictors are different from each other.\n\nMethodologically, the paper first postulates that the globally interpretable predictor at each test point is the empirical risk minimizer under a data reweighting; because the data reweighting is different across different test points, the globally interpretable predictors are different across test points, even though the hypothesis class is the same (either linear model or decision trees). The tunable parameters of this locally interpretable predictor is the data reweighting mechanism $h_{\\phi}$: the paper uses a policy-gradient routine (Equations 2 and 3) to find a good data reweighting mechanism.\n\nEmpirically, the paper shows that the locally interpretable predictor is a high-fidelity approximation of the black-box predictor (Figure 3, Table 1, Table 2, Table 3). They also discuss how the locally interpretable predictor can be used in some explainability use cases: Figure 5 illustrates the discovered feature importance while Figure 6 illustrates counterfactual reasoning.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "# Important\t\nWrite out all aspects of the optimization problem (objective function, decision variables, constraints etc) in Stage 2.\n\nExplain how Figure 3 was generated (and fix the ticks on the x-axis: if you are reporting percentiles, shouldn’t the ticks range from 0 to 100 rather than from 0 to 1?)\n\nExplain how Figure 5 was generated. Section 6 will be stronger if the paper defines feature importance rigorously and show how the locally interpretable predictor yields feature importance.\n\n# (Less important) discussion questions\nPreface: These discussions would be nice to have but not necessary\n\nDoes the instance-wise weight estimator provide some form of decision boundary? If it does, can we visualize it? The synthetic datasets by design had decision boundaries; if the instance-wise weight estimator provides an estimate of boundary, can we determine the approximation error?\n\t\nIs stage 0 and stage 1 necessary? Can we directly use the labeled training data (perhaps with a split into two datasets) to train the instance-wise weight estimator? \n", "broader_impact_concerns": "I do not see any ethical concerns arising from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cFGkfHHIie", "reviewer": "Reviewer_kxUv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe method clearly outperforms existing methods on the synthetic datasets in terms of the quality of local explanation.\n\n## Weaknesses\n### The choice of data-weighted training for interpretability\nIt's unclear to me if a model trained using weighted data can be still seen as a model to give the same interpretation.\nThe paper needs to give some intuitions on why it actually makes sense.\nFor example, I could imagine changing the weights would dramatically change the decision boundary thus giving a different interpretation.\nOn this regard, only one experiment with ground-truth interpretation is performed---I would see more to be convinced.\n### The necessity of policy gradient\nIn section 4.3, it argues that using actual sampling with policy gradient outperforms other tricks to deal with discrete sampling for large $N$.\nHowever, the appears to me that the performance of Gumbel-Softmax and STE are pretty close, rendering me the question: Is it worth the extra computation to use the REINFORCE trick here?\nMaybe the author could convince me by answering the following questions:\n1. How much time does each of the experiment take in the table? Can you also show some training curve to demonstrate how fast each way converges?\n2. How the temperature is set for Gumbel-Softmax? Do you use any scheduling on it?\nI would suggest the author(s) change(s) the claim if the REINFORCE trick doesn't seem to be necessary.\n\n## Questions\n- Where is $N_{mb}$ defined in (2)?\n- Would the inconsistency of sampling vs fixed instance-wise weight gives a worse performance? To me it makes more sense to using the targeted number of instances with the highest weights, essentially replacing sampling with the mode.\n\n## Misc\n- The notation for multiplication is inconsistent: Sometimes omits the notation means multiplication, sometimes $\\cdot$ is used and sometimes $\\times$ is used. Please choose one and stick to it.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new method to obtain local interpretation based on instance-wise subsampling.\nThe method trains a local interpretable model based on instance dependent data-weighted training where weights are from a neural network.\nDuring training, in order to encourage exploration of different weighting, the method selects a subset of data based on sampling,\nwhich causes non-differentiability in the training.\nTo solve this, the REINFORCE trick is used to train the weight estimation model.\nThe paper shows that this method significantly outperforms existing methods in terms of explanation quality and prediction accuracy.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "## Must-have\n- Explain why data-weighted training would give a model with consistent local interpretation\n- More synthetic experiments, especially with different ground-truth models.\n- Demonstrate the necessity of using REINFORCE\n", "broader_impact_concerns": "Although the paper does discuss things like fairness when the Adult dataset is used, this paper doesn't include a section for broader impact or ethical.\nGiven that the main topic is interpretability, I suggest to add a dedicated section for broader impact on how a more accuract model for local interpretation would help.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BcKLmgIOXL", "reviewer": "Reviewer_Hu5A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-  LIMIS is an intuitive solution to the problem of neighbor selection for local approximation-based explanations.  \n-  Using synthetic experiments to highlight the failure cases from past methods that LIMIS addresses is compelling.  \n-  The real dataset experiments show that LIMIS produces higher fidelity explanations than past methods.  \n\nWeaknesses:\n-  I found \"Section 3.1, Stage 2\" hard to understand.  Starting with an abstract summary (eg, something like \"Our goal is to define x.... to do this we need to define y.... once we have done that, we make simplifications in order to make optimization tractable\") and then presenting the details of each of the abstract steps would be helpful (at least for me).  \n-  I'm not sure that the definition of the ground-truth is as simple as presented in Section 4 (see Requested Changes).  Correctly defining the ground-truth is important for these results to be accurate.  \n-  I generally find the neighorhood fidelity metric to be much more compelling than the point-wise version (because the latter has pathological failure cases [eg, the useless explanation of $g(x) = 0^t x + f(x)$ gets a perfect score]).  However, the results for this metric have some missing details (see Requested Changes).  \n-  I'm not sure that the results for Section 6 support the claims (see Requested Changes).  At a high level, it seems like the changes required to support the goal of this section are too large to be made in the review period.  But the paper is publishable without this section (if other concerns are addressed).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new local approximation-based explanation system, LIMIS.  \n\nIn general, this type of explanation works by sampling some points from a neighborhood around the point being explained and then fitting an interpretable model to those sampled points.  As a result, a key techinical detail is \"how are those sampled points chosen?\"\n-  LIME does this using euclidean distance. \n-  SILO/MAPLE do this using a tree ensemble.\n-  LIMIS does this using a secondary model that is trained to optimize the fidelity of the resulting explanations.  \n\nLIMIS's approach can minimize problems resulting from the difference in the representation capacity between the black-box model being explained and the interpretable model being used as the explanation by:\n-  explicitly finding a small set of important points for the explanation to imitate.\n-  directly optimizing for fidelity.  \n\nExperimentally, the benefits of LIMIS are demonstrated in several ways:\n-  Section 4) LIMIS is better able to identify the ground-truth in synthetic experiments.\n-  Section 5) LIMIS produces higher fidelity explanations. \n-  Section 6) LIMIS is useful in practice (I'm not sure that this section actually supports this).  \n\nFor TMLR's two reviewing criteria:\n-  Does someone care about this problem and these results?  Definitely\n-  Are the results and claims supported?  Partially\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Abstract:\n-  \"An alternative approach is to explain individual predictions using locally interpretable models.\"  \n    -  Based on this (and the rest of the abstract), I wasn't certain if the paper is about 1) training a black-box model and then using post-hoc local explanations to explain that model or 2) training a by-design locally explainable model to replace the black-box model entirely. \n    -  Please clarify that it is (1).  \n    -  Minor\n\nIntroduction:\n-  \"They can be utilized ... data sample).\"\n    -  Please provide for citations that demonstrate these explanations being successfully used for these use cases. \n    -  Minor \n-  \"LIMIS offers the unique capability of instance-based explainability via ranking of the most valuable training\ninstances\"\n    -  This claim doesn't seem to be accurate because MAPLE also assigns assigns an importance weight to each training instance based on the instance being explained.  \n   -  Please revise this claim.  \n    -  Major\n\nRelated Work:\n-  \"SILO (Bloniarz et al., 2016)) aims to improve LIME.\"\n    -  This doesn't seem to be accurate because 1) SILO doesn't cite or discuss LIME and 2) the author's do not present SILO as an interpretability method (they only mention that local models are interpertable once and do not evaluate SILO from an interpetability perspective).  \n    -  Please revise this claim.  \n    -  Major\n-  Given Section 2.2 of SILO, how is the use of a tree-ensemble to learn local neighborhood weights \"ad-hoc\"?\n    -  Minor\n\nSynthetic Data Experiments\n-  Why should an explanation method that is meant to approximate the black-box model in some neighborhood around the point being explained return exactly \"w\" near the boundary?  If anything, this seems like it wouldn't let the user know that the black-box model has a discontintuity nearby and, therefore, won't behave as expected.  \n    -  Please comment on this.  \n    -  Major\n-  For Figure 3, where are LIME's results?\n    -  Minor\n\nRobustness of Explantions\n-  In the context of explanations, \"robustness\" is often used to refer to \"adversarial robustness\" or other measures of how sensitive the explanation is to small perturbations of the point being explained.  \n    -  Please consider calling this something like \"generalization\" since this is more aligned with what neighorhood fidelity measures.  \n    -  Minor\n-  What is the neighborhood size used in Table 3?  Without knowing this, these results cannot be interpretted.  \n    -  Critical\n-  Where are the results for the other datasets and black-box models in Table 3 (compared to Table 2)?\n    -  Major\n\nLIMIS Explainability Use Cases\n-  \"we highlight unique explainability capabilities of LIMIS for human-in-the-loop AI deployments.\"  In what ways are these unique?  LIMIS may produce higher fidelity explanations than LIME/MAPLE, but it isn't clear that LIME/MAPLE can't be used for these use-cases.  \n    -  Please comment on this.  \n    -  Major\n-  \"LIMIS can also be useful for improving a human’s reasoning accuracy, shortening their reasoning time, or increasing their bias detection capability.\"  This claim does not seem to be supported by the two experiments presented here (which are purely qualitative).  \n    -  Please comment on this.  \n    -  Critical\n-  \"LIMIS does not discover notable biases of the black-box model, via significant dependence on gender, marital status and race features.  These qualitative results demonstrate how the proposed method helps humans to interpret the decision of the machine learning model.\"  What do direct measurements of bias say? Without knowing whether or not the model is biased, we cannot tell if LIMIS is correct or not.  \n    -  Please comment on this.  \n    -  Major\n-  \"Such a capability can be particularly useful in providing insights on the closest counterfactual input that\nwould yield a different outcome ... this capability can be efficacious.\"  Table 4 provides example counterfactuals but doesn't support the claims that this is the *closest* counterfactual or evidence that users can use these counterfactuals to better perform some task.  How does this result show that LIMIS is useful to end-users?  \n    -  Please comment on this.  \n    -  Major", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you to the authors and reviewers for the great discussions.   Throughout the discussions, most of the critical questions/concerns raised by the reviewers have been clarified/addressed.  All reviewers believe the updated version reads better and crosses the threshold for acceptance.   Some of the remaining issues that authors need to consider addressing/discussing include 1) the fairness claim and the general goal of the "use-case" section raised by Reviewer Hu5A, and 2) the issue of the non-interpretability of the weight estimation model and the relatively limited scope of the experiments raised by Reviewer 2fzZ.  Again, thank you very much for the great work of both the reviewers and authors.  
 
Best,
 
AE

Decision: Accept (poster)