Regularizing Black-box Models for Improved Interpretability
Authors: Gregory Plumb
Maruan Al-Shedivat
Eric Xing
Ameet Talwalkar
Venue:  (2020)
Keywords: Interpretable Machine Learning
Local Explanations
Regularization

Abstract: Most of the work on interpretable machine learning has focused on designingeither inherently interpretable models, which typically trade-off accuracyfor interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality.  We explore an alternative to theseapproaches by directly regularizing a black-box model for interpretabilityat training time.  Our approach explicitly connects three key aspects ofinterpretable machine learning:  (i) the model’s internal interpretability, (ii)the explanation system used at test time, and (iii) the metrics that measureexplanation quality.  Our regularization results in substantial improvementin terms of the explanation fidelity and stability metrics across a range ofdatasets and black-box explanation systems while slightly improving accuracy.  Finally, we justify theoretically that the benefits of our regularizationgeneralize to unseen points.

Under review as a conference paper at ICLR 2020
REGULARIZING BLACK-BOX MODELS FOR IMPROVED INTERPRETABLITTY
Anonymous authors Paper under double-blind review
A ABSTRACT
Most of the work on interpretable machine learning has focused on designing either inherently interpretable models, which typically trade-off accuracy for interpretability, tees about their explanation quality. We explore an alternative to these approaches by directly regularizing black-box model for interpretability at training time. Our approach explicitly connects three key aspects interpretable machine learning: (i) the model's inherent interpretability, (ii) the explanation system used at test time, and (iii) the metrics that measure explanation quality. Our regularization results in substantial improvement in terms of the explanation fidelity and stability metries across a range of datasets and black-box explanation systems while slightly improving accu- racy, Finally, we justify theoretically that the benefits of our regularization generalize unseen points.
post-hoc explanation systems, which lack guaran-
INTRODUCTION
Complex loarning-based systems incrcasingly shaping our daily lives, and, in order to monitor and understand these systems, we require clear explanations of model behavior. While model interpretability has many definitions and is often largely application specific (Lipton, 2016), local explanations are popular and powerful tool (Ribeiro et al., 2016). Recent work on local interpretability in machine learning ranges from proposals of new models that are interpretable by-design (e.g., Wang and Rudin, 2015; Caruana et al., 2015) to model-agnostic post-hoc algorithms for interpreting complex, black-box predictors such as ensembles and deep neural networks (c.g., Ribeiro et al., 2016; Lei et al., 2016; Lundberg and Lee, 2017; Selvaraju et al., 2017: Kim et al., 2018). Despite the variety of technical approaches, the underlying goal of all of these works is to develop interpretable predictive system that produces two outputs: a prediction and its underlying explanation.
Both interpretability be-iesign and post- hoc explanation strategies have limitations. On one hand, the by-design approaches are restricted to working with model families that provide inherent interpretability, potentially at the cost of accuracy. On the other hand, by performing two disjoint steps, there is no guarantee that post-hoc explainers applied to an arbitrary model will produce explanations of suitable quality. Moreover, recent approaches that claim to overcome this apparent trade-off between prediction accuracy and explanation quality are in fact by-design proposals that impose certain constraints on the underlying model familics they consider (e.g., Al-Shedivat et al., 2017; Plumb et al., 2018; Alvarez- Melis and Jaakkola, 2018a). In this work, we propose a novel alternative strategy called Eaplanatioon-lssed Oplimization (ExpO) that aims to address both of these shortcomings by adding an interpretabilily regularizer the loss function of an arbitrary predictive model. We illustrate how ExpO can influence the interpretability and accuracy of model in Figure 1 (left).
Illustration. Consider a situation where Bob's loan application is denied by machine learning system; see the toy illustration in Figure (right). Here, good local explanation can help Bob understand how improve his application in order get the loan. Unfortunately, standard model, a multi-layer perceptron trained with SGD, is not very interpretable because it has many abrupt changes. Indeed, we can quantitatively measure the local interpretability

---

Under review as a conference paper at ICLR 2020
UCI Housing Regression Dataset
LIME Neighborhoos Fidelity
0.4
ExpO
Decision Tiee
441
0.2
Rendom Forest
0.0001
Linear Regression
0.001
0.0
0.01
Tres THe den
0.1
0.25
Model Papp on Hoald
0.15
0.20
0.25
0.30
Mean Squared Error
Figure 1: Left: Neighborhood Fidelity of LIME-generated explanations (lower is better) vs. predictive error of several models trained on the UCI 'housing" regression dataset. The values in blue denote regularization weight of ExpO; note that it can improve both accuracy and interpretability. Right: The effects of ExpO on a model predicting a hypothetical credit rating. The kinks in the unregularized model make local linear explanations have lower fidelity and be less stable to small perturbations. The regularized model much smoother and more interpretable.
of the model using the standard fidelity (Ribeiro et al., 2016; Plumb al., 2018) and stability (Alvarez-Melis and Jaakkola, 2018a) explanation metrics. To make the learned model more amenable to local explanation, ExpO augments the objective function with fidclity- or stability- regularizers, effectively controlling the degree of local explainability.
The specific contributions of our work are as follows:
1. Interpretability regularizers. We introduce two interpretability regularizers associated with the fidelity and stability explanation metrics. The first. ExpO-Fidelity, is designed for semantic features and explainers that dircctly make predictions, such as those in Ribeiro et al. (2016); Lundberg and Lee (2017); Plumb et al. (2018). The second, ExpO-Stabilne is tailored for non-semantic features (e.g., pixels) and explainers that identify features that are influential on a prediction, such as saliency maps (Simonyan el al., 2013). Both regularizers are differentiable and can be used to augment the objective function of an arbitrary model. In Section 3.1, we discuss how they differ from the classical approaches for local approximation and function smoothing.
2. Generalizable explanation quality. We analyze the properties of the explanation quality metrics and show that the benefits of our regularization generalize to unseen points. Specifically,
derive bound on the gap between the fidelity of explanations training and held out points and connect it with the local variance of the learned model.
Empirical results. We evaluate models trained with and without our regularizers on variely of regression and classification lasks with semantic and image features." Empirically, they slightly improve predictive performance across the nine datasets we consider (seven UCI regression tasks, medical classification task, and MNIST). From an interpretability perspective, our results demonstrate significant improvement in terms of explanation quality as measured by the fidelity and stability metrics. In particular, our regularizers improved explanation fidelity by at least 25% on the UCI datasets and on the medical classification found,
task while stability on MNIST was improved by orders of magnitude. Additionally, qualitatively, that our regularizers produce models with simpler and more comprehensible explanations. In summary, black-bor explanation systems work better on models trained with ExpO and, as an additional benefit, those models tend to be more accurate.
BACKGROUND AND RELATED WORK
Consider supervised learning problem, where our goal is to estimate model, & }/ f e F, that maps input feature vectors, x e X, to targets, y E D, and is trained using data, W. If the class of functions used for modeling the data is complex, can
The code our regularizers experiments https://gtttubb.com/aaarar
2

---

Under review as a conference paper at ICLR 2020
understand the behavior of f in some neighborhood, N₂ P[x] (where P[X] is the space of probability distributions over X), by generating a local explanation.
We denote algorithms that produce explanations (i.e., explainers) the set of possible explanations. The choice of & generally depends on whether or not X consists of semantic features, and will be defined more preciscly next.
X' F E, where
2.1 SEMANTIC FEATURES
We call features semantic if people can reason about them and understand what it means when their values change (e.g., person's income, the concentration of chemical, etc.). Consequently, local explanations try to predict how the model's ' output would change if the input was perturbed (Ribeiro al., 2016; Lundberg and Lee, 2017; Plumb et al., 2018), Thus, we can define the output space of the explainer as & == وg E G g X + V}, where G class of interpretable (typically lincar) functions.
Fidelity metric. When the explainer's output space is E,, the explanation is defined as function 9 X' + y, and it natural to evaluate how accurately g models f in a neighborhood N₂ (Ribeiro et al., 2016; Plumb et al., 2018):
F() Ra [((x')) (())))
(1)
which we refer to as the neighborhood with N₂ as a point mass on æ and we call this version the point-fidelity (PF) metric. While Plumb et al. (2018) argued that point-fidelity can be misleading because it does not measure generalization of e(x, f) across N2. it has boen used for evaluation in the prior work (Ribeiro et al., 2016; Lundberg and Lee, 2017; Ribeiro et al., 2018) and we report it in our experiments along with the weighborhood-fidellt for completeness.
(NF) metric. This metric is sometimes evaluated
Black-box explanation systems. Various explainers have been proposed to generate local explanations of the form g X' V, typically assuming that g is linear. In particular, LIME (Ribeiro et al., 2016), one of the most popular black-box explanation systems?,, solves the following optimization problem:
e(z, arg min ((fgg,N2) + R(g),
(2)
,EE,
where M(e) stands for additive regularizer that encourages certain desirable properties of the explanations (e.g., sparsity). LIME's objective function is closely related to the fidelity metric and subsequently to our proposed ExpO-Fidelity regularizer. Consequently, we expect our regularizer to improve the quality of LIMEi-gnnrrted explanations. Our experimental results in Section 4.1 corroborate this hypothesis.
Along with LIME, we consider another black-box explanation tool, called MAPLE (Plumb et al., 2018). It differs substantially from LIME in that its neighborhood function is learned from the data rather than specified as parameter. In our experiments, we evaluate the quality of MAPI.E-generated local explanations for models regularized via ExpO-Fidelity, but do not use MAPLE's learned neighborhood function to define ExpO-Fidelity. We view this as a good test case to see how optimizing the fidelity metric for one neighborhood generalizes to another one (see Section 3 for more detailed discussion of this point). J. In Section 4.1, we see that rogularizing for LIME neighborhoods improves MAPLE's explanation quality.
2.2 NON-SEMANTIC FEATURES
Non-semantic features lack inherent interpretation, with images being canonical example. When X' consists of non-somantic inputs, we cannot assign meaning to the difference between and x', hence it does not make sense to explain the difference between the predictions fx) and (aa). As a result, fidelity is not an appropriate explanation metric. Instead, in this
"SHAP (Lundberg and Lee, 2017) is another popular method that proposes theoretically- motivated neighborhood sampling function, but requires explanations to be linear models that act on binary features. This requirement too limiting in our case.
3

---

Under review as a conference paper at ICLR 2020
context, local explanations try to identify which parts of the imput are particularly influential on a prediction prediction (Sundararajan et al., 2017). Consequently, we consider explanations of the form Ens := Rd, where d is the number of features in X.
Stability metric and saliency maps. When the explainer's output space is Ens, the explanation a vector in R and cannot be directly compared to the underlying model itself, in the case of the fidelity metric. Instead, the focus in this setting is on the degree to which the explanation changes between points in local neighborhood, which we measure using the stability metric Levarez-Melit and Jaakkola, 2018a):
eff.e, == Bx' Ile((, f) - e(æ', f)ll
(3)
Various explainers (Sundararajan et al., 2017; Zeiler and Fergus, 2014; Shrikumar et al., 2016; Smilkov et al., 2017; Adebayo et al., 2018) have been proposed to generate local explanations in Ens. with saliency maps (Simonyan et al., 2013) being the approach that we consider in this work. Saliency maps assign importance weights to image pixels based on the magnitude of the gradient of the predicted class with respect to the corresponding pixels.
Recent work on model interpretability emphasizes that more stable explanations tend to be more trustworthy (Alvarez-Melis and Jaakkola, 2018a; Ghorbani et al., 2017; Alvarez-Melis and Jaakkola, 2018b). Note that the stability metric can also be considered in the context of semantic features in addition to the fidelity metric.
2.3 RELATED METHODS
A few recently proposed approaches to model interpretability are closely related to our work. First, self-explaining neural networks (SENN) (Alvarez-Melis and Jaakkola, 2018a) (a variation of contextual explanation networks (Al-Shedivat et al., 2017)) is an interpretable by-design approach that additionally (indirectly) optimizes their models to produce stable explanations. Second, "Right For The Right Reasons" (RTFR) (Ross et al., 2017) selectively penalizes gradients of the output with respect to certain input features at some points to discourage their use by the model. Finally, a work concurrent with ours (Lee et al., 2019), which expanded on (Lee et al., 2018), proposed to regularize models of structured data to encourage ex explainability in a way that is similar to ExpO but differs substantially in how the the target explanation for the regularizer is defined and in how the final objective function is optimized.
From a technical standpoint, SENN and RTFR both assume that the local explanation is close to the first order Taylor approximation of the model at that point. In Section 3.1, we demonstrate how Taylor approximations are often quite different from and more difficult to use than the neighborhood-basee regularizer requires the neural work to have a very particular structure and, therefore, unlike ExpO, cannot by applied to an arbitrary model. While RTFR's regularization can be used with arbitrary models, it is not directly related to a measure of explanation quality and is defined using specific domain knowledge; on the other hand, ExpO aims to directly improve quality of explanations with respect to a specific metric and does not require domain knowledge. In the Appendix A.1, we compare ExpO to simple I and le regularization since other basclines are either model specific or require domain knowledge. We found that they do not significantly impact model interpretability.
local explanations that we use in ExpO. Further, SENN's
3 EXPLANATION OPTIMIZATION
Running black-box explainers on arbitrary models does not guarantee the quality of the resulting explanations. To address this, define regularizers that can be added to the loss function and used to train an arbitrary model f. Specifically, we want to solve the following optimization problem:
N (C( w:) +RIS, (N
f :== arg min
Σ
fEF N
4

---

Under review as a conference paper at ICLR 2020
where C(f. F1, yr) is standard predictive loss (c.g., squared error for regression or cross- entropy for classification), R(f. Nr.)) is a regularizer that encourages explainability of f in the neighborhood of 2g, and y > 0 controls the regularization strength. Because our regularizers are differentiable, we can solve Equation 4 using any standard gradient-based algorithm; in our case, SGD with Adam.
We define RJf R based on either the neighborhood- fidelity, Eq. (1), or the neigiborrhodd-stabbiity Eq. (3). In order to compute these metrics exactly, we would need run an explainer algorithm, e; this may be non- differentiable or too computationally expensive to use as a regularizer. Thus, for ExpO-Fidelity, we approximate using local linear model fit on points sampled from Ng (Algorithm 1). For ExpO-Stability, we simply require that the model's output not change too much across Nreg (Algorithm 2).
Algorithm ExpO-Fidelity
input So. v, NT",
1: Sample: 2i...., Km N
2: Compute predictions:
}5(0) = fo(nz) for jor:=1 I....,m
3: Produce a linear explanation:
9,(0) arg ming Σ (0,(0) 2)))
output mo Ey= (0) - 0z(0))x))
Choosing the neighborhood. Defining good regular- ization neighborhood, requires considering the following. On one hand, would like N to be similar to Nz, used in Eq. 1 or Fq. 3, so that the neighborhoods used for regularization and for evaluation match. On the other hand, we also would like Nz to be consistent with the 'local neighborhood' defined by e internally, which may differ from Nz. For LIME, this is not a problem since the internal definition of the 'local neighborhood is a hyperparameter that we can set. However for MAPLE, the 'local neighbor- hood' is learned from the data, and hence the regularization and explanation neighborhoods may differ. Ultimately, we left resolving this tension to future work.
A Algorithm ExpO-Stability
input fo. N
1: Sample: 21...., Zm ~ N 2: Compute predictions:
(P) = fo(a)), for 1, 1....
output nr -, (v. (0) fe(x)))
Computational cost. Algorithm 1 could be prohibitively expensive since the number of samples, m, from Nyea, has to be proportional to the dimension of x, d, resulting in (d3) evaluations of f to compute the regularizer at x. So we also test a randomized version of Algorithm 1, ExpO-1D-Fidelie to No and penalizes the error of a local linear model along that dimension. This breaks the dependence of the computational cost of the regularizer on d and allows us to compute each gradient step with O(1) increase in the number of evaluations of f-
that randomly selects one dimension of x to perturb according
3.1 UNDERSTANDING THE PROPERTIES OF EXPO
The goal of this section is to compare the behavior of local linear explanations and our regularizers to some existing theoretical function approximations and measures of variance to help develop an intuitive understanding of ExpO. First, we compare ncighborhood-ban based local linear explanations to first order Taylor approximations to show that they can have fundamentally very different behaviors. Second, we compare ExpO-Fidelity to the Lipchitz Constant (LC) and Total Variation (TV) of the learned function.
Local explanations vs. Taylor approximations. A natural question to ask is, Why should we sample from N₂ in order to locally approrimate f when there are easier and theoretically motivated approcimations? One possible way to do this is via the Taylor approximation (Alvarez-Melis and Jaakkola, 2018a). The downside of a Taylor approximation- based approach is that such approximation cannot readily be adjusted to different neighborhood scales and its fidelity and stability strictly depend on the Icarned function. This can be seen in Figure 2 (left) where the Taylor approximations at two nearby points are both radically different. and not faithful to the model outside of an small neighborhood.
Fidelity regularization and the model's LC TV. From theoretical perspective, our regularizer is similar to controlling the Lipschitz Constant or Total Variation of f across N₂ after removing the part of explained by e(x, f). From an interpretability perspective, there is nothing inherently wrong with having a large LC or TV, which is demonstrated in
AA similar procedure was explored previously in (Zheng et al., 2016) for adversarial robustness.
5

---

Under review as a conference paper at ICLR 2020
12
approomation a Tayler approdmatiin a
100 925
0.5 110
0.4
12
Figure 2: Left: A function (blue), its first order Taylor approximations at 0.4 (green) and 2 0.5 (red), and local explanation of the function (orange) computed with 0.5 and N₂- 1 Right (top row): Two functions (blue) and their local linear explanations (orange). The local explanations were computed with x = 0.5 and N₂ [0. 1]. Right (bottom row): The unexplained portion of the function (residuals).
Figure 2 (right). However, once we take into account what be explained by e(ar, f), then upper bounding any one of ExpO-Fidelity, the LC, or the TV will upper bound the others.
3.2 GENERALIZATION OF LOCAL LINEAR EXPLANATIONS
To conclude our analysis, we study the quality local lincar explanations in terms of generalization. Note that ExpO regularization encourages learning models that are explainable in the ueighborhoods of each training poinl. However, how would this properly generalize to unseen points? We answer this question by providing generalization bound in terms of neighborhood-ficee didiediiitty metric for local linear explanations (see Appendix A.2 for derivations).
Proposition Let the neighborhood sampling function N₂ be characterized by some param- eter (e.g., the effective radius of neighborhood) and the variance of the trained model f(x) across all such neighborhoods be bounded by some constant C(a) > 0. Then, the following bound holds with at least probobilily:
(²(o) log
E [((ff
Σ r(f.za) +
(5)
2n
Remark The obtained bound tells that explainable models with smaller local variances across the neighborhoods are likely to have explanations of higher fidelity on the held out points. This further motivales the approcimation used in Algoritlm 2.
4 EXPERIMENTAL RESULTS
In our first set of experiments, demonstrate the effec- liveness of ExpO-Fidelity and ExpO-1D-Fidellty on datasets with semantic features using seven regression problems from the UCI collection (Dheeru and Karra Taniskidou, 2017) as well as an in-hospital mortality classification problem. Dataset statistics are in Table 1. Our second ex- periment demonstrates the effectiveness of Stapilttayllty for creating saliency maps (Simonyan et al., 2013) on MNIST (LeCun, 1998). We found that a model trained with our regularizers is more interpretable than model trained without them because black-box explainers pro- duce quantitatively better explanations for them; further, they are often more accurate. Finally, we demonstrate qualitatively that the explanations for the regularized model tend to be simpler.
Dataset
# samples
# dims
autompgs
392
communities
1993
day
731
14
housing
506
11
1059
winequality-red MSD SUPPORT2
1599
11
515315
90
9104
51
MNIST
60000
784
Table 1: Statistics of the datasets.

---

Under review as a conference paper at ICLR 2020
4.1 NEIGHBORHOOD-FIDELN REGULARIZATION
We compare models trained with our regularizers to models trained without them. We report accuracy and three interpretability metrics: Point-Fidelity (PF), Neighborhood-Fideee and Stability (S). The interpretability metrics are evaluated for two black-box explanation systems: LIME and MAPLE. Consequently, the "MAPLE-PPS label corresponds to the Point-Fidelity Metric for explanations produced by MAPLE.
(NF),
Experimental setup. All of the inputs to the model were standardized to have mean zero and variance one (including the response variable for regression problems). The network architectures and syper-paramett results, set N₂ to be M(x, o) with a 0.1 and Nrea to be (z, with o 0.5. In the Appendix A.3, we discuss how these values were chosen.
were chosen using a simple grid search. For the final
UCI regression experiments. The effects of ExpO-Fidelity and ExpO-1D-Fidelity on model accuracy and interpretability are in Table 3. ExpO-Fidelity frequently improved the interpretability metrics by over 50,, with the smallest improvements being around 25%. Further, it lowered the prediction error on the 'communities', 'day', and 'MSD' datasets, which lets us conclude that it has a small positive effect on accuracy as well. ExpO-1D-Fidelity generally had similar effect on the interpretability metrics.
Feature
x
Unveg
ExpO)
CRIM
2.5
-0.00
-0.03
INDUS
-0.01
NOX
0.9
-0.23
-0.18
RM
1.4
0.22
0.2
AGE
1.0
-0.08
0.02
DIS
1.6
0.24
TAX
1.5
-0.27
-0.11
PTRATIO
0.8
-0.11
-0.14
B LSTAT
0.4
0.12
0.1
-0.34
-0.53
We ran experiments on the 'MSD' dataset* to understand the scalability of ExpO to larger tasks. On this dataset, evaluating the interpretability metrics using MAPLE was fairly slow, and hence we only evaluate them using LIME on the first 1000 testing points. Both ExpO-Fidelity and ExpO-1D-Fidelity improved LIME's interpretability metrics by at least 50% and both improved the model accuracy.
Table 2: LIME's ex-
planation coefficients for unregularized and ExpO- regularized models.
7
7
Medical classification experiments. The 'support2' dataset used for in- n--hospital mortality prediction. Because the output layer of our models is the softmax over logits for two classes, we run each explanation system on each of the individual logits. Table presents the results. We observe that ExpO-Fidelity improved the interpretability metrics by 50% or more. ExpO-1D-Fidelliy slightly decreased accuracy and improved the interpretability metrics by at least 25%.
2
A qualitative example on the UCI 'housing' dataset. While we have demonstrated quantitatively that black-box explainers pro- duce better explanations for ExpO-regularized models, here we qualitatively analyze the changes in the explanations. More exam- ples, that show similar effects, are available in the Appendix A.4.
4
Figure 3: Original im- ages (left) and saliency maps of an unregularized (middle) and regularized (right) models.
Cousider the UCI hoousing dataset, regression problem to predict housing prices in the Boston area. After sampling a random point I, use LIME to generate local linear explanation at this point for a model trained without regularization ("unregularized explanation") and for model trained with ExpO-1D-Fidere The unregularized and regularized explanations are shown in Table 2.
("regularized explanation").
Quantitatively, using ExpO-1D-Fideee decreased the LIME-NF metric from 1.15 to 0.02; i.c. ExpO produced model that is much more accurately modeled by the explanation around x. Note that the regularized explanation has fewer significant coeffieient (those with absolute value greater than 0.1), and hence it is simpler as the effect is attributed to fewer features.
'The task is to predict release year of song from a set of acoustic features, treated as a regression problem as in Bloniarz et al. (2016) "http://biose" vanderbilt. hath//iikk/Maii//ugpotttee.
7

---

Under review as a conference paper at ICLR 2020
Table 3: Uregularized model vs, the same model trained with ExpO-Fidelity or ExpO-1D-Fidelity on the UCI regression datasets. Results are shown across 20 trials (with the standard error in parenthesis). Statistically significant improvement (p = 0.05) due to Fidelity is denoted in bold and due to 1D-Fidelity is underlined.
Metrio
Regularizer
autonipss
communiiie
day' (10 "]
bousiig
musie
wirnngualliyy.ed
MSD
Nows
MSR
Fidelty
0.55 (040)
0.08 (CON))
0.002 (0C00C)
0.15 (0405
6.6 (0.09)
0.54 (0.00
0.5ST (0.0182)
10 Fideity
0.13 (002)
0.53 (0.04)
5,900 (5 s0u)
0.15 (007))
0.74 (0.77)
0.60 (D.
0.548 (100)
Neve
0.000 (6011)
6.100 (1015))
0.11 (1000))
0.100 (1100))
000t0 (0.1100)
1.116 (UU01))
LIME-PP
Fidelly
298o (8-017)
0,0025
(0.00770))
Fideity
0.22 (0.007)
(0.020)
(0000
(0,001)
None
[UML (0.012)
6,110 (000:2)
1:20 (0.30)
ILuao (0000)
1.12 (1000))
0,0590 (u.t2M)
0.117 (0ULSS))
LIME-F
Fidelly
0.011 (0.003)
0,079
(8.002)
0,00 (0.07)
eUST (0:018)
0.000 (0 11
0.0005
(0-00055
0.029 (0.007)
(0/007)
(0-30)
1a9t (3017)
0.0000 (0 (1l000)
None
0,001 (0.000)
6,622 (0:00))
01100 (0:0011
02007 (0:0012)
cot 70008)
6,00300 10 (100iN
Groes (a (0000.00)
LIME-S
Fidelly
0.0001 (0.0003)
n.nus (0.0033
o.une (0:004)
0.0022
(0-000))
0.00ZE
(000006)
0. 0.00007
(000000)
0.000T1 (0.00034)
ID-Fidelty
pome (0.2045)
c.01s m0000
pi0005 10/0001
Ci (20000))
None
(0.00s))
L0000
MAPLE-PF
Fidelty
0.0014 (0.0006)
0.13 (ouo))
0.0002
(0.0000)
0.02N (0.013)
2:35 00001
0.0027 (0.0010)
ID Fidelty
0.c0no (0 (0,0cas)
nimg (0.00)
0.700m (0:0000))
Iu27 (0015)
6.77 (.1s)
0.0006 i (1
Note
(0.00o7)
(0.04)
1:200 (0.2200)
JuaIs (0:044)
(0x0))
0,0:20 (LUBTY)
MAPLE-NF
(0.014)
10-Fideity
0.0051 (0.00-0)
0.16 (0/66)
0.900 (0:3500)
0:02
6:30 (0.05)
0.9017 (0.000))
Noue
0,0090 (0,0000)
1.2 (0.21
Duc00 (00008)
0.8 10.14)
0L08: 10.05
BJo0es (0.000)) 0,0009 (8.0004)
MAPLE.S
Fidelty
0.0017 (0.0005]
0.8 (0:4)
00004 (0.0004)
0.10 (0.00)
0.05 (0.02)
ID-Fidelty
(0.0001)
(0:9)
'The relationship between imputs and targets the 'day' dataset is very close linear and hence all
orders of magnitude smaller than across other datasets.
Table 4: Uregularized model vs. the same model trained with ExpO-Fidelity or ExpO-1D (-pO-1D-Fideliiy Fidellty on the 'support2' binary classification dataset. Each explanation metric was computed for both the positive and the negative class logits. Results are shown across 10 trials (with the standard error in parenthesis). Improvement due to Fidelity and 1D-Fidelity over unregularized model is statistically significant (p= 0.05) for all of the metrics.
Outpot
Regolarizer
LIME-PP
LIME-NF
LIME-S
MAPLE-PF
MAPLE-NF
MAPLE-S
None
0.177 (0.065)
0,182 (0.065)
0,0255 (0.0084)
0,025 (0,008)
0,055 (0.0L0)
0.5L (0.06)
Powitive
Fidelly
0.050 (0.008)
0.051 (0.008)
0.0047 (0.0008)
0.015 (0.004)
0.018 (0.005)
0.13 (0.06)
1D-Fidelity
0,082 (0025)
0J085 (0.025)
3,0066 (0.1022)
(0,00)
0,025 (0,05)
0.16 (000)
Noue
1.15s (oL07)
11300 (0.00))
1.0290 (1.0122))
(0,030)
0,000 (0.014)
1,37 (0.38)
Negative
Fidally
0.050 (0.008)
(0.008)
0.0047 (0.0008)
0.013 (0.004)
0.018
0.13
1D-Fidaity
0.081 (0/226)
0.082 (0.02))
(0.0021)
(0.00)
0.021 (0.007)
0.16 (0.06
Accuracy (%): None: 83.0 = 0.3, Fidelity: 83.4 + 0.4, 1D-Fidelity: 82.0 = 0.3.
4.2 STABILITY REGULARIZATION
For this experiment, we compared normally trained convolutional neural network MNIST to one trained using ExpO-Stability. Then, we evaluated the quality of saliency map explanatious for these models. Both N₂ and Ng where defined as Unif(x 0.05, x + 0.05). Both the normally trained model and model traiend with ExpO-Stabilityy achieved the same accuracy of 99%. This demonstrates one of the practical differences between SENN and ExpO: SENN places strict structural constraints on the network and subsequently lowers the testing accuracy to roughly 97%. Quantitatively, training the model with -tabiittyyl decreased the stability metric from 6.94 to 0.0008. Qualitatively, training the model with ExpO-Stability made the resulting saliency maps look much better by focusing them on the presence or absence of certain pen strokes (Figure 3).
5 CONCLUSION
In this work, we have introduced the novel idea of directly regularizing arbitrary models to be more interpretable with respect a general metric. We contrasted our regularizers to classical approaches for function approximation and smoothing and provided a generalization bound for them. We demonstrated, across a variety of problem settings and explainers, that our regularizers slightly improve model accuracy and improve the interpretability metrics by somewhere from 25% to orders of magnitude. We believe that potential future work may focus on three areas: (1) exploring alternative neighborhood functions, Nreg, that match those used by other black-box explanation systems, (2) exploring how to regularize for non-local interpretability metrics, and finally (3) exploring the interaction between the regularizers and the optimization process, e.g., progressively changing the importance of the regularization during training or using the regularization as an additional training step.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Zachary C Lipton. The mythos of model interpretability. arXiv preprinl Xiv: X 66d600446999 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144. ACM, 2016.
Fulton Wang and Cynthia Rudin. Falling rule lists. In Artificial Intelligence and Statistics, pages 1013-1022, 2015.
Rich Caruana et al. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1721-1730. ACM, 2015.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. arXi
preprint 160: 04155, 2016.
Scout M Lundberg and Su-In Lee. unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, pages (665-477,, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE International Conference Computer Vision, pages 618-626, 2017.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (teav). In International Conference on Machine Learning, pages 2673- 2682, 2018.
Maruan Al-Shedivat, Avinava Dubey, and Eric P Xing. Contextual explanation networks. arXiv preprint 10b.:1705.10000 2017.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. Model agnostic supervised local explanations. In Advances in Neural Information Processing Systems, pages 2516-2528 2018.
David Alvarez-Melis and Tommi Jaakkola. Towards robust. interpretability with self- explaining neural networks. In Advances in Neural Information Processing Systems, pages 7785 7794, 2018a.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXie preprint arXie:1312.608; 2013.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: figh-prrecsion model- agnostic explanations. AAAI, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 3319 3328. JMLR. org, 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. Springer, 2014.
In European conference computer vision, pages
A vanti Shrikumar, Peylon Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint orXiv:1605. 1Tft::6600001133 01713, 2016.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. Xix preprint arvi 706 03825, 2017.
9

---

Under review as a conference paper at ICLR 2020
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neurol Information Processing Systems, pages 1525-9536, 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. arXic preprint wxxvv:1100 110.17770,0577 2017.
David Alvarcz-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXi preprint Xrx: Xiw:1806.08049, 2018b.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi- Velez. Right for the right reasons: Training differentiable models by constraining their explanations. ar preprint vrXix:1703.1 03717, 2017.
Guang-Hu He Lee, Wengong Jin, David Aivarez-Melis, and Tommi Jaakkola. Functional transparency for structured data: a came-theorettin approach. In International Conference on Machine Learning, pages 3723 3733, 2019.
Guang-He Lee, David Alvarez-Melis, and Tommi S Jaakkola. Game-theoretic interpretability for temporal modeling. arxiv preprint 18070000.001 2018.
Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving the robustness of deep neural networks via stability training. In Proceedings of the ieee conference compuler vision and paltern recognilion, pages 4480 4480-4488, 2016.
Dua Dheeru and Ef Karra Taniskidou. UCI machine learning repository, 2017. URL http://archive. ics. udi/edu/m.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com hon/eadb/mmiise 1998.
Adam Bloniarz, Ameet Talwalkar, Bin Yu, and Christopher Wu. Supervised neighborhoods for distributed nonparametric regression. In Artificial Intelligence and Statistics, pages 1450 1459,,149,,11 2016.
10

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 COMPARISON OF EXPO TO BASELINE METHODS
Although there are related methods to ExpO, none are necessarily appropriate to act as bascline for comparison:
SENN (Alvarez-Melis and Jaakkola, 2018a) requires that the model has a specific structure. ExpO makes no assumptions about the model's structure.
RTFR (Ross al., 2017) requires that we have the domain knowledge to specify that "Featurr 'i' should not be relevant to the prediction for point 'x' ExpO does not require this or any other domain knowledge.
Finally, the regularizers defined by (Lee et al., 2019) are designed for structured data types. ExpO is designed to work with semantic features or images, so there isn't a common experiment that we could run.
As result, we consider two standard regularization techniques: l2 and 4 regularization. These regularizers may make the network smoother or simpler (due to sparser weights), which may make it more amenable local explanation. The results of this experiment are in Table 5; notice that neither of those regularizers had significant effect on the interpretability metrics compared to ExpO.
Metric
Regularizcs
ausompes
communities
day
horasing
mussc
wineeuuaity.rre
MSE
None
0.14
0.49
0.001
0.14
0.72
0.65
0.13
0.00012
0.15
1.7605
0.15
MAPLE
None
0.016
0.16
0.001
0.057
0.17
0.013
0.015
0.17
32e-05
0.05
0.17
0.02
0.014
0.015
Nono
8.0022
0.066
0.013
L2
0.016
0.52
0.3e-05
0.058
0.17
0.021
0.016
0.32
2.6e-05
0.065
0.18
0.016
MAPLE-Subility
0.015
0.011
1.3
3.26-06
0.17
0.065
0.0058
L1
0.013
1.2
Se-07
0.21
0.072
0.001
LIME-PP
Nove
0.04
0.1
0.0012
0.11
0.033
0.037
0.00014
0.000
0.047
0.035
0.12
0.00017
0.13
0.1
0.034
NE
None
0.641
0.11
0.0012
0.14
0.11
0.033
0.037
0.12
0.00015
0.12
0.098
0.947
0.00018
0.13
0.1
0.034
LIME-Stahility
Nose
0
0.022
0.00015
0.0047
0.011
0.0013
L2
0.00097
0.032
1.7e-05
0,001
0.011
0.0021
0.0012
36405
0.0348
0.011
0.0016
Table 5: Using la or 4 regularization has very little impact impact on the interpretability of the learned model.
A.2 DETAILS ON THE GENERALIZATION OF LOCAL LINEAR EXPLANATIONS
Here, we provide a derivation of the bound (5) on the explanation fidelity. First, we assume that local linear explanations, 3r, are obtained by solving the ordinary least squares regression problem (as given in Algorithum 1):
B = [X'X']]'' X'f(X').
(6)
where each column of X' denotes a sample from the neighborhood N₂ and f(X') is colunn-veclor of the corresponding function values. The expected fidelity of the explanation Oz can be computed analytically:
rff,z) EN. [f(x']] EN, [f(x'>=7 EN, [|=''']]] EN, If(r')=1
(7)
where expectation E, is taken with respect to x' over the neighborhood N₂. Note the equality in (7) is the expected value of the squared residual between (xx and the
11

---

Under review as a conference paper at ICLR 2020
Evall
0.25 Heg
0.10
Figure 4: A comparison showing the effects of the a parameter of N₂ and 2 on the UC1 Housing dataset. The LIME-NF metric grows slowly with a for N₂ as expected. Despite being very large, using = 0.5 for No is generally best for the LIME-NF metric and possibly for accuracy.
optimal local linear explanation, which is upper-bounded by the variance of the model in the correspouding neighborhood:
S r(f,=) s EN, [f(z\)] EN. [f(z')] VarN. [f(x')]
(8)
For instance, if fz) is L-Lipschitz and the neighborhood N₂ is defined a uniform distribution within ø-ball centered x, then the variance of f(x) within the neighborhood can be further bounded by 4L, hence r(J,a) < 4
For the explanations to generalize, we would like make sure that the gap between the average fidelity the training set and the expected fidelity is small with high probability. More formally, the following incquality should hold:
PP E [r([[z)]]]
- Σ rr) > < da(e)
1
The following is restatement of Proposition 1 with a short proof.
Proposition Let the neighborhood sampling function N₂ be characterized by some param- eter a (c.g., the effective radius of neighhorhood) and the variance of the trained model f(x) across all such neighborhoods be bounded by some constant C(a) > 0. Then, the following bound holds with at least 1 probobilily:
n
C²
[r((f S Σ (f.xz)) +
2n
Proof. By assumption, the variance of the model f(x) is bounded in each local neighborhood specified by Nz. Then (7) implies that each residual is bounded as 0 s r(f,z) s C(a). Applying Hoeffding's inequality, we get:
E[r(J.)))
E
[ exp
{
-2ne (2((a)
Inverting the inequality gives us the bound.
A.3 CHOOSING FOR N AND Arreg
In Figure 4, we see that the choice of for N₂ not critical (the value of LIME-NF only increased slightly with o) and that this choice of for Nr produced slightly more accurate and interpretable models.
12

---

Under review as a conference paper at ICLR 2020
A.4 MORE EXAMPLES OF EXPO's EFFECTS
Here, we demonstrate the effects ExpO-Fidelity on more examples from the UCI 'housing' dataset (Table 6). Observe that the same general trends hold true:
The regularized explanation more accurately reflects the model (LIME-NF metric)
The regularized explanation generally considers fewer features to be relevant. Again, a feature 'significant" if its absolute values is 0.1 or greater.
Neither model appears to be heavily influenced by CRIM or INDUS. The regularized model generally relies more on LSTAT and less on DIS, RAD, and TAX to make ils predictions.
Esample Numberr
Value
CRIM
INDUS
AC1E:
TEAXX
LSTAT
LIME-NF
0.36
Unnegalaciee Explanation
0.11
-0.14
-0:29
0.27
-0107
-0.24
0,008E
Regalarised Explenation
0.0
0.01
-0.14
0.25
0.03
-0:16
0.15
-0.1
-0.12
-0.01
-0.47
0.0033
Fosture
-0,82
-0.77
1.70
-0.42
Unmgnlariie Explanction
601
0.00
-0.15
0.52
-0.1
-0.20
0.34
-0.27
-0.12
-0.21
Regalartadd Explination
0.0
0.0
0.15
0.25
0.01
0.15
0.15
0.12
0..3
0.01
0.47
0.00076
Festure
-0.55
-0.05
-0.52
-0:13
-0.65
0.1
Unvogularied Explanction
8.01
0.00
0.16
0.29
0.08
0:31
0.27
0.27
0..11
0.1
0.15
0.076
Regalartadd Explamation
0.03
0.01
0.13
0.19
0.0
0.15
0.14
0.11
0.12
0.0
0.43
0.088
Poature
0.36
0.34
0.25
.0.0
0.77
0.56
0.51
00.12
1.14
0.44
0.14
Unngalarised Explonatio
0.22
0.06
-0.16
0.29
-0.1
-0.34
0.31
-0.21
-0.00
0.12
-0.27
0.30
Beg alaciood Explematiion
-0.02
0.01
-0.15
0.21
0.02
-0.16
0.17
-0.11
-0.32
-0.0
-0.47
0.013
Fost
-0.37
-1.14
-0.58
0.45
-0.28
-0.21
-0.86
-0.76
-0.38
0.00
-0.92
Unmgnlarized Explanation
0.02
0.08
-0.17
0.55
-0.11
-0.30
0.29
-0.27
-0.08
0.1
00.28
0.060
Regalarised Explenatton
-0.0
-0.0
-0.14
0.26
0.0
40.
16
0.15
-0.11
-0.15
0.01
-0.07
0.0021
Table 6: More examples of how regularizing model using ExpO-Fidelity affects the explauations. For each example we show, the feature values of the point being explained, the coefficients of the unregularized explanation, and the coefficients of the regularized explanation. Note that the bias terms have been excluded from the explanations. We also report the LIME-NF metric of each explanation.
The same comparison for examples from the UCT 'winequality-red" are in Table 7. We can that the regularized model depends more "volatile acidity" and less on "sulphates" while usually agreeing about the effect of "alcohol". Further, it is better explained by those explanations than the unregularized model.
Ecuspa Narke Vodas Hoea
Bodd wadie
vearde adday
dits 5435
coddeal man::
DAAl PAESY Sacad
plf
rsplato
akoded
4126
-1.31
-1Uf2
-3.28
5.12
-2.25
0.41
6:28
Regiliiee beli Explovaiins
0.30
8.53
03:31
0.81
3.64
0/6
6.15
8.59
600
0.17
0.2
6il CC
Fratio
1.32
-31
0/6
-335
0:20
8115
01:1
3.14
606
6:9
0.18
6.50
nuca
Reg Eeplo
-01
01.33
1885
-318
012
464
4333
4u8
11:6
8.29
Cheeps exeried Esplasatio
0.16
0.32
0.5
313
606
681
3.13
0.36
6:51
DGI
4030
4033
838
Eoplasation
4313
0.39
600
4.16
6:51
Bay Expiiaaiion
435
0.30
3.65
6.1
4.88
0.66
Table The same setup, but showing examples for the UC1 'winequality-red" dataset
13


Reviews:
[{"review_id": "SkgZLZH2Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a new type of regularizer to improve explainability in neural networks. The proposed regularizer is largely based on two metrics, namely fidelity and stability. It optimizes for fidelity and stability as a regularization objective in a differentiable manner.\n\nI would recommend for accept, as the paper shows in its experiments that the explainability of neural networks can be improved with the two proposed regularizer, which outperforms simple baselines of l1 and l2 regularizers. The paper's method is generic, and can be applied to almost all machine learning models with gradient-based optimization, making it helpful to building explainable machine learning systems.\n\nHowever, I would also like to note that the results in this paper are somewhat unsurprising. The ExpO-Fidelity and ExpO-Stability regularizers can be seen as (almost) directly optimizing the fidelity and stability metric for explainability, so one would naturally expect that models trained with these regularizers will do better on the two metrics above. In addition, I do not see much value in the derivations of Section 3.2. The conclusion that \"explainable models with smaller local variances ... are likely to have explanations of higher fidelity\" is unsurprising and almost a straightforward claim.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lHgrse9S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the problem of training black box models for improved interpretability, and  proposes to penalize black-box models at training time using two regularizers that correspond to fidelity and stability explanation metrics. As computing the regularizers exactly is computationally intensive they propose two approximating algorithm. In addition, as the one for fidelity is still prohibitive, a randomized variant is proposed. Connections are established between the regularizers and the model's Lipschitz constant or total variation. A generalization bound is presented for local linear explanations. The proposed approach is evaluated on a variety of datasets. \n\nThe paper deals with an important problem and the exposition is clear.  While regularizing deep learning models is a pertinent  direction, I feel the paper makes a couple of overstatements, and overall I am not fully convinced by the approach and empirical evaluation, as outlined below.\n\n- The paper states \"recent approaches that claim to overcome this apparent trade-off between prediction accuracy and explanation quality are in fact by-design proposals that impose certain constants on the underlying model families they consider\" and that they are addressing this shortcoming. But in fact, the proposed regularizations do exactly the same: they impose certain constraints. Indeed the regularizers encourages models with lower Lipschitz constant or with small total variation across neighborhoods.\n\n- I also find that it is unsurprising that regularizing via fidelity or stability will lead to models with better fidelity/stability so it's an artificial way to yield \"improved interpretability\" and this is more of an issue because fidelity and stability are kind of proxy metrics to evaluate interpretability.\n\n- It would be important to investigate further the difference between regularization and explanation neighborhoods. This might not be a bad thing which in fact help with generalization. \n\n- Proposition 1 supports algorithm 2, but it is not a given at all that Algorithm 1 will have smaller local variances across neightborhoods and hence might generalize well.  It would be important to proceed with an empirical study of the local variance across neighborhoods for Algorithm 1.\n \n- Computational complexity remains an issue as ExpO-1D-fidelity is performing much worse.\n\n- Comparison against alternative approaches beyond SENN are lacking (e.g. Lee et al 2019, Wang and Rudin,2015 etc).\n \n Overall I feel that more work is needed to convincingly demonstrate the importante of the proposed approach.\n ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eqkesCFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an approach for local post-hoc explanation with introduction of a new regularization that helps regulate the \"fidelity\" and \"stability\" of the output explanation (in the style of LIME).  The fidelity regularizer is essentially the squared error of the explainer as compared to the given model in the local neighborhood, whereas the stability regularizer measures the total squared differences between the explanation outcomes between the sample in question and other samples in the local neighborhood. \nIn the experimental evaluation section, the authors evaluates the performance fo the proposed regularizers, used as part of both LIME and MAPLE, against three interpretability metrics: point fidelity, neighborhood fidelity and stability. The results verify that indeed the use of the regularizers improve the performance of both LIME and MAPLE over the unregularized versions, with respect to the corresponding metrics. This is in a way \"expected\", since the regularizer used in the method and that in the metric are closely related, and is an unsatisfying aspect of the work. \nUsing image data, they also demonstrate that qualitatively the use of stability regularizer seems to significantly improve the saliency of the output explanation. \nThe paper is well written, the proposal is reasonable, but the contribution is modest and experimental evaluation is not entirely convincing. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates a promising direction on the important topic of interpretability; the reviewers find a variety of issues with the work, and I urge the authors to refine and extend their investigations.
Decision: Reject