RbX: Region-based explanations of prediction models
Authors: Ismael Lemhadri
Harrison H Li
Trevor Hastie
Venue: Submitted to ICLR 2023 (2023)

Abstract: We introduce region-based explanations (RbX), a novel, model-agnostic method to generate local explanations of scalar outputs from a black-box prediction model using only query access. RbX is based on a greedy algorithm for building a convex polytope that approximates a region of feature space where model predictions are close to the prediction at some target point. This region is fully specified by the user on the scale of the predictions, rather than on the scale of the features. The geometry of this polytope - specifically the change in each coordinate necessary to escape the polytope - quantifies the local sensitivity of the predictions to each of the features. These “escape distances” can then be standardized to rank the features by local importance. RbX is guaranteed to satisfy a “sparsity” axiom, which requires that features which do not enter into the prediction model are assigned zero importance. At the same time, real data examples and synthetic experiments show how RbX can more readily detect all locally relevant features than existing methods.

Under review as a conference paper at ICLR 2023
RBX: REGION-BASED EXPLANATIONS OF PREDICTION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce region-based explanations (RbX), novel, model-agnostic method that uses only query access to quantify the sensitivity of the scalar predictions from a black-box model to local feature perturbations. RbX is based algorithm for building a convex polytope that approximates a region of feature space where predictions geometry of the learned polytope - specifically the change in each coordinate necessary to escape the polytope then explains the local importance of each feature in changing the model predictions. In particular, these "escape distances" can be standardized and ordered rank the features by local importance. The RbX method is informed by goal of detecting as many relevant features as possible in locally sparse prediction models, without including any features that do enter in to the model. We provide a real data example and synthetic experiments to illustrate the encouraging performance of RbX in this respect.
greedy
close the prediction at some target point Zo. The
INTRODUCTION
Suppose we have a prediction model f(xx to estimate a scalar outcome y given a set of features E Rd. We assume we do not have any knowledge about the functional form of f. Rather, we consider f as black-box function to which we only have query access, meaning we may compute the value of f(x) for any desired input x.
After making a prediction at a target point Zo, we seek to quantify the local importance of each feature on the prediction. Perturbations in certain features will be more influential than others in changing predictions from f near Tn- We would like systematic way of identifying these features.
We distinguish our problem, which we call local prediction importance, from the questions of feature selection and feature importance. Feature selection methods, such the LASSO for linear models [Tibshirani] 1996) and modern extensions like LassoNet for black-box models (Lemhadri el al., 202T). aim select small subset of features to generate predictive model with greater accuracy and/or interpretability.
our setting, the prediction model is fixed, and we seek only faithfully explain the predictions of that model, without regard to the unknowable data- generating process that created the features and response. Feature importance methods include popular permutation- approaches introduced by Breiman (001) for random forests, which were extended to generic black-box models by Fisher et al. (2019) and to a local method by Casalicchio et al. (2018). These approaches also fix the prediction function f. but provide importance measures based on changes in the predictive performance of f as various features are ignored, permuted, or otherwise perturbed. That is, they consider the extent to which changing features impacts the ability of ff to approximate some ground truth function f. By contrast, the term prediction importance emphasizes the singular role of the numerical outputs of f. setting aside how well these predictions approximate reality.
The distinction between local prediction importance and local feature importance is not always made in the literature. However, it is relevant for a user who only cares about understanding the output of given black-box model, and does not want prediction explanations conflated with the underlying signal the model is trying to approximate.
Our proposed approach to local prediction importance is via region-based explanations (RbX). The method is "model-agnostic," " meaning it does not use any knowledge about the structure of f. relying

---

Under review as a conference paper at ICLR 2023
only on query access. We defer detailed description of the algorithm to Section 3.1 but the main idea is to construct polytope that approximates the region in feature space with prediction values "close" to the prediction a target point Zo (Section[2.) We then argue that distances from 2n the boundaries of this polytope in directions parallel to the coordinate axes inform the local sensitivities of f to each feature in desirable ways according to our evaluation properties defined in Section
1.1 PREVIOUS WORK
Existing approaches to local prediction importance be broadly divided into two categories: surrogate methods and gradient- methods. Surrogate methods locally approximate f by fitting a simpler prediction model that treats the predictions of f in region near 20 as the response. The weights assigned to each feature in this model are then used for local importance. For instance, LIME (Ribeiro et al., 2016) draws feature instances from a density centered at the target point zg and uses linear surrogate. Lundberg & Lee (2017) propose Kernel SHAP (hereafter just SHAP), which they showed is an algorithmic approximation to fitting an additive surrogate model with weights corresponding to Shapley values.
Gradient-basse methods consider infinitesimal regions on the decision surface and use the resulting first-order approximation to derive local feature importance. For example, [Baehrens al. (2010) provide local prediction importances based on the absolute value of the components of the gradient vector V (zo); their approach for estimating this gradient is by fitting global surrogate model using Parzen windows. Integrated gradients (Sundararajan et al.) 2017) considers the line integral of the components of the gradient of f over a straight line path in feature space from a baseline point x to
To. Other gradient methods are not model-agnostic. For instance, DeepLIFT (Shrikumar et al. 2017)
relies on backpropagation to estimate gradients in neural networks.
WHY REGION-BASE EXPLANATIONS?
In general, the "ground truth" local prediction explanation for given model ff and target point zo is ill-defined. While the explanations from procedures like SHAP and IG are derived based on some particular set of axioms, evidently there is not consensus to which axioms are more "desirable". Thus, we choose to develop RbX based on two less restrictive but likely less controversial properties which we call sparsity and detection power:
Property 1. (Sparsity) A feature not involved in the prediction model f is assigned no importance. Property 2. (Detection power) The locally relevant features for f are assigned highest importance.
Sparsity requires that any feature that cannot change the predictions from f is assigned no importance. Of course, sparsity is not sufficient for good local prediction importance method, though we view it as necessary. Conversely, method that always assigns zero importance every feature trivially satisfies sparsity, but fails to specify any potentially important features, hence the need for detection power. designing RbX, we seek to maximize detection power while preserving sparsity.
Ideas similar to Property 1 and are not new. Indeed they are used by the authors of LIME and L2X (Chen et al.) 2018). a method that computes local feature scores by maximizing a variational relaxation of the mutual information between y and the features æ encoded by classifier f. to evaluate their methods. For instance. the experiments Ribiiio et al. (2016) show that LIME does a better job than some baseline methods in finding the features used in sparse logistic regression models and decision trees.
SHAP, IG, and other gradient methods satisfy sparsity axiomatically, yet LIME and L2X do not. While there is some subjectivity in the definition of locally relevant, we believe a reasonable sufficient condition for local relevance of feature j is for the j-th component of V f (200), the gradient of the prediction at To. to be nonzero. Then in the case that f is an additive regression model, Property corresponds to assigning zero importance to all features with zero coefficients, while Property means assigning nonzero importance to all other features. A simple gradient- method using finite differences would then always perfectly satisfy both properties, the set of features with nonzero gradients would always be precisely the relevant features. By contrast, LIME only does this 90%-92% of the time in the experiments from Ribeiro et al. (016)).
2

---

Under review as a conference paper at ICLR 2023
What remains is improve detection power in nonlinear prediction models without sacrificing sparsity. In such models, a feature might be locally relevant near the target point 20, even Vf(za) = O. Sundararajan et al. 2017) motivate IG this way, noting that gradient methods fail when f has zero gradient with respect to particular feature, but still varies in that direction within non-infinitesimal neighborhood that considered locally relevant.
IG addresses this 7ero-gradient issue by accumulating gradients along the entire line segment between some baseline feature values x and the target point Zig- However, this still only detects features with respect to which f varies infinitesimally somewhere along this line segment. There are a lot of additional areas of the feature space near To where f could depend on a given feature. Our approach, RbX, examines the sensitivity of f in a large number of directions, while ensuring sparsity from a finite differences gradient method. It does so in a non -on-infinitesiall neighborhood of ze. adapting the search space to cover much larger portion of the region the user specifies as "close" to No. which we describe next.
2.1 CLOSE REGIONS
RbX asks the user to specify. for any given target point 2g, the values of predictions from f that are "close" to f(x0). the prediction at the target point. There are often natural choices. For example, if f is a class probability from binary classifier, the close region might contain all prediction values on the same side of the decision boundary. If f predicts numeric medical outcome, and 2o corresponds to a healthy patient, then the region might be the accepted range of healthy outcomes. For ease of exposition we assume the close region is an interval I of the form [f(=) es, f(xo) + eHl. The interval I then depends on the user's choice of two nonnegative parameters e (cL, < which can vary with zg. For instance, f predicts body mass index, for which the healthy range is 18-25, we might take eg = 4 and ε when the target patient satisfies f(zo) = 22, while for f (xx)) = 26 and ии co. The close region & {ar E IRd I} will hereafter be referred to
we'd d take e,, as the "e-close" region to emphasize the dependence on e. Points outside & are said to be "c-far" and the boundary of & is referred as the "c-boundary."
We view the specification closeness on the scale of the outcome as an advantage relative to methods like IG and SHAP which require specification of a "baseline" feature value æ. The sum of all feature explanation scores from these methods any target Zg is constrained to equal f(xo) f(x). While Sundararaaan et al. (201)) note that natural baselines exist in settings like image classification and sentiment analysis, for a general prediction or classification setting there may not be canonical choice, and the feature attributions will be sensitive to the choice of baseline. SHAP's reliance baseline is eliminated by the cohort Shapley method of Mase et al. 2019). though cohort Shapley still retains an additivity constraint that all feature attributions must add up to (zo) - f.. where is the mean prediction set of n observations. If ʃ is not meaningful value then the individual feature scores do not have direct interpretation.
3 A POLYTOPE APPROXIMATION ALGORITHM
We now ready to describe the RbX algorithm, which approximates the e-close region E by polytope P. It does so by approximating the e-boundary at various points by affine hyperplanes. Definition 1. A polytope P C IRd is any finite intersection of affine halfspaces, i.e.
P OisksK H
where H, {a E Rd Tuu S c+} is defined by its normal vector uy Rd and intercept E IR
The use of a polytope approximation, as opposed to a smooth shape like an ellipsoid, enables sparsity.
3.1 A POLYTOPE APPROXIMATION ALGORITHM
RbX is a greedy procedure that constructs the polytope approximation P the e-close region E one halfspace time (Algorithm [] The algorithm requires set of context samples X' {m:isis
3

---

Under review as a conference paper at ICLR 2023
that form the basis of the sampling procedure. These should be dense collection of representative feature values, for instance from a possibly unlabeled training or validation set. To make the polytope P scale equivariant, all the features scaled to have standard deviation 1 across the context samples. Only c-far context points are used by the remainder of the procedure.
Algorithm 1 The RbX Algorithm
Input: target zo R closeness thresholds e = O, prediction model f with query access, context samples {a; hgigs, maximum number of splits K.
2: Compute 8, the vector of standard deviations of the features across the context samples {xijisisn.
3: Standardize x; + diag(s) for 0,1,,,1........ I. n.
4: for E [1 nn] if æ, is e-far do
5: Shrink 24 onto the e-decision boundary using line search: E; t Line-Search(zz, 20)
6: end for
7: Initialize
{
8: Initialize the set of support vectors S <-
9: Initialize k
10: while R + 0 and k: < K do Find a(()) <- arg min || xoll
11:
FER
Estimate gradient of f at (k): gk + Estimate-Grad f,
12:
Compute halfspace: H₁ + {ar E Rd x" ga S (z( (F0g))gg))) gx}
13:
14:
Update region: R + R n int Ha
15:
Update support vectors: S + SU (
Increment h: l k
17:
end while
18:
return {Hx Hisksn the collection of halfspaces defining the polytope P
19: Notation: SSarch is defined in Algorithm Estimate-Grad is given in Algorithm
The first step in the RbX algorithm is to shrink each (standardized and e-far) context point x; along the line segment in TRd between Ty and zo a point the e-boundary. This be done quickly to exponential accuracy via standard bisection-based line search (see Algorithm in the Appendix for an example), noting that zo always e-close.
Next, RbX finds z(1, the context point whose shrunken counterpart z(() is closest To in Euclidean distance. The first halfspace H1 of the polytope P is chosen to pass through ({(1) and have normal vector equal to estimate of the gradient of f. computed using finite differences (Algorithm 2 (1()), This is motivated by the fact that in the case that f is differentiable, Hi is first-order approximation of a level set of f. Finally, all shrunken context points outside the interior of H are discarded, and the process is iterated with the remaining shrunken context points until either K halfspaces have been learned, or there are no more shrunken context points remaining.
Note that (*k), the shrunken context point chosen the k-th iteration of the algorithm, lies outside the interior of the k-th halfspace H& by construction. Thus the number of remaining shrunken context points decreases by least one after each iteration, and so the algorithm terminates at most iterations. In practice, far fewer iterations are often needed. To further reduce computation, one can impose early stopping by specifying a maximum number of halfspaces K
be learned.
The "greedy" nature of the RbX algorithm refers to how it chooses the closest shrunken context points first. This helps enforce better approximation of the parts of the e-decision boundary closer to Zo. In general, any given shrunken context point may not be the closest point to that lies on the c-boundary and also along the ray from To to the original context point. Thus, the algorithm may miss parts of the e-boundary extremely close to f. But such an issue is mitigated by having a sufficiently dense set of context samples covering the full range of plausible feature values. Then
the greediness ensures the closer parts of the decision boundary - among those corresponding to plausible feature values, where we most interested in the predictions are prioritized.
4

---

Under review as a conference paper at ICLR 2023
3.2 FINITE DIFFERENCES GRADIENT ESTIMATION AND SPARSITY
The gradient of / is estimated at each shrunken context point using finite differences (Algorithm 2 We allow the user average gradient estimates m "jittered" points that are the original points corrupted by a small amount of Gaussian noise. This jittering is designed to smooth the gradient estimate when f has discontinuities on the c-boundary. If has no dependence on the i-th feature, it is clear that the i-th component of V fz) will be O according to Algorithm 2 regardless of the parameters used. Note this would not be the case Algorithm 2 were replaced by a smooth gradient estimator, such as the Parzen windows used by Bachrens al. (2010). Having zero gradient estimates for irrelevant features ensures that our local importance scores satisfy sparsity, as described in Section 4
All results in this paper presented with parameters 0.01, ò 0.1, and 10.
Algorithm 2 Estimate-Grad Algorithm Finite Differences
1: Input: point Rdd, prediction model f, step size 6, jitter radius r, number of jitter samples m
2: for j E [1: m] do
3: Generate 2 Ld)
4: f
( i ,,.... (1) 7: return vf(x) no E== (oL) .... ""..9"" g24)
(3)
5:
9:
6: end for
3.3 TOY EXAMPLE
We briefly illustrate the RbX algorithm in Fig. toy example with d 2, f(z) x1 Z2. To = (0, 0), and e = (0.5, 0.5). 500 context points generated from standard bivariate Gaussian distribution were used.
Each iteration of the algorithm approximates a plane tangent to the e-boundary at the closest shrunken context point. All context points on the side the plane not containing Zy are then discarded. The resulting P at termination (after splits) is diamond-shape region, which is non-convex and extends infinitely along the coordinate axes in both directions.
region that truncates the true c-close
Figure 1: An illustration of the RbX algorithm for the pairwise interaction model f(z) xn Dg with e == (0.5, 0.5) and target point zo = (0, 0), highlighted in the center of each plot. The smaller dots the context points, colored by whether they c-far. Each panel shows the additional halfspace constructed in one iteration of the RbX algorithm. The lines are the halfspaces learned at each step of the algorithm; the larger dots along these lines are the shrunken context points used.
FROM POLYTOPES TO LOCAL PREDICTION IMPORTANCE
Given the polytope P output by the RbX algorithm, we derive local prediction importance scores Sy(P) using "feature escape distances" for each feature {l,...,d].
(T) inf > 01 an + oe, & P); SJ(P) = inf > 0 xo - -ae; $ P):
Sy(P) (P), (P)) sign(S; (P))
5

---

Under review as a conference paper at ICLR 2023
Here Ej is the j-th standard basis vector in R.. Then (y) is the minimum signed distance needed to escape the RbX polytope P by varying only the j-th feature from the target point zu. As argued by Mase et al. 2019), predictions from f at feature values that are implausible should likely not be trusted. Thus, whenever the "escape path" out of the polytope P in the direction of feature passes outside the region 7 where predictions from f are "trustworthy," we set (P) to oo. Appendix A.2 provides some methods and examples to determine how to construct T based on the context points.
Note that the escape distances (P) are reported on the original scales of each feature (before standardization in step 3 of Algorithm D. which enables them to be interpreted individually without reference to the escape distances of the other features. Alternatively, we can report the escape distances (P))) on the standardized scale. Sorting these standardized distances (from smallest to largest) enables us to obtain a ranking of the local importance of the features (from most important to least important).
doesn't i depend on feature z,, then all of the halfspaces defining P will have normal vectors with 0 component in the zj direction. This implies that the corresponding (P) will be co, and thus such features will have no importance, showing our procedure satisfies sparsity.
It is natural to compare the feature escape distances S, (P) with the "simple feature escape distances" (E)), which use the original c-close region & in place of the polytope P. They can be computed via a line search similar Algorithm []] without running RbX. Clearly, feature ranking based on the SE( would also satisfy sparsity. Furthermore, it would have better detection power than a gradient-based method. This is because the (y()) look beyond infinitesimal neighborhood of Jo. instead focusing on a typically larger region defined in terms of the prediction values of f to be meaningful. However, it still cannot capture changes in ʃ near z0 that cannot be detected when only one feature is changed a time from its value at 37p.
Since the polytope P looks in many directions around vo, using the RbX distances Sy provides better detection power. A simple example of this be in Fig. 1 For Tig (0 have Si (E) (2) oo since all points along the coordinate axes are e-far. Yet (P)) ~ (P) ) ~ 1.4, giving the features equal and nonzero importance. Further examples are the next section.
5 DATA EXAMPLE AND SYNTHETIC EXPERIMENTS
We begin by comparing RbX to existing methods for local prediction importance on a credit scoring example, considering sparsity, detection power, and robustness across similar target points. Then consider simulated experiments designed to more systematically evaluate detection power.
5.1 CREDIT SCORING EXAMPLE
The home equity line of credit (HELOC) dataset from the FICO XML Challenge (community. community.fico.. fico. com/s s xml) contains the tiskeerformann This is binary indicator whether they were ever more than 90 days past due in the first two years after account opening. The goal is to interpretably classify each individual as having either "Bad" or "Good" RiskPert erfferrforeece based on 23 quantitative or ordered categorical predictors. We consider the local prediction importance of shallow and sparse decision tree. In Appendix A.3 we consider sparse gradient boosted tree ensemble fit to the same data. After splitting the dataset randomly into 1,751 training observations and 751 test observations, the tree classifier fit to the training observations to have depth 3. The final classifier is visualized in Fig. 2 The scalar prediction output is taken to be the predicted probability of a "Good" RiskPerformance.
of 2,502 credit applicants.
We begin by examining a specific target point, corresponding to the individual labeled 5,238 in the dataset. Selected feature values for that individual (who has prediction value 0.191) are given in Table T along with feature scores from various local importance methods. For LIME and SHAP, compute the scores from the open-source implementations and default hyperparameter settings, except that for LIME we do not discretize the features, which greatly improves its performance. For RbX we assume a decision boundary of {ø fzx = 0.5}. and take E to the set of points on the same side of this decision boundary zp- All training observations used context points.

---

Under review as a conference paper at ICLR 2023
Extemadlaekiidtmma 72.5
0.462
value [1032, (58]
dlass = Bad
hwtpre/weooelloeeeeeoeeeeeceeeee get
samples value - [863, 411] cas5 Bod
volua = [133, 248
class Good
MSSrooClaaetTaaaaCCyy 2530
External Eattmne
oellostfacce seseettt (Dii)
Auwragabdl
0.354 samples 905
samplio 465 varue e (262, 200] dlsso = Uad
samplea 185
samples =
vilue = (807, 206]
valuae = [91, 104) clsss Good
value i (42 144)
cass Bad
class Good
giill 0.456
citll vallae [147, 58) class Bad
diiit
giii
686
178
value [585, 1811
value = (142,
value (115, 144] class Good
valuo (40, 211
valuo (51, diiss -ood
value (7,
value [35, 141]
class = Bad
clsss e Bad
class = Bad
clsss + Bad
casss - Good
Figure A visualization of classification tree fit the HELOC data from the FICO xML Challenge.
Table 1: Feature values and various local prediction importance scores for the sample target point analyzed in the main text, with the prediction model as in Fig. Note the scores from LIME and SHAP (for all features) are the scale of the outcome, whereas the unnormalized escape distances (J(P) from RbX are on the scale of each feature j.
Feature
Feature value
S2(P)
Sy
LIME
SHAP
ExternalRiskEstinne MSinceeoldestt7radeee0
61.0
2.8
0.37
0.12
-0.13
149.0
00
OC
0.018
0
MS Wssinceelsssseeeeettnnexx../."""
0.0
0.5
0.12
0.084
-0.23
MS MiinceeMstteeenntDD...
3.0
00
0C
0.0067
0
AverageMInFile
49.0
00
06
0.016
0
NumT Slan
1
00
0C
0.0011
0
The gains to detection power provided by the polytope P constructed by the RbX algorithm is seen by noting that the simple feature escape distances S,(E) are infinite for every feature, as changing any single feature in the target point and keeping all others fixed - cannot change the target individual's classification from Bad to Good under the tree model. Thus simple feature importance is fully uninformative. By contrast, we have finite feature escape distances S,(P) of 2.8 for Externm sensible, as we can see from Fig. that increasing both of these features simultaneously by these two amounts is close to the shortest path for the individual to change their classification Good.
Rist i mate and 0.5 for MSi www.ostteeeeeeeeeooo I Inqexcl 7Days. This is
Next, we evaluate the stability of the various local importance methods across parameter choices. We consider 100 randomly chosen target points from the training data. At each target point run 259 iterations of SHAP, LIME, and RbX. Each SHAP iteration corresponds to a different choice of baseline among those points in the leaf of the classifier with gini impurity score 0.494 (Fig. 2). One such baseline is used for Table I Each LIME iteration corresponds to independent run of the LIME sampling procedure, and each RbX iteration uses a different set of context samples. Each set of context samples is generated by adding independent Gaussian noise to each feature dimension of the points in the training set. The standard deviation this noise is taken to the (marginal) sample standard deviation of the relevant dimension across the 259 baselines used for SHAP.
We consider the standard deviation of the ranks of the features that enter into the tree classifier across iterations for each target point. The rank of any feature assigned zero importance is defined to be one greater than the number of features with nonzero importance. We then average these standard deviations across target points and the dimensions. The result is 0.225 for SHAP, 0.193 for RbX, and
7

---

Under review as a conference paper at ICLR 2023
0.316 for LIME, demonstrating RbX predictions quite stable with respect to the choice of context samples. We also note that for LIME, one of the 18 features not entering into the tree classifier at all is assigned greater importance than one of the other 5 features in 7.9% of all iterations across the target points. Note this cannot occur for SHAP and RbX thanks to sparsity.
5.2 DETECTION POWER EXPERIMENTS
We now evaluate the detection power of various local prediction importance methods on locally sparse models, where the "ground truth" set of locally relevant features is clear. One closely related evaluation framework would be that proposed by Zhou et al. (2022), which involves cleverly introducing noise into a training dataset and then fitting various non-sparse prediction models to such dataset. If these models have sufficiently high performance, then it is clear they must place primary importance on the noisy features. They evaluate (global) feature attribution methods based on the relative importance of such features. While such a method enables measuring the performance of more complex, truly black-box models that might be used in high signal-to-noise applications like image classification, is not necessary for the simpler models examined here.
We consider the following four synthetic data generating scenarios, inspired by Chen et al. (2018) 8):
Generate (X Xg) from a spherical standard Gaussian distribution. Generate X10 independently from an equally weighted mixture of two Gaussian distributions with standard deviation I, centered at +3 and -3.
2. Let X (X1, X10). Then an outcome Y is generated follows:
XOR: E(Y X = x) = + x1 = px (x1,
Orange skin: E(Y X x) = [ [1 exp (Σ -4) = poo 22, 23, x4)
Nonlinear additive: IE(Y | X = x) = (1 + exp(-100 sin(2x11 + 2/202 + 23 +
exp(-r4))) = PN (I1, I2, #2, T.4)
Feature switching: E(Y X = x)) polo 2.)))(21)) + PN (15. 23) r(110))
where r(310) exp (- (_1X1033))
(exp (- (- [X10-3)3 3))
exp (- (xx1++))²)
2
Feature switching can be conceptualized as the following method of generating Y given X. If X10 is drawn from the Gaussian component with center +3,, Y generated according to the orange skin process with features X,...., X4. Otherwise, Y is generated according to the nonlinear additive process with features Xs...., Xs. The function r(=10) is the posterior probability that X was drawn from the component with center +3, given X T00-
For each scenario we consider two prediction models the Bayes prediction model f(x) IE(Y X = x) and a K-nearest neighbors (KNN) regressor. The KNN regressor for each scenario is fit to a random sample of 1,000 training points with K = 5, considering only the locally relevant features. For feature switching, features 1 1-4 and 10 are considered locally relevant for a target point with 3io O; otherwise, features 5-8 and 10 are locally relevant. In the other scenarios, the locally relevant features for all target points are the ones that can affect predictions in the Bayes classifier, e.g. 1-4 for the orange skin and nonlinear additive settings.
We evaluate the detection power of each local importance method across 1,000 randomly chosen target points by computing the average proportion of locally relevant features recalled among the top M most important features. Here M is the true number of locally relevant features. Ties in importance scores are broken randomly but features with zero importance are never selected. We compare the RbX escape distances (P) with the local importance scores from the following methods, which have well-documented public implementations or are trivial to code: simple feature importance (SFI, gradient method using Algorithm ("Gradient"), IG, LIME, and SHAP. For RbX we use 1,000 independently sampled context points. Only LIME does not satisfy sparsity among the approaches considered. The e-close region for each target point consists of all points the same side of the decision boundary (x f(x) 0.5). For SHAP the baseline point is the origin.
For all scenarios, RbX shows the highest recall among the methods considered, as the only procedure with perfect performance on both classifiers for all scenarios besides feature switching (Fig. 3
8

---

Under review as a conference paper at ICLR 2023
ZPA
Thokgge dei
02
TAT
Ducki 10
Cootei DHe MAA
10
10:
391 2420
2010
Figure 3: (Left) Recovery rates, based on 1,000 random target points, of the locally relevant features for various local prediction importance methods, using the smooth, sparse Bayes prediction model for each of the 4 simulated scenarios described in the main text. (Right) Same the left panel, but for the nonsmooth KNN prediction model.
The gradient-based and simple feature importance methods particularly struggle with the nonlinear additive model since there are regions where the classifiers are locally flat for instance, wherever sin (2X,,) ) is sufficiently greater than zero so that the predictions in neighborhood are all numerically equivalent to 1. RbX overcomes this by examining non-infinitesimall neighborhood around Zn-
DISCUSSION, LIMITATIONS, AND FUTURE WORK
We have proposed region-based explanations (RbX) novel, model-agnostic approach for local prediction importance. The idea to successively refine polytope approximations to a region of feature space with similar predictions the target point. The user directly specify what prediction values are "similar" based on context. By contrast, existing methods specify a "locally relevant" region in terms of the values of the features. When the number of features is moderate and there are interactions between them, specifying such a region becomes difficult. We've argued that RbX strong ability detect locally relevant features compared to existing methods while preserving sparsity, i.c. guarantee that features not used for prediction are assigned zero importance.
Whereas many local prediction importance methods like LIME and SHAP were designed with binary or unordered categorical features in mind, RbX is motivated by numeric or ordered categorical features. Further work needed to extend the ideas to settings with one or more unordered categorical features, such as image classification. In such cases the feature space becomes a zero-volume subset of Rdd, and estimating gradients no longer meaningful. Also, while our data example suggests that RbX may be reasonably robust to the choice of context points, the number of context points needed to adequately cover the space plausible feature values increases drastically with dimension d due to the curse of dimensionality. The sampling procedure for LIME and the computational complexity of SHAP suffer from similar issues, while the gradient based methods do not but at the expense of only examining f in a small number of directions, which we have argued hurts detection power.
Finally, there may also be other ways in which the polytope P constructed by Algorithm variant thereof could be useful for local prediction importance, such as escape distances from P in directions not parallel the coordinate axes. We also believe theoretical work on polytopic approximation (Bronstein) 2008 Arya et al. 2012) could be leveraged in future study to develop mathematical guarantees about how well approximates E. Any such guarantee would require some prior knowledge about the structure of f. such as smoothness or local monotonicity; for a completely generic f. the level sets can be arbitrarily non-convex. Requiring knowledge of such structure would deviate from the strict model-agnosten paradigm, but would be practically useful in setting where some information about f can be made known to the party interested in local prediction importance.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sumil Arya, Guilherme Da Fonseca, and David M Mount. Polytope approximation and the mahler volume. In Proceedings of the twenty-rhird annual ACM-SIAM symposium Discrete Algorithms, pp. 29-42. SIAM, 2012.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Efim M Bronstein. Approximation of convex sets by polytopes. Journal of Mathematical Sciences, 153(6):727-762, 2008.
Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databuses, pp. 655-670. Springer, 2018.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information- theoretic perspective on model interpretation. In International Conference on Machine Learning, pp. 883-892. PMLR, 2018.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. J. Mach. Learn. Res., 20(17):111)111 2019.
Jerome H Friedman. Multivariate adaptive regression splines. The annals of statistics, 19(1):1-67, 1991.
David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management, 5(1):81-102, 1978.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Ismael Lemhadri, Feng Ruan, and Rob Tibshirani. Lassonet: Neural networks with feature sparsity. In International Conference on Artificial Intelligence and Statistics, pp. 10-18. PMLR, 2021.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
Masayoshi Mase, Art B Owen, and Benjamin Seiler. Explaining black box decisions by shapley 2019.
cohort refinement. arXiv preprint arXiv:
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 3153. PMLR, 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), (8(1):267-228 1996.
Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9623-9633, 2022.
10

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 LINE SEARCH ALGORITHM
Here we provide sample line search algorithm for shrinking the context points given the RbX algorithm to the e-boundary.
Algorithm 3 Line-Search Algorithm
1: Input: e-far point R to be shrunk, thresholds / 0, prediction model f, target 20 E R and maximum number of iterations M.
2: Set Les <- 4, 0.
3: for iter E [1 MM] do
4: Compute fM f (tH + tr)
5: if' vn + tM (r mn) E & then
6:
1 2 (t# + +t)
7: else
8:
1H ((H + lr.)
9:
end if
10:
end for
11:
return (tH + tL)
A.2 TRUSTWORTHY REGIONS
As discussed by Mase et al. (0199), pitfall of baseline methods such as LIME, Kernel SHAP, and IG is that they often rely on predictions at implausible combinations of feature values, such as a graduation date before a birth date, due to interactions between features. Local prediction explanations utilizing such information have questionable fidelity.
To prevent the feature escape distances (P) from using information about the classifier near implausible feature values, we can establish "trustworthy region" T containing To that corresponds to the set of plausible feature values. Then if 5J (7) < Sj (P) meaning that in order to escape P by increasing the j-th feature from zo. we must leave the trustworthy region T we set S (P) = 00. We do the same thing for the S, (P). We could also make the simple feature escape distances SE)
more trustworthy in the same way.
In some settings, domain knowledge informs a reasonable choice for T. Otherwise, there are many plausible ways to define T, assuming access to a large collection of plausible feature values, such as the context points for the RbX algorithm. One such method is given in Appendix 5 of Mase et al.
(2019). Another, based on Section 14.2.4 Hastic et al. (2009), would be to estimate r r(z) (=(=)
giz)
where g(-) is viewed as an unknown joint density of the data generating process for the context points, and go(-) is known baseline density that is positive each context point, e.g. uniform rectangular region containing the context points. This function r(z) can be estimated by any binary classification procedure that outputs class probabilities. By generating large number of i.i.d. points from (-). we can learn the probability that a point at æ came from g rather than 9o. using both the original context points and the feature combinations generated from go(-). From this, an estimate f(x:) of r(x) follows via Bayes' rule. Then we define T = {x f(z) 2 (8} for some trustworthiness threshold 3 0.
We illustrate the latter algorithm using a simple quadratic logistic regression classifier fit to the popular Boston housing dataset (Harrison Jr & Rubinfeld] 1978). The goal, as Mase al. (2019) predict whether median neighborhood value is less than $20,000 based on two features CRIM
(crime rate) and RM (median number of rooms per home). We use the same target point as Mase et al. (2019). marked with red X in Fig.
The trustworthiness classifier was fit with multivariate adaptive regression splines (Friedman 1991) using logistic link, allowing for order-two interactions. The contours of this classifier are shown in Fig. 4 If T is defined with any trustworthiness threshold / larger than about 0.15, the escape distance in the positive CR direction is set because then it is impossible to escape the RbX
11

---

Under review as a conference paper at ICLR 2023
8
0.01
7
0.05
0.10
RM
0.20
0.10
0.05
0
20
40
60
80
CRIM
Figure 4: An illustration of the trustworthiness region classifier for the Boston housing data example described in the text. Its contours are indicated by the dashed lines. The solid lines indicate the halfspaces defining the polytope from the RbX algorithm for the target point Zo. denoted with red X. The arrow indicates the "polytope escape path" from xo in the CRIM direction. For any B such that the arrow crosses the 3-contour (or smaller) of trustworthiness region classifier, we set SCRIM(P (P) to co. The dots candidate context points from the entire dataset, colored by whether they are e-far.
12

---

Under review as a conference paper at ICLR 2023
polytope without leaving T. As Mase et al. (2019) suggest, this is desirable since there context points with similar RM values in the e-far region but high values of CRIM.
A.3 RESULTS FOR BOOSTED TREE ENSEMBLE ON FICO DATASET
We replicate the results given in Table but replacing the tree classifier in Fig. 2 with sparse gradient boosted ensemble fit using XGBoost (Chen & Guestrin 2016) on the same training dataset. Sparisty enforced by only training on the 5 features that appear in the simple tree classifier of Fig. 2 which are also the first 5 features in Table The training loss is the negative logistic log likelihood with early stopping after 10 iterations (chosen roughly minimize out of sample classification error). Table A.3 is analog of Table I in the main text, for the same target point Zo- Under the ensemble classifier, the predicted probability of "Good" RiskPerfoo rmance is 0.241.
Table Same as Table I but for the boosted tree ensemble classifier
Feature
Feature value
S;(P)
LIME
SHAP
ExternalRiskEstimee imate
61.0
0.48
0.062
-0.12
-0.21
MSinnceo.destrrradeeee
149.0
27.5
0.30
-0.056
-0.020
MS ResncceNess
0.0
4.16
0.97
-0.074
0
MiinceeMttRecenttelel
3.0
5.17
0.25
-0.038
-0.022
AverageMInFile
49.0
41.4
1.54
-0.028
-0.090
1
00
0C
-0.0004
13


Reviews:
[{"review_id": "kWF3AUMUpD", "reviewer": "Reviewer_2PQz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A novel method (RbX) is presented for generating localized explanations of a fitted model output by creating a region in input space (a polytope formed by the intersection of affine halfspaces ) around a given input vector such that outside the polytope, the fitted model's prediction is outside a user-specified interval around the prediction at the input vector. The method only requires query access to the fitted model, i.e., it only depends on output values at a finite user-specified set of points in input space. RbX does not require a reference/baseline input vector, unlike some competing explanation methods. RbX is also able to detect directions in input space involving simultaneous changes to multiple input variables which achieve classification output changes in cases where a single change along the coordinate of any individual input variable would not achieve a classification output change. The method is illustrated on a single input vector for a decision tree on one real-world credit scoring problem as well as on 4 toy problems. A gradient boosted tree ensemble is also examined in an appendix. ", "summary_of_the_review": "The paper presents a promising idea for model explanations by creating a local region around an input vector where the prediction doesn't change much, but it makes an unstated and unjustified assumption of locally monotone predictions which needs to be examined or defended in future versions of the work and it also does not have sufficient experimental results to justify acceptance. There is also a very confusing typo in the presentation of a key algorithm.\n\n** UPDATE post-rebuttal **\n\nGiven the typo fixes and additional discussion of local monotonicity, I have raised my score to a 5.", "strength_and_weaknesses": "The method is novel to the best of my knowledge and I think it has a lot of potential. I like the focus on how the output space changes locally and I like the ability of the method to detect complex directions in input space involving interactions between multiple variables. \n\nHowever, I don't think this research is ready for publication yet.  One serious concern (which could potentially be addressed in a future version of the work) is that the LineSearch bisection algorithm implicitly makes an assumption (which may be unfounded) that the prediction is locally monotone along the line between x_0 and a context point x. Let's take the example desired interval specified at the bottom of page 3 (prediction 13, desired interval between 10 and 20). Let's say the prediction at the initial, unshrunk context point is 21. So you do the bisection and then evaluate the function at halfway between x_0 and x and the prediction at 0.5*(x_0 + x) is now 19. OK, but that doesn't necessarily mean that the prediction is between 13 and 19 (or even 13 and 20) on the entire line between x_0 and 0.5*(x_0 + x). How do we know it doesn't rise to 25 at e.g. 0.25*(x_0 + x) and then fall down to 17 before rising again to 19 at 0.5*(x_0 + x) if we have no knowledge of the fitted function aside from query ability at individual input points?\n\nThis problem is addressable, because in general, you do have access to the properties of the fitted function, which tends to be piecewise constant for a tree or K-NN or piecewise linear if using a ReLu deep network or at least otherwise differentiable and capable of anaytic local analysis if a sigmoid-style deep network or something along those lines is used. So RbX as currently presented plus some verification of locally monotone properties would be on much more solid ground and seems achievable. Although the authors might be tempted to do rewrites in the rebuttal which would address this issue, I think it would be wiser to take more time to address the issue and resubmit either next year or in a different venue.\n\nI also think the presented experiments are much too thin. Just one data point on one real-world problem in the main text and then 4 toy problems is not enough I would like to see much more empirical evidence on real world data that the issues with competing methods like SHAP and LIME arise frequently. \n\nIt would also help to spend a bit more time on the distinction between local explanations and more global explanations which are nonetheless important but might be forgotten by a purely local explanation.  For instance, suppose a credit applicant makes $90K/year but doesn't have a college degree. The person gets denied for credit and the local explanation is a lack of college degree.  Suppose the fitted function is a tree which denies everyone with a salary below $70K but ignores additional salary above $70K. A strictly local explanation might tell the applicant that the degree is what is needed and the applicant might erroneously get the impression that approval would be achieved with a college degree and a $60K salary. While it's valuable to have a local polytope that RbX creates, it would still be good to clarify that it may not explain non-local effects like these. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EjWXVGdU_Y", "reviewer": "Reviewer_UFKv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for explaining any prediction of a black box model by constructing a convex polytopes that surrounds the prediction, such that leaving the polytope changes the prediction, in some sense of closeness. The proposed algorithm works by iteratively finding the closest “shrunken” point to a test point, estimating the gradient at that point, and using the gradient to compute a corresponding halfspace. The combination of halfspaces are used to construct the polytope, and distances to the edge of the polytope help to characterize how changing certain features affects predictions. The method is employed in two experiments, one with real data and one with synthetic data. In the real data experiment, the method proves to be more consistent than SHAP and better able to identify unimportant features than LIME. In the synthetic experiments, the proposed method is able to identify relevant features better than other competing algorithms.\n", "summary_of_the_review": "The method put forth in this paper seems reasonable but understanding its details was difficult due to exposition. Experiments were run with only a handful of simple models and 2 datasets, making it difficult to gauge the utility of the method in practice and its limitations.\n", "strength_and_weaknesses": "Strengths:\n- The method only requires query access to a black box model, like LIME or SHAP. This is useful for explanation methods.\n- The method is simple, intuitive, and reasonable.\n- The proposed method satisfies the sparsity property.\n- In experiments, especially synthetic, the proposed method recovers the most relevant features consistently.\n\nWeaknesses\n- The method is only applicable for prediction problems with a scalar output. This seems limiting. \n- The method requires a user to specify a region in output space that is close to a test point. No evidence is given that this is a simple or intuitive thing for a user to do in most cases. While it is possible for a user to specify a single set of epsilon parameters for all test points, it’s conceivable that specific test points merit specific epsilons. While this flexibility is possible, it also burdens the user of the method. (**Update after Rebuttal**: the paper now includes some examples of how setting \\epsilon could be intuitive.)\n- Empirical study should be improved. There is only 1 experiment with real data and one with synthetic data. In the real experiment, only the explanation of 1 point is evaluated. \n- Experiments are performed with random forest, bayes, and knn classifiers. No deep models are used, which calls into question the efficacy of the proposed method with these models. To be clear: it is not necessary for the work to be about deep models, but some discussion of whether or not the method is likely to work with these models is necessary given their prevalence. Intuitively, it seems like a convex polytope approximation of the closeness region may not be effective with highly non-convex models.\n- No explicit discussion of the limitations of the method; consider adding a limitations section. (**Update after rebuttal**: the revision includes some discussion of limitations, but the discussion should be expanded).\n- Many details are unclear; see below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9XHuhkj2T7", "reviewer": "Reviewer_EFoo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method (RbX) for generating local feature importance explanations for given predictions (scalar outputs) of a black-box model as a convex polytope that encloses a region (specified by a user) that is close to a target predicted point. The polytope is used to determined the boundary of features that are relevant vs. those that are not (sparsity). Thus, is is some sort of model-agnostic feature importance method, and the authors highlighted this difference with other feature importance methods. The algorithm consists on using a closeness threshold $\\epsilon$ to determine a boundary that is build by \"shrinking\" some context sample points along the line segment joining them with the boundary, i.e. by constructing the segment, finding the point in the boundary closest to the point of interest, and discarding any point outside this boundary. In addition to the boundary RbX needs the gradients of the region which are obtained using finite differences. These are used to compute the importances as scape distances for each feature. The model is evaluated with both real and synthetic data experiments. The paper uses a credit score dataset example (that predicts bad or good riskperformance) to exemplify how RbX Scape Distances compare to scores from LIME and Shap (baselines) in terms of possible interpretation. The synthetic experiments Evaluate 4 data generating scenarios to compare the detection power of local importance methods. ", "summary_of_the_review": "The paper has its pros, such as the relevance, the clear stated goals and the use of non-linear data importance generation. However, there are several reservations with respect to the applicability as the performance is comparable to existing methods. The advantage seems to be more due to the applicability across scenarios (non-linear additive, XOR, etc) but more evaluations are highly recommended. I detailed my comments on what could be improved in the Strength and Weaknesses section.\n\n%%Post rebuttal comments%%\nThank you to the authors for the reply. I think the clarifications, evaluations, and discussions are important additions to the paper. Thus, I have increased my score.", "strength_and_weaknesses": "With respect to its strengths, the paper is well organized, the problem is relevant to the ML community, particularly to model interpretability and black-box model analysis. The paper states a clear goals although the difference of \"local prediction importance\" vs feature importance assessment is rather small and due to the black-box nature of loca prediction importance. The model is geometrically intuitive which could make it appealing to a wider community. During evaluation the paper  includes a wide range of data generating scenarios including non-linear which would provide support to its validity. \nThere are some areas where the paper could be improved. For instance, the evaluations seem promising but not fully convincing. The improvement of performance w.r.t the baselines on synthetic data seems small (or none for a few data generating cases such as XOR, Orange skin).  Another weakness is that no evaluation is provided with respect to the choice of parameters e.g. the neighborhood size $\\epsilon$. How does the method decay as the number of dimensions grow is not totally clear.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1ala5TPmqZA", "reviewer": "Reviewer_4Xhj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A new region based explanation method is proposed that computes attribution scores for features as \"escape distances\" from a region/polytope defined by the user as input. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "strengths\n\n- The proposed technical idea of region based explanations is quite novel and useful. The idea of using escape distances is good. \n- Although the authors do not mention this or show via empirical results, the method might even have other indirect benefits such as the variance of explanations across similar samples might be low (needs to be checked), which would make this method a strong contender compared to LIME. \n\nweakness\n\n- Although the idea is good, the more evaluations are needed. The synthetic experiments are good. However it would be good to evaluate the method using several explainability metrics already available in the literature that do not require ground truth explanations and some that proposes to test against ground truth (https://arxiv.org/pdf/2104.14403.pdf) \n\n- Potential limitations of this method are not fully evident from the experiments. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject