Regional Explanations: Bridging Local and Global Variable Importance
Authors: Salim I. Amoukou
Nicolas J-B. Brunel
Venue: NeurIPS 2025 poster ()
Keywords: variable importance
shapley values
lime
feature importance
explainability
interpretability
leave-one-out importance
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: We analyze two widely used local attribution methods, Local Shapley Values and LIME, which aim to quantify the contribution of a feature value $x_i$ to a specific prediction $f(x_1, \dots, x_p)$.  Despite their widespread use, we identify fundamental limitations in their ability to reliably detect locally important features, even under ideal conditions with exact computations and independent features. We argue that a sound local attribution method should not assign importance to features that neither influence the model output (e.g., features with zero coefficients in a linear model) nor exhibit statistical dependence with functionality-relevant features. We demonstrate that both Local SV and LIME violate this fundamental principle. To address this, we propose R-LOCO (Regional Leave Out COvariates), which bridges the gap between local and global explanations and provides more accurate attributions. R-LOCO segments the input space into regions with similar feature importance characteristics. It then applies global attribution methods within these regions, deriving an instance's feature contributions from its regional membership. This approach delivers more faithful local attributions while avoiding local explanation instability and preserving instance-specific detail often lost in global methods.

Regional Explanations: Bridging Local and Global Variable Importance
Salim I. Amoukou" J.P. Morgan Al Research
Nicolas J-B. Brunel
LaMME, ENSIE,, University Paris Saclay & Capgemini Invent
Abstract
We analyze two widely used local attribution methods, Local Shapley Values and LIME, which aim to quantify the contribution of feature value I; to a specific prediction f (.1,..., Tp.. limitations in their ability to reliably detect locally important features, even under ideal conditions with exact computations and independent features. We argue that sound local attribution method should not assign importance to features that neither influence the model output (e.g., features with zero coefficients in a linear model) nor exhibit statistical dependence with functionality-relevar features. We demon- strate that both Local SV and LIME violate this fundamental principle. To address this, we propose R-LOCO (Regional Leave Out COvariates), which bridges the gap between local and global explanations and provides more accurate attributions. R-LOCO segments the input space into regions with similar feature importance characteristics. It then applies global attribution methods within these regions, deriving an instance's ss feature contributions from its regional membership. This approach delivers more faithful local attributions while avoiding local explanation instability and preserving instance-specific detail often lost in global methods.
Despite their widespread use, we identify fundamental
Introduction
Machine learning models are valued for their predictive capabilities, but often lack transparency. This opacity poses issues high-stakes domains such as healthcare, finance, and criminal justice, where understanding the "why" behind decision is critical the decision itself. In response, eXplainable AI (XAI) has emerged, developing methods and tools to explain model predictions.
These tools can be categorized into local and global methods. Local explanations aim to provide insights into individual predictions, while global explanations focus on understanding the overall behavior of model across the entire input space. Popular local methods, such as Local Shapley Values L-SV [Lundberg and Lee, 2017b) [b] and LIME [Ribeiro et al., 2016b), seek to create local linear approximation of the model within the vicinity of given instance. In contrast, global methods primarily consist of "leave-one-out" approaches [Lei et al., 2016, Williamson and Feng, 2020, Covert al., 2020b, Gan al., 2022], which assess performance changes when variable is excluded.
In this paper, we focus on local attribution methods, specifically L-SV and LIME, which are widely used but have limited theoretical understanding. While the intuition behind these methods are appealing, the exact quantities they estimate remain unclear. Apart from the case of linear or additive models [Bordt and von Luxburg, 2023, Garreau and Luxburg, 2020], there is little work explaining the specific quantities these methods compute. A thcoretical study on LIME, by Garreau and Luxburg [2020], shows that in the derivatives. However, it also reveals that the coefficient of important variables can vanish by simply changing parameter of the method.
of a linear model, the LIME coefficients proportional the partial
"Conresonndence to: Salim Amoukou calin..bbhhnnamoooooo@pppmgggn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Beyond these established concerns, we highlight fundamental limitation of L-SV and LIME that is orthogonal to the well-known issue of being "true to the data vs true to the model" [Chen et al., 2020]. Building on our conclusion in Amoukou et al. [2022], we posit that reliable local attribution should not assign importance to features that neither influence the model output functionally nor exhibit statistical dependence on features that do. Even under ideal conditions- -wihh independent features, exact computations, and no estimation error-we wwe show that both L-SV and LIME violate this princi- ple by attributing importance to irrelevant features. This highlights often-overlooked vulnerability in these popular methods, distinct from issues of approximation error or feature correlation.
In contrast, most global methods are supported by the existing literature on feature importance [Breiman, 2001, Lei et al., 2016, Williamson and Feng, 2020] and global sensitivity analysis [looss and Lemaitre, 2015], and are backed by strong consistency and inference results. Our goal is to leverage global methods to define more accurate local attributions for each individual while benefiting from the advantages of global methods. We aim to find partition of the input space where observations in each cell of the partition exhibit the same behavior concerning the underlying model. Then, we associate each observation with the global importance conditional on the cell it belongs to, hence deriving an attribution that is more locally faithful and possesses sound statistical properties.
Notations.
Consider a dataset represented as D, = {(X,, Yo where X; (X....., Xip) E X' C R" denotes the input variables and Y; E y C R represents the output, and (X,, Yú) are i.i.d. observations of (X, Y) ~ P = Px Pyj x- We use Xs = (Xi)ses to denote the subset of features, and [p] = (1....,p}, p). and P(D) represents the power set of a set D.
2 Limitations of Local Shapley Values
A cooperative game is pair t)), where D= (X,...., Xp) represents set of p players, and e P((I...., p}) + R denotes a value function that assigns a value to every possible coalition of players, reflecting the worth of each group. The value function v is generally assumed to be positive and increasing monotonically [Dubey and Weber, 1977], which means that if A C B, then v(A) S BB).. Here, v(A) represents the value of A- A key concept in the definition of SV is the marginal contribution, denoted as (S) e(S Ui) - vSS). The marginal contribution is the improvement of the value of a coalition S when a given player is added the coalition. The SV of X; is the weighted average of the marginal contributions of X; across all subsets, expressed as:
(iD) (91/77)
-1
Øx. = Σ w(S)A,(S)
Σ SCD(()
[u(SU - u(S]].
[S|
SCDV())
We can establish feature importance, by defining the value function. In the global sensitivity literature, a frequently used value function is v(S) = V(E[Y ( xxs)//((Y). which represents the explained This
variance or the variance of the best approximation of Y given T value function is nonnegative and monotonically increasing, resulting in a positive global importance measure. When features are independent independent, this SV is closely related to the functional ANOVA decomposition [Efron and Stein, 1981, Hoeffding. 1948] and Sobol indices [Sobol', 1990, Chastaing et al., 2012, Hooker, 2007]. The resulting SV are commonly referred to Shapley Effects [Owen, 2014, Owen and Pricur, 2017J.
In contrast to the global Shapley Values (SV) approach or Shapley Effects, Lundberg and Lee [2017a]. Lundberg et al. [2020] adopt the game theory paradigm to explain a specific prediction f fIz Tpp) with players D {X1 = F 2i,...,,,, = =pp] using the value function v(S) = Ef (z) Xs ægl or v(S) = El (æs,, x)) Although debates continue over the choice between these two value functions [Heskes et al., 2020, Janzing et al., 2020, Chen et al., 2020], we assume this work that the variables are independent, making these value functions equivalent. We refer to the resulting Shapley Values in this context as Local Shapley Values (L-SV).
A key distinction between the global SV approach (Shapley Effects) and the local approach L- SV hinges on the definitions of their respective value functions. While the global value function v(S) = Y | XY))/V(Y) serves as an effective measure of the predictive power of variables Xs for the overall model,
unclear whether the local value function u(S) E fx)) X = æs] genuinely represents the predictive power of Xs = as for the specific prediction f(æ). In the global case, a high value of v( v(S) = X (Y) indicates a strong predictive power of Xs, while the values taken by v(S)= = E[f(a)
æs)] in the local case do not have any intuitive order, i.e. high or low value of v(S) does not necessarily reflect the importance of Xs = ag for the prediction f (x)) in regression problems, for example.
2

---

Moreover, the value function of L-SV, v(S)= - satisfy the monotonic property. which may result in negative L-SV. Therefore, a cancelling effect can occur, where variable that influences the decision ends up with zero or low Shapley Value because the (S) values across all subsets cancel each other out. It important to note that Shapley Effects do not encounter the issues mentioned above, as they satisfy the non-negativity criterion suggested for feature importance [Johnson and LeBreton, 2004, Grömping, 2007, Feldman, 2005], In fact, Feldman [2005] emphasized that importance measure should be positive, as it evaluates the relative information a variable contributes to the model, and information is inherently non-negative.
= =z). can be negative and does not
While the global SV can be interpreted as percentages of the output's 's variance [[drissi, 2024], the quantities estimated by the local SV are less clear. For example, what is the L-SV for a particular X1 = 2 really telling us about its contribution to the specific prediction f Zp)?
Lastly, there is strong justification for calculating contributions across all possible subsets, as some of these subsets might be poor predictors, thus introducing noise into the feature importance. The average performance of a feature across all submodels may not be indicative of the particular performance of that feature in the set of optimal submodels. Additionally, averaging over all subsets tends to reduce the local aspect of the contribution. To demonstrate this, let's assume we have independent variables X E IR, and a piece-wise linear predictor f defined as:
f(x) = (a1 + a2Xx) Ix + (a3X3 + 14x20)xxx00
X,>0-
(1)
Even if we choose an observation x = (II...., Tp)) such that ø6 S 0 and the predictor only uses T2 the L-SV of Φzs, 22, is not necessarily zero. Using straightforward calculations, we can show that for all i E {3,4) 0z., K (a.(2. E(X.)) where K x P(X. > 0) is a constant.
This highlights that the L-SV are not purely local measures but also exhibit global influences. This occurs because when calculating the L-SV of X3 I3 X, F.g we also consider subsets S that do not contain X6. By marginalizing and changing the sign of X6. we use the other linear model not used for this observation. We can extend the result above to show similar issue with continuous piece-wise linear function. This class of functions is quite versatile, as it encompasses neural networks with piece-wise linear activations such as ReLU or hard tanh which correspond to max(0, .) and max(-1, min(1, .)) respectively. Indeed, we can view feedforward neural networks as biece-wise linear functions that divide the input space into multiple linear regions, where the network itself behaves as an affine function within each region [Pascanu et al., 2013, Hanin and Rolnick, 2019a,b, Chen et al., 2022].
The important local variables of this model correspond to the coefficients of the linear model associated with the region A which the observation belongs. However, in the following, we provide result demonstrating that these explanations cannot be retrieved using Local Shapley Values.
Theorem 2.1 (Local Shapley Values for Piecewise Linear Models). Let f: X - R. be a piece- wise linear function defined on a feature space X' C. TRP, which is partitioned into m disjoint hyperrectangles, {As] 1m where each A& = 1
with L.k. k E R {-00, 0o). On each region Ag, the function is defined by linear model (x( = = ai,x + by. The com- plete function is thus S(X) = = [ XEA= Consider an observation
(51,...,,Ip))
located in a specific region Ag- for some K* E fI....,m) The model's prediction therefore determined solely by the local model fie, i.e., f(x) (x), and the set of truly local active features is Jk- = {i E [p] Og,K- & 0}. We also define the set of features important globally G = {i E [p] 3K s.t. Ai,k F R.). Let the covariates X, be mutually independent. The Local Shapley Value (SV) for the feature-value X1 21 is given by oz: Em dr where each component &y, is:
tt, =
Σ
((S)
P(XI E AI.))
SCDVE))
E EX
+ 01,k I1
Σ SCD((1)
w(S) II
II JED((SU(())
P(X, E Aj,z)
P(Xi E ALd)
iESuft)
(2)
-( with w(S) ((")))
and vs(S) (X) Xx xs). The key implication of Equation (2) is that Local SVs can assign importance to a feature j even if that feature locally irrelevant (i.e.. j & Jk.) and globally irrelevant (i.e., j & G).
3

---

See the proof in the Appendix A. Theorem 2.1 demonstrated that SV also present difficulties in expressing local importance measures for neural networks with piece-wise linear activation layers and, more generally, for continuous piece-wise linear functions.
3 Limitations of LIME
The core idea of LIME [Ribeiro et al., 2016b] is to approximate a complex model with a simpler, interpretable model (e.g. linear model) in the neighhorhood of an instance x². However, LIME has several limitations due to its dependence on heuristic choices, such as the sampling distribution and the definition of locality. The typical sampling distribution ignores feature dependencies, which can lead to inconsistent explanations. Moreover, defining the neighborhood T (c.g., using a Gaussian kemnel) and tuning the kernel width h is challenging, particularly in high-dimensional settings, which affects the stability and reliability of the explanations. LIME also exhibits instability, with different runs producing varying results due the randomness in synthetic sample generation. Several studies [Zafar and Khan, 2019, Visani et al., 2020, Zhang et al., 2019, Alvarez-Melis and Jaakkola, 2018, Zhou al., 2021] ] highlight these issues. Additionally, we have also demonstrated that LIME, like L-SV, struggles to detect the locally important variables of the piece-wise model as in (1). For more detailed analysis and further empirical analysis these limitations, including sensitivity to bandwidth and reproducibility issues, please refer to the Appendix D.
4 From Global Explanations to Regional Explanations
In this section, we follow the presentation of Williamson et al. [2021], which introduces a general framework for global variable importance. We consider a comprehensive class F of functions mapping from to y, and F-, ; be the subset of F containing all functions that disregard the variable Xy. The conformity score ((f(X), Y) assesses the predictiveness of prediction function f E F on the observation (X, Y), where a high value implies high predictiveness. We define the oracle predictor with respect to the conformity score V and distribution P = Px Pr as follows:
fo = arg max (V FEF
In similar manner, we define ʃ-j as the function maximizing Ep[V (J(X), x))Y))) Y) over all f F-j- Subsequently, we define the population-Ievel importance for the variable X; as the decrease in predictiveness when excluding X, from Leave Out COvariates (LOCO) importance in existing literature [Lei al., 2016, Verdinelli and Wasserman, 2024]. The LOCO importance for X, is defined as:
(X1...., xp)). This commonly referred to as the
(P) = Ep (A,, where 4,
V (Jo(X),Y) - V (J-)(X)YY)
We can use any conformity score to measure variable importance, depending the problem. For regression tasks, we can use (X)(),, Y) = |Y - - /(x))=/o", where o² = Ep [Y - [Y]] represents the variance of the target variable Y. This conformity score corresponds to the traditional R² score at the population level, i.e., R2 [V (J(X), Y)). Alternatively, for binary classification problems,
use ((f(X), population level. Regarding the choice of the conformity score, there is no ground truth for variable importance as there are multiple definitions of what makes variable important [Hooker et al., 2019, Hama et al., 2022, Verdinelli and Wasserman, 2024]. Consequently. the choice of a conformity score should be contingent upon the specific context and goals of the analysis.
f(x) which corresponds to the accuracy score at the
Our approach aims to derive local explanations for specific observation (X, Y) from the population- level importance (P). This involves identifying partition U;A, X' containing observations with similar explanations, which means V Y) simultaneously. In other words, the observations within each partition have low variance in their feature importance. As a result, we can use , = EPA, [A, = Ep (X, X e
E A,) 0 for all j E {I.....P)
A,] as local explanations for all observations in A,, since the random variable A, R Y) exhibits low variance within A,. Essentially, our approach involves identifying a homogeneous group with respect to importance measure A, (X, Y) and attributing the global importance of this group as the local explanations of its members. Hence, it permits us to have local explanations while benefiting from all the inference results available for global explanations.

---

4.1 Estimation and inference
To compute our local explanations, we need to compute two quantities: the LOCO importance (P))) = Ep A,(X, Y)] for any distribution P and the partition U,A, X that group observations by their feature importance similarity. The former has been extensively studied in [Williamson et al., 2021], where the authors proposed nonparametric efficient estimation procedure using the following plug-in estimator:
- Ex)))_ElA, where ,, - (f6())))) - - (F-(()),))
(3)
P E is the empirical distribution of P based on D, (,), and fo. f-j are estimators of the population minimizers fo and f-; respectively. F- are obtained by building predictive model for Y using all features in X and after removing the variable X, respectively. This can be achieved using any machine learning algorithm. Williamson et al. [2021] demonstrated that the estimator defined in Equation (3), is asymptotically efficient and enables valid statistical inference under regularity conditions.
Having obtained consistent estimator for (P we now propose a method to derive the par- tition U,A, = X. The initial step involves creating a new representation for each observa- tion X, = (Xa....,XX, Xip) in D, using the conformity score V. This expressed as X; (X, Y.)...,, (x,,,, Representing the data in the space of feature importance, rather than the original covariate space, allows for the grouping of observations that exhibit consistent behavior with respect to the importance measure (x, V(fo(X),)) - v(f-,(X),)) across all variables j E {l,...,p} PP} simultaneously.
The subsequent step involves clustering similar observations based on their new representations X. This can be achieved using any clustering algorithm, such as K-means [Macqueen, 1967], DBSCAN [Ester et al., 1996], Tree-based clustering approach [Maimon and Rokach, 2014] or Affinity Propagation [Frey and Dueck, 2007], ultimately resulting in partition of D, into K sets C CC(Ci..... Ck) J. Note that clustering in the feature importance space does not necessarily imply that observations will be closer in the input space. In the experimental section, demonstrate that clustering in the neatur-immortance space provides a more meaningful basis for capturing the model's actual behaviour, and offer a theoretical explanation in Section 5.4.1 within controlled setting.
The final step entails using the identified clusters C define a partition of X. This is accomplished by assigning any new points E X' to their nearest partition C with respect to similarity function d: X x X - R We can use any similarity function. We use Fuclidean or Manhattan distance.
More formally, we define the corresponding region AŞ of cluster C₁ as
x E:
Σ d(s, X.EC&
Σ X,) for +1 X,EC:
In order to establish a partition of X, we must also account for the set of "undecidable" ob- servations, which are those that simultaneously belong to multiple groups We define this set Ak+1 as {x X 3,, E [k], Ex.ec, d(z, Xi) Σx,ec, d(x, X.)) Hence, the local expla- nations of given observation X, that belongs to the region As can be represented by the vector (Vi(PA)) p......... (P(()))) )) where the contribution of the feature Xj. (P).)).. is defined as
X Yt)
V (P,)),) EE
A (X
[ X E Ax]
Σ
[c
(X.Y))EC&
We refer to this approach as R-LOCO, and details can be found in Figure 6. R-LOCO's construction addresses key limitations of L-SV, particularly for models like piecewise linear functions.
Theorem 4.1. Let J be the piecewise linear function from Theorem 2.1. When the R-LOCO clustering algorithm identifies cluster Cs+ within regions Ag- the R- R-LOCO altributions for x: = (x1,...,. xp) E Age, with K- E (1,...; m) are solely influenced by fk.. They do not depend on coefficients of any a from other regions Av, where k' & k².
5

---

Unlike L-SV, which can be influenced by inactive components even under ideal conditions ($2), Theorem 4.1 shows that R-LOCO assigns importance exclusively the active region whenever Ck. C Ag.. See the proof in $B. In $B.1, we further analyze non-homogeneous (mixed) regions, where the attributions can depend on contamination from other regions; moreover, the induced bias scales proportionally with the contamination level (e.g., with the fraction of the mass of Cg- contributed by points from Ag. k + k*).
5 Experiments 1: Controlled Setting
In our first set of experiments, we evaluate our approach, R-LOCO, by comparing it with three baseline methods: two local attribution techniques, L-SV and LIME, and LOCO, which serves as the global variant of our approach. We conduct experiments in a controlled environment where the ground-truth explanations are known, using independent features to assess each method's performance without confounding bias. Our objective is to demonstrate that R-LOCO, which lies between the local and global regimes, offers greater local fidelity than state-of-the-ar art local attribution methods and provides more nuanced understanding of model behavior compared to global approaches.
R-I.OCO Variants: We explored several variants of R-LOCO in our experiments, each employing different clustering algorithms to capture local patterns.
R-LOCO: Uses Affinity Propatatooo clustering on the feature importance space. is the default strategy of our proposal.
R-LOCOTe: Uses optimal clusters based the base model structure, expected to perform better than other variants due to privileged model information. We use it as control baseline to isolate the effect of the clustering in our method.
R-LOCO free: Utilizes& tree-based algorithm that clusters by minimizing the variance on the feature importance space within each cell.
R-LOCOIc: Performs clustering directly in the input space rather than the feature impor- tance space to investigate the benefits of clustering in feature importance space.
More details of the parameters for all methods can be found the appendix C.
5.1 Models Setup
To ensure rigorous evaluation, selected model setup that inherently allows to determine ground-trute truut local explanations. Specifically, we chose models that are locally additive, enabling us to use their inherent structure to evaluate the quality of local attribution. To avoid biases that might arise from feature dependencies, we ensured that all features were independent. For data generation, we created set of n samples denoted by (X,) drawn from distribution X Px IT= Pxy- Using model f, then constructed dataset D, {(X,, in J (Xx)))==
The local attribution methods L-SV and LIME require both the model ff and dataset Dn, to generate explanations. In contrast, R-LOCO only required fixed input-output pairs, D,.- This highlights key advantage of R-LOCO: it does not need direct access to the model's 's prediction function. Instead, only needs a fixed dataset with predictions, making it suitable for scenarios where model access is restricted or for being applied directly to the data rather than a specific model. We conducted our experiments using three different models:
1st-order model: We used a piecewise linear model where local feature importance can inter- preted similarly to classical linear models. Let X (X1,.... Xg). with each component being independently distributed as X, 21| [4|-1117 1] for i E {1,...,6). The predictor function defined as follows:
f(x) (X1 + X2) 1x,50 (X, + X.): 1x,>0
(4)
2nd-order model: This model incorporates non-linear terms to introduce complexity while still being locally additive. Again, using independent variables X = all the predictor function was:
X10), where X; ~ u[-1, for
f(x) = (x + X3
Xxl) 1 1x + (X4 + X2 + V X 1 X1020

---

2nd-order interaction model: We increased the model complexity by including interaction effects, based on the function used in [Bénard et al., 2021]. The predictor was:
f(x) = 3/3 x X Lx,20 + v3 x X1X51xx500 +3 x XeX, 1 1x,20 + X9X1o x
Table 1: Performance metrics for methods across datasets from Ist-order to 2nd-ordeerinteractions TP: True Positives, FP: False Positives, NI attr:: Mean (g0.1 q0.95) of non-important features attributions.
Ist-urder
2nd-order
2ndlordde-inteerac.
Method
TP
NI att::
TP
FP
Ni astr.
TP
FP
Ni att::
R-LOCO
0.97
0.03
0.02 (0.01 0.04)
0.86
0.14
0.02 (0.01 0.044)
0.67
0.33
0.03 (0.02- 00007)
R-LOCOte
1.00
0.00
0.03 (0.03-0.03)
1.00
0.00
0.03 (0.03-0.03)
0.94
0.06
0.04 (0.03-0.04)
R-LOCOpa
0.10
0.05
0.73
0.27
0.94 (0.02 0.06)
0.50
0.50
0.06 (0.05 0.07)
R-LOCO1c
0..1)
0.74
0.07)
0.35
0.007
LOCO
0.50
0.50
0.09 (0.09-0.09)
0.50
0.50
0.05 (0.05-0.05)
0.50
0.50
0.06 (0.05-0.07)
L-S
0.41
0.59
0,12 (0.08 0.18)
0.30
0.70
0.08 (0.07 0.10)
0.44
0.36
0.06 (0.05 0.59)
LIME
0.53
0.47
0.12 0.18))
0.43
0.57
0.07 (0.05-0.10)
0.44
0.56
0.06 (0.04 (.000.099
Table 2: Performance metrics for R-LOCO using different clustering algorithms across datasets from 1st-order to orderr-nteeractionn TP: True Positives, FP: False Positives, NI attr:: Mean (q0.1 g0.95) of non- non- con-important features attributions.
de-order
2nd-order
2ud-ordee-interaat.
Method
TP
FP
NI altr.
TP
FP
NI aitz.
TP
FP
R-LOCOgMeess
0.96
0.02
0.03 (0.03-0.03)
0.92
0.08
0.04 (0.03-0.06)
0.57
0.43
0.05 (0.03-0.00)
R-LOCOKMMened
0.97
0.03
0.03 (0.02-0.08)
0.83
0.17
0.03 (0.01--D:06)
0.62
0.38
0.04 (0.02-0.00 08)
R-LOCOgMenms
0.04
00.02-
(0,00-000)))
0.59
(0.02-0.0)
00
0.97
0.03
0.02 (0.02-0.04)
0.96
0.04
0.03 (0.02-0.04)
0.64
0.36
0.04 (0.02-00.8)
R-LOCO As
0.97
0.03
0.02 (0.01-0004)
0.86
0.14
0.02 (0.01-0:04)
0.68
0.32
0.03 (0.02-00.77)
R-LOCOAmaln
0.97
0.03
0.02 (0.01-0.04)
0.86
0.14
0.02 (0.01-0.00)
0.67
0.33
0.03 (0.02-0.07)
R-LOCO
Alfiniy 8.1.OCOAfliity..
0.9
0.97
0.03
0.02 (0.01-0.44)
0.86
0.14
(0.11-000))
0.67
0.33
0.03 (0.02-0.07)
Figure 1: Diabetes dataset.
Figure California dataset.
5.2 Evaluation
For each model, we identify the & variables that contribute locally to the prediction. We exclude the variables used for defining the region (e.g., in the 1st order model, we discard the variable Xa). as these variables may also be interpreted as globally important, which could bias the analysis. Instead, we focus solely on the k variables directly involved in the computation of the final prediction. As a measure of true importance, we use the sum of the absolute feature-coefficient products across all functional components that include the feature, analogous to classical functional ANOVA contributions. In some settings, we introduce dummy variables that are irrelevant to the model and mutually independent, included solely to assess how different methods assign importance to uninformative features.
To ensure consistent comparison focused purely on importance magnitude, we normalise all attribution values. We divide the absolute value each score by the total absolute sum per observation, thereby restricting all scores to the range [o, 1] and disregarding their original directionality.
We then evaluate the performance of each method by determining how many of the k truly local important variables (those with non-zero coefficients) are correctly identified by their top k most
7

---

important features (True Positives) and how many irrelevant variables are incorrectly identified as important (False Positives). We also compute the average, as well as the 10th (q0.1) and 95th (q0.95) percentiles of the attributions assigned to non-important variables, which we denote as NI attributions. This assess the tendency of each method to over-attribute importance to irrelevant features.
5.3 Analysis
Table presents the aggregated performance metries of experiments from 50 runs on the first three synthetic datasets. The key observations from these results are as follows: Poor Performance of L-SV and LIME: L-SV and LIME consistently perform poorly across all experiments, with true positive (TP) rates of 30% to 66% and false positive (FP) rates of 34% to 70% on Table 1. For example, in the Ist-order model, L-SV achieves TP of 41 % and FP of 59%, while LIME achieves TP 53% and FP of 47%, indicating frequent misattributions. Moreover, their performance is almost identical that of the global approach LOCO, which is not designed to highlight local importance.
Superior Performance of R-LOCO: R-LOCO outperforms the baselines, with significantly higher accuracy. In the Ist-order model, R-LOCO achieves TP of 97% and FP of 3%. Although TP rates decrease slightly as model complexity grows, R-LOCC consistently outperforms the baselines.
R-LOCO assigns minimal importance to non-important features (1% to 4%). whereas L-SV and LIME assign 7% 18%. This indicates R-LOCO more effectively distinguishes between important and non-important features.
When using perfect clusters (R-LOCOTc). R-LOCO achieves near-perfect performance, suggesting that clustering quality is the key factor for R-LOCO' success. Results of R-LOCOjc also show that clustering in the feature importance space (R-LOCO) yields better results compared to clustering in input space.
Impact of Clustering:
Summary: Overall, the experimental results show that the baseline methods L-SV and LIME consistently perform poorly to identify locally important features and have performance close to the global approach LOCO. In contrast, R-LOCO and its variants significantly outperform these traditional local attribution methods, providing more locally faithful feature attributions. The effectiveness of R-LOCO largely depends on the quality of the clustering process, particularly when clustering is performed in the feature importance space.
5.4 Impact of Clustering in R-LOCO
To illustrate how clustering quality can be assessed in our method, we present qualitative example using the first-order model. We focus on an observation where X6 is positive, implying that the only locally important variables are X3 and X4.
Figure compares feature attributions from R-LOCO and R-LOCOjc (which clusters in the input space), highlighting the variance of feature-importance scores within each cluster. Both methods identify X3 and X, as the most important variables, but R-LOCO exhibits sub- stantially lower intra-cluster variance. Specif- ically, the average variance across all observa- tions and features 0.002 for R-LOCO, com- pared to 0.012 for R-LOCOJc. This lower variance suggests more consistent attributions within clusters, indicating higher clustering quality. By assessing the intra-cluster variance, practi- tioners can evaluate the clustering quality and make informed choices about clustering methods.
R-LOCO
R-LOCO
Figure 3: Feature attributions from R-LOCO and R-LOCOjc on the first-order model, highlighting the intra-group variance of feature importance.
To further assess interpretability, we train a decision tree to predict the cluster assignments. This yields a transparent and human-readable proxy of the regions identified by each method. Interestingly, R-LOCO discovers regions aligned with the true data-generating process by first splitting on the sign of Xs. whereas clustering the input space fails to do so, prioritising other non-important variables. The learned tree representation is displayed in $F.
8

---

Parameter Variations: To further assess clustering robustness, we conducted additional experiments using K.Maans with cluster counts 2, 4, 8, and 20, and Affinity Propagation with damping values of 0.5, 0.6, and 0.9.
Table 2 summarizes these findings, underscoring the robustness of R-I.OCO clustering variations.
Our key observations are:
For both first and second-order models, as long as intra-cluster variance was minimized and clusters contained enough points, performance remained stable.
Increasing the number of clusters had little adverse effect, provided cluster sizes were sufficiently large.
In second ooder interaction models with four ground-truth clusters, performance only de- clined when clusters were smaller than the true regions. Assigning more clusters than ground-truth also had minimal impact.
These results are consistent with Theorem 4.1, which states that local accuracy may be achieved once the identified cluster is part of the ground -ruth cluster or only minimally contaminated.
5.4.1 Theoretical Analysis
To formalize the advantage of clustering in importance space, we analyze the first-order model ($5.1)
with independent features X, [[[ for ...... 6 and the function f fXX (X1 + X2)
I Xes0 + (X3 X4) . X.200 This function operates in two distinct regimes based on the "switching" variable Xe: RA == {x XB S 0} and RB == (x X8 > 0}. We compare the K-Means (K = 2)
decision rules derived from clustering in the input space versus the R-LOCO importance space.
Clustering in Input Space. The true cluster centroids are the conditional means E[X | Ral and E|X RB], defined as CA = (0, 0, 0,0,
and Cp = (0, 0, O, O, 0, 0.5). This follows for i S 5) and the uniform distribution
from feature independence (B[X; RA] = E[x.] (E X6 I 0] -0.5). The K-Means separating direction is CA CB (0.0.00 0, 0, 0, -1). The decision rule, x E RA d(x, CA)² 2 < d(x, CB) simplifies to a condition based only on X6- Therefore, input-space clustering merely rediscovers the "switching" variable X6. The variability from the functionally active features ; X4) acts as noise, hindering a clean separation.
Clustering in R-LOCO Importance Space. Each sample X is mapped to its R-LOCO importance vector > (X) (A1..... As), where A, (X) = -f(X)- (x) and f (X = (X) X
The importance-space centroids for R and Rn are C,,A = (x A
(,00000)
and C.,B (X) Rg] (o, 0, 1 1. 0, (),, respectively. See for detailed derivations. The
K-Means separating direction in this space is: C..AA Cy (3. 3. - 1., 0, This decision
rule only depends on the components (A,, Ag, Az, A4) corresponding to the locally active features. Clustering in importance space successfully identifies functional similarity:
* RA samples have large (A1, A2) and As,1 0.
Rg samples have large (As, A1) and 4 #s O.
In short, input-space clustering is confounded by noise from irrelevant features, while importance- space clustering is robust to this noise and correctly groups samples by their underlying explanatory behaviour.
Remark: Importance Representation and Failure Cases We note however counter-example:
X() = X2 sgn(Xi), with regimes RA = {X, < 0} and Rg (X1 2 0}. If we use the squared-loss importance A, (x(X) f-)(X))² in importance vector >X) (AI, A2) (f(x)², f(x)² (X).x)) X²). This representation identical for both regimes, making them inseparable. The issue that the squared loss A, is sign-agnostic and discards the directional information that distinguishes the regimes.
find f-1 f-2 0. This results
This motivates enriching the importance representation. For instance, including the signed R-LOCO residual, (X) f(X) - (X), yields an enriched vector:
n' (X) (Ai, A 61, 82) XX3,x3, f(X), fX)).

---

This enriched vector maps the two regimes to distinct, separable locations:
xx) = [X2, X2, X2) if X < O, and (X2, X3 X2, X2) if 2 0.
In this space, the regimes are easily distinguished by a clustering algorithm, underscoring the importance of the chosen representation.
Experiments 2: Real-world Settings
In the preceding sections, we focused on the piecewise model setting, providing both theoretical and empirical evidence that L-SV and LIME can fail under ideal conditions (e.g., exact computation and independence assumptions). We also introduced R-LOCO to address these shortcomings in controlled context.
In this section, we extend our analysis to real-world tabular datasets, where feature dependencies naturally arise, and use widely adopted, perh-rerffrmigg models, XGBoost, for tabular data. We introduce two additional baselines:
FDT Laberge et al. [2024]: This approach is conceptually related to ours in that it seeks "regional" explanations; however, its objective to find regions where multiple explanation methods agree. In contrast, R-LOCO aims to identify zones of influence that are distinct to the model's behavior, using predictive loss as the criterion.
Clustering SHAP Values (CSHAP): As described in the original SHAP paper [Lundberg et al., 2018], I.-VV can be clustered to identify distinet zones of influence. Since SHAP does not directly employ group-wiss averages, adapt it for comparison by clustering SHAP values or L-SV (using the same algorithm as R-LOCO) and then computing averages for each group.
To evaluate the faithfulness of these methods, we measure the change in predictive performance when zero-masking features deemed most important (top-k) and least important (bottom-k) by each approach. See $H for details. A larger performance drop when masking the most important features indicates higher fidelity, whereas a smaller performance drop when masking the least important features desirable.
Figures and show the results on the Diabetes and California datasets from Dua and Graff [2017], respectively. In Figure I, our method consistently outperformed all baselines. While FDT performed best among the baselines, clustering SHAP values (CSHAP) did not outperform standard SHAP. This highlights that, unlike LOCO features, SHAP values lack sufficient information to effectively identify the distinct zone of influence of the given model, and underscores the critical importance of selecting appropriate inputs for clustering. In Figure our method was superior for identifying and perturbing the top three most important features, and FDT was slightly better for the remaining, but R-LOCO outperformed when removing non-important features, indicating superior region definition. Finally, as shown in Appendix I, our method is also significantly faster at inference, since it only requires assigning a cluster compute feature importance.
7 Conclusion
We presented rigorous evaluation of widely used local attribution methods in settings where the true local features are known. Our findings reveal that, despite their intuitive appeal, common local methods (L-SV, LIME) can fail even under ideal conditions with no feature dependencies and perfect estimations. To address this, we proposed regional method that partitions the input space into homogeneous regions-thus enabling a hybrid approach combining the fidelity of global LOCO importance with more localized attributions. Unlike existing basclines, our framework offers a clear interpretation of a feature's 's contribution as the increase in error when that feature is removed.
Our method lies between fully global and purely local approaches: avoiding the instability of pointwise explanations while still preserving instance-level insights, which purely global approaches often lack. R-LOCO implicitly assumes each region of the model can approximated by an additive structure. A key challenge is selecting and refining these regions. One promising direction is to enrich the feature-importance representation (used for clustering) with second- or higher-order interactions, thus capturing more complex dependencies. While our initial analysis focused on independent variables for clarity, we have demonstrated that the approach also outperforms baselines on real-wordd datasets where dependencies naturally arisc
10

---

Acknowledgements
We thank the anonymous reviewers for their valuable suggestions, which helped improve this work, especially regarding the counterexample illustrating the failure of the base feature-importance mapping.
Disclaimer
This work was in part conducted during Salim I. Amoukou's PhD at LaMME, Université Paris-Saclay, collaboration with the Stellantis Group.
References
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv: 1806. 0804,, 2018.
Salim I. Amoukou, Tangi Salaün, and Nicolas Brunel. Accurate shapley values for explaining tree-based models. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 2448-2465. PMLR, 28-30 Mar 2022. URL https: httpss://pccceeineeeee mwweeediinnmmm........mmmmm.mm.m press, / haw11//mmuuuu22a.... html.
Clément Bénard, Gérard Biau, Sébastien Da Veiga, and Erwan Scornet. Shaff: Fast and consistent shapley effect estimates via random forests. arXiv preprint 1724 2021.
Sebastian Bordt and Ulrike von Luxburg. From shapley values generalized additive models and back. In International Conference on Artificial Intelligence and Statistics, pages 709-745, PMLR, 2023.
Leo Breiman. Random forests. Machine learning. 45(1):5-32 2001.
Gaëlle Chastaing, Fabrice Gamboa, and Clémentine Prieur. Generalized hoeffding-sobel decomposi- tion for dependent variables-applicatioon to sensitivity analysis. arXiv preprint arXiv: I1Iiv 1788, 2012.
Hugh Chen, Joseph D Janizek, Scott Lundberg, and Su-In Lee. True the model or to the data? arXiv preprini arXiv:2006. /6234, 2020.
Kuan-Lin Chen, Harinath Garudadri, and Bhaskar D Rao. Improved bounds on neural complexity for representing piecewise linear functions. arXiv preprint arXiv.2210002.36, 2022.
Ian Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. arXiv preprint 1.. 2020a.
Ian Covert, Scott Lundberg, and Su-In Lee. Understanding global feature contributions through additive importance measures. CORR, abs/2004.00666. 2020b. URL https://arxxii......//// org/ /bs/ 2004.00663
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL /tLlhtttp/aarrciee..cc uci. edu/ml.
Pradeep Dubey and Robert Weber. Probabilistic values for games. 1977.
Bradley Efron and Charles Stein. The jackknife estimate of variance. The Annals of Statistics, pages 586-596, 1981.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters large spatial databases with noise. In kid, volume 96, pages 226-231, 1996.
Barry Feldman. Relative importance and value. Available at SSRN 2255827, 2005.
Seth Flaxman, Dino Sejdinovic, John Cunningham, and Sarah Filippi. Bayesian learning of kernel embeddings. arXiv preprint
2016.

---

Brendan J Frey and Delbert Dueck. Clustering by passing messages between data points. science, 315(5814;:972-975 2007.
Kenji Fukumizu, Arthur Gretton, Gent Lanckriet, Bernhard Schölkopf, and Bharath K Sriperumbudur Kernel choice and classifiability for rkhs embeddings of probability distributions. Advances in neural information processing systems, 22, 2009.
Luqin Gan, Lili Zheng, and Genevera Allen. Inference for interpretable machine learning: Fast, model-agnostic confidence intervals for feature importance. arXiv preprint arXiv:2206.02088, 2022.
Damien Garreau and Ulrike Luxburg. Explaining the explainer: first theoretical analysis of lime. In International Conference on Artificial Intelligence and Statistics, pages 1287-1296. PMLR, 2020.
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint arXiv: 770700226799 2017.
Ulrike Grömping. Estimators of relative importance in linear regression based on variance decompo- sition. The American Statistician, 61(2):139-177, 2007.
Naofumi Hama, Masayoshi Mase, and Art B Owen. Model free shapley values for high dimensional data. arXiv preprint arXiv:2211.00 0844144, 2022.
Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference Machine Learning, pages 2596-2604. PMLR, 2019a.
Boris Hanin and David Rolnick. Deep relu networks have surprisingly few activation patterns. Advances in neural information processing systems, 32, 2019b.
Tom Heskes, Evi Sijben, Ioan Gabriel Bucur, and Tom Claassen. Causal shapley values: Exploiting causal knowledge to explain individual predictions of complex models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL https://proceeding htwpee//00002eeaaaaaaaaa33330000 - html.
www.ripscccpppprr///eeee /22pp///020/haah
Abstract
Wassily Hoeffding. A class of statistics with asymptotically normal distribution. Annals of Mathe- matical Statistics, 19:308-334, 1948.
Giles Hooker. Generalized functional anova diagnostics for high-dimensional functions of dependent variables. Journal of Computational and Graphical Statistics, 16:709 - 732, 2007.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
Salim I. Amoukou and Nicolas Brunel. Consistent sufficient explanations and minimal local nules for explaining the decision of any classifier or regressor. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neu- ral Information Processing Systems, volume 35, pages 8027-8040. Curran Associates, Inc., 2022. URL https://prooeeedings neurips cctccc/ppppao poc/paper_fttt filler_ wti.sss/aapee/022//5555a2 0thb:3399bbbsa000460003333382223 Paaae Conference. .df..
Marouane Il Idrissi. Development of interpretability methods for certifying machine learning models applied critical systems. PhD thesis, Université de Toulouse, 2024.
Bertrand looss and Paul Lemaitre. A review on global sensitivity analysis methods. Uncertainty management in simulation-opiimizaaion of complex systems: algorithms and applications, pages 101-122, 2015.
Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in explain- able ai: causal problem. In International Conference on Artificial Intelligence and Statistics, pages 2907-2916. PMLR, 2020.
12

---

Jeff W Johnson and James M LeBreton. History and use of relative importance indices in organiza- tional research. Organizational research methods, 7(3):238 011-238-57... 2004.
Gabriel Laberge, Yann Batiste Pequignot, Mario Marchand, and Foutse Khomh. Tackling the xai disagreement problem with regional explanations. International Conference on Artificial Intellignnce and Statistics, pages 2017-2025. PMLR, 2024.
Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry A. Wasserman. Distribution- free predictive inference for regression. Journal of the American Statistical Association, 113:1094
1111, 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4774.. Neural Information Processing Systems, 2017a.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017b.
Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for 2018.
tree ensembles. arXiv preprint
Scott M Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence, 2(1):56 67,666676666 2020.
J Macqueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probabilit/UUnivesstty of California Press, 1967.
Oded Z Maimon and Lior Rokach. Data mining with decision trees: theory and applications, volume 81. World scientific, 2014.
Art Owen. indicess and shapley value. SIAM/ASA Journal on Uncertainty Quantification, (1):245-251, 2014.
Art Owen and Clémentine Prieur. On shapley value for measuring importance dependent inputs. SIAM/ASA Journal Uncertainty Quantification, 5(1):986-1002 1002, 2017.
Razvan Pascanu, Guido Montufar, and Yoshua Bengio. On the number of response regions of deep feed forward networks with piece-wise linear activations. arXiv preprint arXiv:1312.6098, 2013.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016a.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin, why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016b.
ya Meerovich Sobol', On sensitivity estimation for nonlinear mathematical models. Matematich- eskoe modelirovanie, 2(1): 1990.
Isabella Verdinelli and Larry Wasserman. Feature importance: closer look at shapley values and loco. Statistical Science, 39(4))623-636, 2024.
Giorgio Visani, Enrico Bagli, and Federico Chesani. Optilime: Optimized lime explanations for diagnostic computer algorithms. ArXiv, abs/2006.05714, 2020.
Brian D. Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using shapley values, 2020.
13

---

Brian D Williamson, Peter B Gilbert, Noah R Simon, and Marco Carone. A general framework for inference on -agonihm- variable importance. Journal of the American Staristical Association, pages 1-14, 2021.
Muhammad Rehman Zafar and Naimul Mefraz Khan. Dlime: A deterministic local inter- pretable model-agnostic explanations approach for computer-aides diagnosis systems. ArXiv, abs/1906.10263, 2019.
Yujia Zhang, Kuangyan Song, Yiming Sun, Sarah Tan, and Madeleine Udell. why should you trust my explanation?" understanding uncertainty in lime explanations. arXiv preprint arXiv: (904: 12991, 2019.
Zhengze Zhou, Giles Hooker, and Fei Wang. S-lime: Stabilized-lime for model explanation. Pro- ceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
14

---

A Proof of Theorem 2.1
Theorem A.l. Let be a piecewise linear finnctin with m components defined by the collection {f A.....; SlAu where Uk AA = X. The regions A₁ are disjoint hyperrectangles, specifically A& 0=1 Ai.k, where Ai, Ti.k) with l,,,, Ti,k R.. Each component fíAx is represented as (x) Ei= 0i bb, where the coefficients 0i and by are real numbers. Consequently, f is defined as:
17
1(X)
Σ
a + by (X).
Consider an observation x = (#1...., Tp)) located in a specific region Ag. for some k E {I...., m). The model's prediction therefore determined solely by the local model fk., i.., f(x = fi. (x), and the set of truly active featuress is Jk- -i E [p] ai,k* + 0}. We also define the set of features important globally G {i [p] 3k AL, + R.. Let the covariates X; be mutually independent. The Local Shapley Value (SV) for the feature- value X, = I2 is equal to
Rm
and ti, defined as
st, -(P(Xi E A(.)))
Au (21)
-1)
Σ w(S)eR(S)
P(Xi
SCD(O)
E (XII
01,
11
P(Xi E ALd)
E SCDV!)
w(S) x II
AA, (x))
PIX E Ai,)), (5) jEB
SESUI
Xs xs]. Equation (5) demon- strates that even if the model only uses fi- (x) for a given observation æ, the Local SV x may depend on the coefficients of the unused linear models fO for le E {1...., .......m........... (k*), which are not globally
where w(S) p 1S|
1( and (y((S) (I)A(X)
important neither.
for all 1..... m without loss of generality. Given an observation æ = (.i,..., Fp). we consider the Shapley Value of feature-value X₁ xq defined
Proof. Let's assume that IT P(X; E A.,)) > and intercepts
Σ w(S) A(())]
PR
SCD
where w(S) p 1((D)11)~ (S) v(S Ut) vSS) represent the marginal contribution, and v(S) = E[]() 2.. Note that we can decompose v(S) into m separate terms as following:
m(S)
1A (X)
X = xs
77
IE
di
(X) X = 88
(S(S)
Hence, decompose the Shapley Value of day due to the linearity property of SV
Φr.
Rim
15

---

where oz corresponds to the SV compute using the value function (S). Therefore, we only need to prove that of for & + k- is not necessarily null to prove the Theorem. We have
(x((S) E
lxx
Xs =
E
aj + Σ ai,
II 1=1
A (X,) Xs 2s
iES
iES
Σ
Ci 2
II jES
1l Ay (xz))
II
IP(x E Aji)
EE
jES
&
X11 A (X))
II P(X, E Az,k) [A))
SES
jES:jfi
JES
Similarly, we can write u(S U1) as:
(SUI) = Σ
di
II
11 A5Aa (zz)
II P(X, E Aj.A)
JESUI
JESUI
jESUI
+
Σ
B X,IAA (X.)
II JESULIAi
P(X, E A1.x)
II jESUL
A (x,)
NESU
(x1)
Σ OiAA
II JES
L A3,a (xy)
II P(X, E AJ.e) x P(Xi E A(,e)
WES
jES
(())
+ 01,871
II jE9
Agg (xn)
II P(X, E A,,x) x P(Xi AI.E)
JEB
A1, (21)
+
B
X1 A (x.)
II P(X, A5,a) II
P(Xi E A(,))
SE
JEB:jz
JES
(x))
a. A (Xi)
II JES:jæ1
P(X, E Aj,,)
II JES
(z,)
x P(X E ALk))
The terms highlighted in red and teal respectively represent the negative and positive contributions of the variable X, Iı in es(S (S U 1), the other terms will be put together form (x) follows
(SU)) = A (21)
(S(
P(Xi A1,x)
X11 Ara (XiI)
+
II JESUI
I (Z5)
II
P(X, E Az.k) 01
P(Xi ALk)
P(X: AL,))
jES
Hence, the marginal contribution of coalition S of the SV dh is equal to:
(F = (S - (S)
= (M()) ( (PiX) E AI.e) -1)
-11)
IA, (21)
E [1 Ar, (Xi)
+ II (x,) II P(X, E A1A) X OLe
P(Xi E AI,))
JESUI
16

---

Finally, we have
A
-1 Σ (SS))u(S)
P(X, E A1,k)
SCDi14
E X1 Ar
a1,k
P(X: E AL,)
E SCDY!
w(S)
II jESU
1AAA (x,)
II jES
P(X, E A3,e)
B Proof of Theorem 4.1
Theorem B.1. Let f be the piecewise linear function from Theorem 2.1. When the R-LOCO clustering algorithm identifies clusters Cg- within regions Ag- the R-LOCO attributions for (Ii....,Ty E Ag,, with k* E {l...., m) (the active region), are solely influenced by fee They do not depend on coefficients any 0i from other regions Aw, where k' + k..
Proof. The R-LOCO attribution for a feature X5 within the region defined by cluster C&- is given by the average change in the conformity score V when that feature is removed:
V (Pc..
Σ
[V(((x)) v(f-s(x)).
[Ck-1
(x..))ECCx)
where f is the full model and f-,(x) ized out.
- the model with feature j marginal-
The theorem's central assumption is that the clustering algorithm is that the identified cluster Ck- is subset of the true model region Ag., i.e., Ck. S Agr. We analyze the implications of this condition on both terms in the R-LOCO formula.
1. Analysis of the full model, f(x:): For any observation (x1, 31) E Ck., our core assumption implies X₁ Age By definition, the global piecewise model f(x) = - Su (x) x Ag simplifies to the active local model for any point within its region. Therefore, for all X1 E Cy.:
f(x)) = fkr (xi) = aT. x + by- This shows that the first term, (x ut). depends exclusively the parameters of the local model fk..
2. Analysis the reduced model, (xi)::: The proof is particularly concerned with features j that are not globally important, meaning they not used to define the region boundaries. Let = {i E [p] Ek s.t. Ar, * IR} the set of globally important features. If j & G, the condition
Ag does not depend on the value of Iz--
Consequently, for any Xı E Ck- C Ax,, any other point that agrees with x on all other coordinates (i.e., X-j x1.-)) must also lie within Age. This implies:
fx)( (X]f(X))_)) x1,-
E
f fX)
X, XI,-
The last step holds because the indicator 1x is always under the conditioning, while all other indicators A, for ke' + 1e are The resulting expression for f=i(x;) depends only on the definition of Jk.
Since both f(x) and (xi depend solely on the parameters (ags, by- for any observation in Ck+, each term in the summation for V is function of fa- alone. The attribution, being average of these terms, therefore completely independent of the coefficients (ag, by) from any other region k' * kt. This concludes the proof.
17

---

B.1 Analysis under Cluster Contamination.
We now consider the practical scenario where the cluster contaminated, meaning Ck- g Ag.. Let C.2, C₁+ n Ag+ be the set of correctly clustered points and C Cs- A be the set of contaminating points. For any point x Ch there exists some region A with k & k' such that X₁ E Ag.. For these contaminating points, the global model evaluates to s(xi) = (xi).. Consequently, the R-LOCO summation for V becomes mixture:
Φ . |Ck-1
Ax(x+ xxEC'
Σ (xi) X.ECK
where the terms A for points C₁2, depend on fe. as shown above, but the terms for points in C% depend on the parameters of other local models fe. As a result, the attribution is no longer local. It becomes "contaminated" by the influence of coefficients from the regions where the misclustered points originate. The magnitude of this effect is proportional to the fraction of contaminating points [0% |-
18

---

C
Experimental Details
Each dataset is divided into three parts: training, calibration, and test. Initially, split the data into a training set (75%) and a test set (25%). The training set is then further split evenly into a new training and calibration set. The calibration data used by each method as background for computing approximations.
For the baselines, we compute L-SV using the exact explainer from the SHAP library.
For LIME, we utilize the LIME package with default parameters, as described in the original paper [Ribeiro et al., 2a]
For R-LOCO, use XGBoost (default parameters) the base model approximate the leave-one- out functions Jo and f-j. We use Affinity Propagation with damping 0.8. Further implementation details will be given in the code repository. We split our data into two sets: one for model fitting and the other for estimating attribution quantities.
D Limitations of LIME: Detailled
The main idea behind LIME [Ribeiro et al., 2016b] is to approximate complex model using simpler, more interpretable model, such as linear model, in the vicinity given input instance. Given model f, the local explanation {(x=*) of an instance æ² is interpretable model y G, where G the set of linear models, such that
{ arg min [(f, -T) S(())
(6)
gEG
where L(f, 9: x..)) measure of how unfaithful g is in approximating f over n measure of locality around x² with width h, and (2(g) is a measure of complexity of the local model The loss C is defined as
Σ [f(x') ((z'))=(z')]]= (x').
a'-P!
In the original implementation [Ribeiro al., 2016b], nh Gaussian kernel, and the sum is taken over samples x' P' where p' = I[. Px, is the marginal law the features.
2
-2
-4.
-
Ki
X1
X3
Figure 5: Variability LIME coefficients wrt bandwidth selection on the German credit dataset (n 1000, p 20) from UCI [Dua and Graff, 2017]
Distribution of LIME coefficients for X, and X3 among observations with neg- ative 25 for the model (1)
Figure 4:
A primary concern with LIME stems from its reliance arbitrary heuristics in its definition. Specif- ically, choosing the sampling distribution P' poses challenges, the commonly used distribution disregards feature dependencies, and there is no guarantee that the model's local behavior on p will be consistent with that on observed data. Another significant issue lies in defining neighborhood whh and tuning the kernel width h, especially in high-dimension. The stability and sensitivity of LIME are heavily influenced by the selection of the perturbation sampling distribution, the definition of proximity wh and the bandwidth h, which may result in varying explanations for
19

---

the same instance under slightly varying settings. To illustrate this issue, we apply LIME on the piece-wise linear model defined (1), with a = 0, a2 2. 0g = =5
In Figure 4, the distribution of LIME coefficients for X1 and X3 among observations with negative F6 values shows that LIME assigns a non-null score both X1 and X3, although the latter is not used locally for the displayed observations. Thus, LIME shares the same problem as L-SV in the piece- wise linear model with independent variables. is also important to note that such discontinuities are not uncommon, as tabular data often contain discontinuities with categorical variables, and the meaning of constructing a linear approximation in such cases remains unclear. Besides this empirical evidence, we are currently working to compute the theoretical quantity of LIME coefficient for continuous micc--wse--lnean functions, yielding impossible results similar Thereom 2.1 for Local Shapley Values.
To demonstrate the sensitivity of LIME to bandwidth selection, we applied it to the German credit dataset (n = 1000, p 20) from UCI [Dua and Graff, 2017]. We trained a RF on 80% of the dataset and computed LIME coefficients on the remaining data using two bandwidths, h and h', of the proximity kernel - We set f using the median heuristic [Fukumizu et al., 2009, Flaxman et al., 2016, Garreau al., 2017], where h is the median of the pairwise distance of ||X; - Xjll2 and assigned h' = h.. In Figure 5, we compare the relative absolute error of the LIME coefficients for all variables using the two bandwidths, i.e., (a. Lz, //L. where h h' m; m.; represent the LIME coefficient of the variable X; using bandwith h, h' respectively. It reveals that the difference between the LIME coefficients could be much different after slightly modifying the bandwidth. Furthermore, observed that 20% of the coefficients also changed signs. In real-world scenarios, we lack information about the true local importance, making the bandwidth selection process indefinite. Moreover, LIME exhibits issues related to instability or irreproducibility. For example, Zhang et al. [2019], Zafar and Khan [2019], Visani et al. [2020] have shown that repeated runs of the same setting of the algorithm on the same model and data point can yield different results. This inconsistency stems from the randomness introduced during the generation of the synthetic sample around the input. Several works [Zafar and Khan, 2019, Zhou et al., 2021] attempt to address this issue, using asymptotic analysis of the method to identify the minimum number of the sampled observations required for stability. Another aspect of stability is related to input perturbations. Alvarez-Melis and Jaakkola [2018] show that nearby observations may have completely different LIME coefficients.
E
Visual Description of R-LOCO
Apple the treatienntie d//i E2 seroos bertero oberretiee
Disttton the dena lan IS siglee todlo tiee meve lnanary yo 1o
CLu cark cherraaiii to the dhoogled ryllaceb
exmmaki oach Eaeoar www.rreaaaaaa.....
Clanko gollo LOCO
3 k.
Figure 6: This diagram presents and contrasts our proposed method (R-LOCO) with the global approach (LOCO). Both approaches first transform the features into the feature importance space. The global approach (in grey) then averages the importance scores column-wisg across all observations, while R-LOCO (in green) averages the scores within each cluster, providing a more localized attribution. However, note that nothing stops from applying any other global methods after the clusters are identified. The estimation of the region-based attributions is very generic; we could simply learn a linear model within each identified region and use it as a global explanation, or apply any more advanced global attribution techniques. For example, we could re-estimate the fo. and f-, of (X, V (fo(x), y) -V -V(f((X),)) used to estimate T (Px,) within the corresponding cluster instead of using the one learned to identify the clusters. We found that both strategies perform similarly in our experiments.
20

---

Learned Tree of Section of 5.4
The decision rules defining the regions shown in Section 5.4 are: Regions identified by R-LOCO in feature-importance space
X6 <= 0.00
X5 -0.86
---- class: 0
--- X5 -0.86
---- class: 0
0.00
--- X4 <= -0.56
---- class: 1
X4 -0.56
---- class: 0
Regions identified by clustering in the input space
-0.01
--- X2 <= -0.00
---- class:
--- X2 > -0.00
---- class:
---- X5 > -0.01
--- X6 0.05
---- class: 2
X6 0.05
---- class: 3
G Derivations of the centroids for the 1st-order model
To formalize the advantage of clustering in importance space, we analyze a model with independent features X, U[ 1,,1, for .,.
. 6 and the function:
S(X) = (X1 X2) 1x,50 + (X3 + X4) 1 1x,00
This function operates in two distinct regimes based on the "switching" variable Xg:: RA = fx X6 S 0} and RB (x X > 0}. We compare the K-Means (K 2) decision rules derived from clustering in the input space versus the R-LOCO importance space.
Clustering in Input Space The true cluster centroids are the conditional means E[X RA] and E|X Rg]:
CA (0,0,0,0,0, -0.5), Cg (0, 00,0, 0,,00,0,
This follows from feature independence (E(X; RA] = E = 0 for < 5) and the uniform distribution (E[X6
X6 S 0] 0.5).
The K-Means separating direction CA - CB (0, 0, 0,0, - The decision rule, d(x, CA)² d(x, 2. simplifies to condition based only X6. Therefore, input-space clustering merely rediscovers the "switching" variable X6. The variability from the functionally active features (X,....,44 Xx) acts as noise, hindering clean separation.
Clustering in R-LOCO Importance Space Each sample X is mapped to its R-LOCO importance vector x(X) (Ai,.... As), where A (S(X) [x)( and f-j(X) [E[(())
Y-==X -=,=X_1 X-
Consider X E RA, where X6 I 0 and f(X) = X1 X2. We find the components A, and their expectations [[AA) RA]:
2, (Active): f-i(X)= E(Y + X21 X2-
21

---

A1 (X1 -2,
-B| Ra]= [4]==E|X3 [[X]]] 1/3. By symmetry, EB|2 RAl 1/3.
j = 3, 4,5 (Inactive): f-3(X) = E[x1 + X2] = =x + X2.
A1,455 = (X1 + (X1 ( x
B|A3,4,5 RA]=0. =0
6 (Switching): [-o(X) = Bral(X, + X2)1yo50 + (Xa X4)1y = 2Xx
X2) [X3 X4).
As ((X) X2) -X1 + X
+ + -x3 X3 X4)²
+
B|As RA] ! + + + x -4
The importance-space centroid for RA is C,.A 0,0,0, By symmetry, for X Rg (where f(X) X3 + X4), the centroid Cy, (0, 0.
Decision Rule and Interpretation The K-Means separating direction in this space is:
1
- 3' 3'
3 3' 0,0).
This decision rule only depends on the components (A,, A2, Ag. A1) corresponding to the locally
active features. Clustering in importance space successfully identifies functional similarity:
RA samples have large (41, A2) and 43,4 R
RB samples have large (As, A1) and 41,2
In short, input-space clustering is confounded by noise from irrelevant features and only finds the geometric switch Xg. Importance-spe space clustering is robust this noise and correctly groups samples by their underlying explanatory behavior.
Remark: Importance Representation and Failure Cases We note known counter-example: f(X,, X2) = X₂ sgn(X (X1), with regimes RA {X1 0} and Rs = (X1 2 0).
If we use the squared-loss importance A, (f(x) (X)) we find f-1= 0. This results in importance vector (X) (A..A)) (f f(X))) (X3 X2). This representation identical for both regimes, making them inseparable. The issue that the squared loss A, is sign-agnostic and discards the directional information that distinguishes the regimes.
This motivates enriching the importance representation. For instance, including the signed R-LOCO residual, (X) f(X)= fX)( yields an enriched vector:
(X) (Ai, A2, 81, 62) (X2, X2 /(),, ((X)).
This enriched vector maps the two regimes to distinct, separable locations:
7()) [13
if X (Regime RA),, if Xi (Regime Rp).
(X3,
In this space, the regimes easily distinguished by clustering algorithm, underscoring the importance of the chosen representation.
H Evaluation methodology for Real-World Data
H.1 Definitions
We begin by outlining the core components of our evaluation framework:
Dataset D: A collection of n samples. Each sample a pair (x, v). where x denotes the input and y the ground-truth target.
Input x: A d-dimensional vector of numerical features.
22

---

Model f(x): The predictive model that maps to prediction.
Attribution Method A(x): A function that assigns a positive importance score to each of the d input features. These scores correspond normalized attributions from each explanation method (e.g., R-LOCO, SHAP).
Error Function [x, y): metric quantifying the discrepancy between model's predic- tion for and the true label y:
Classification:
E y) = 1 if the prediction is incorrect, 0 otherwise.
Regression: Exx, y) may be the absolute difference between the predicted and true values.
H.2 Masking Procedure
To assess feature importance, we apply masking (or occlusion) technique as follows:
1. For each input x, compute the feature importance scores using A(x).
2. Identify the indices of:
The top-k most important features (Top-k Indices).
The bottom-k least important features (Bottom-k Indices).
3. Create modified versions of
mosked.top-k
Replace values at Top-k indices with zero,
x mottocd
Replace values at Bottom-k indices with zero.
H.3 Error Calculation
We compute the average model error over the dataset under three conditions:
Original Average Error (Baseline):
AvgError Original n (x.y)ED
1
Σ
E(x,
y)
Top-k Masked Average Error:
AvgError top- T
Σ (x,9)E
E maskedltpp- 3))
Bottom-k Masked Average Error:
AvgError Bottomn
Σ
E(x masked. y)
n (X.v)ED
H.4 Final Metrics for Figures
The values shown in Figures and represent the change in model error due to masking, computed follows:
Top-k Error Change (Left Figure): Indicates the degradation performance after removing the Top-k most important features:
ChangelnError (k) = AvgErroT AvgError
Bottom-k Error Change (Right Figure):
Reflects the performance change after removing the
Top-k least important features:
ChangeInErrorpooleo Boltom (k)
AvgErroronginnl AvgError I.EErroBoooion--. Boliom
23

---

Runtime and Memory Usage Analysis
We find that R-LOCO is significantly faster inference time than the baselines. While our method incurs additional computational cost during training (due to clustering and model fitting), it performs inference with minimal overhead, as only the cluster assignment required per sample. similar pattern is observed for memory usage.
In terms of scalability, R-LOCO's s training cost grows approximately linearly with the number of features. This behaviour is consistent with widely used explanation approaches such as SHAPLoss and its variants [Covert al., 2020a]. Importantly, this training step is performed only once.
Moreover, R-LOCO can be further optimised for tree-based models using efficient approximation methods such as Projected Forests [Bénard et al., 2021, I. Amoukou and Brunel, 2022], where single model can be reused to compute multiple conditional expectations efficiently.
Method
Train (s)
Infer (s)
Total (s)
Train (MB)
Infer (MB)
Peak (MB)
R-LOCO
33.521
0.032
33.553
1687.9
8.6
2743.0
LIME
0.038
34.684
34.722
0.5
2.3
1.0
SHAP
0.002
20.698
20.700
0.0
0.8
6.8
Table Runtime and memory usage comparison on the California dataset.
Method
Train (s)
Infer (s)
Total (s)
Train (MB)
Infer (MB)
Peak (MB)
R-LOCO
1.611
0.010
1.621
12.2
0.5
1.7
SHAP
0.002
1.433
1.436
0.0
145.0
16.7
LIME
0.008
0.838
0.846
0.2
1.3
0.4
Table 4: Runtime and memory usage comparison on the Diabetes dataset.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Yes, the main claims in the abstract and introduction accurately reflect the paper's contributions and scope. We clearly state the identified limitations of L-SV and LIME. Specifically, we argue that a sound local attribution method should not assign importance to features that do not influence the model output (e.g.. features with zero coefficients in a linear model), and lack statistical dependence with functionality- features. We introduce R-LOCO method to bridge local and global explanations, which overcomes these limitations. These claims are supported by theoretical results (Section 2, Theorem 2.1; Section 4.1, Theorem 4.1, Theoretical analysis in $5.4.1) and empirical evidence in controlled (Section 5) and real-world settings (Section 6). Guidelines:
Justification:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Most of the experiments and theoretical results this paper focus on piecewise linear models. Extending these results to other settings should be done with caution, particularly because the notion of a "ground truth explanation" " difficult to define. Our second theorem the utility of R-LOCO is proved under the condition that clusters either align with or are fully contained within the true underlying regions. To address more realistic scenarios, we also provide an analysis of cases where this assumption does not hold (See in $B.1 or 5.4. 1), offering deeper insight into R-LOCO's behavior under imperfect clustering. While the independence assumption may be isolate the effects of attribution without confounding dependencies. In addition, we do evaluate R-LOCO on real-world datasets that naturally include feature dependencies, and our results demonstrate that the method remains effective and better than the baselines in these settings.
limitation, made it primarily to
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution
25

---

is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
For Theorem 2.1 (Limitations of L-SV for piecewise linear functions): The assumptions (piecewise linear function f with m components, disjoint hyperrectangle regions Ag. independent covariates) stated within the theorem. The full proof is provided in Appendix A.
For Theorem 4.1 (R-LOCO behaviour for piecewise linear functions): The assumptions (piecewise linear function from Theorem 2.1, R-LOCO clustering algorithm correctly identifying regions corresponding to or included in Ag) are stated. A detailed proof is in Appendix B.
For the theoretical analysis of R-LOCO in $B.1, all assumptions and complete proof' are also given.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material. Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Indeed, we have provided all essential details required to conduct our ex- periments. Details regarding model setups (Section 5.1), R-LOCO variants (Section 5), evaluation metrics (Section 5.2), baseline parameters and R-LOCO implementation specifics (Appendix C), and real-world experiment configurations (Section 6) are included. The code will be released upon acceptance.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
26

---

If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We report on generating the synthetic experiments (Section 5.1), citations for the open source data used and commit to releasing the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Baaii///guidess/oeeSuuniin wwt.ssummisssooosiioooooooooo onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips.cc/pppiiii wttpccccppppcccceee g/ggi//guuuuo/ des/dd/e////eeee Cooo//uooieieeeeeeeee cittmmsssionii ssionPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
27

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, the experimental details are all given. Specifically, data splitting pro- cedures are described in Appendix C. Hyperparameters for baseline methods like LIME (defaults) and R-LOCO's Affinity Propagation (damping=0.8) are mentioned in Appendix C. Section 5.3 explores variations in clustering parameters for R-LOCO. The models being explained (c.g.. piecewise linear, XGBoost) are also specified. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The analysis section (Section 5.3) mentions that results presented aggre- gated over 50 runs. Table and Table report mean values and quantile ranges (q0.1 - q0.95 for NNI attr:), which provide information about the distribution and variability of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We execute all experiments on MacBook Pro M2, with none of them being particularly costly to conduct. All Results are obtainable within few hours.
Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : /ttp///eerppp.eeeee httpppllccEthhssSuuuse ines?
Answer: [Yes]
Justification:
The research conforms to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper focuses primarily on the positive social impact by improving model understanding and trustworthiness. This work advances the analysis of widely used explainability tools that play crucial role in high-stakes fields such healthcare. We highlight their limitations and propose a more theoretically sound and empirically robust approach to improve interpretability and reliability in model explanations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
29

---

Answer: [NA]
Justification: Not applicable. The paper does not release new large-scale models or datasets with high intrinsic risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all data are correctly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Not applicable. The paper primarily introduces a new method, R-LOCO. Code for this method is a new asset that is committed to be released upon acceptance (as stated in answers to Q4 and Q5).
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
30

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMss if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy (https: /tt////eeiipp///eeee for what should or should not be described.
31


Reviews:
[{"review_id": "I0mG7KgU1Q", "reviewer": "Reviewer_dyeU", "summary": "The paper identifies flaws in commonly used local explanation methods: Local Shapley Values (L-SV) and LIME, even under idealized assumptions of independent features and exact computation. The authors prove via Theorem 2 that L-SV can assign nonzero importance to features outside the true active region of a piecewise-linear model, and similarly show LIME failures under the same setup. To remedy this, they propose R-LOCO (Regional Leave Out COVariates), which:\n\n1. Computes global LOCO importance for each observation.\n2. Embeds instances in the “feature-importance space” defined by their LOCO vectors.\n3. Clusters these embeddings into K sets (variants in table are based on different clustering schemes).\n4. Assigns each point the region’s averaged LOCO scores as its local explanation.\n\nThey prove in Theorem 4.1 that if a cluster lies entirely within a true region, R-LOCO recovers exact local attributions for piecewise-linear models. Through controlled simulations with ground-truth explanations (first and second-order additive and interaction models), R-LOCO achieves true-positive rates up to 97–98% with false-positive rates below 4%, outperforming L-SV and LIME (TP 30–66%, FP 34–70%). In real-world tests on the Diabetes and California datasets (using XGBoost), R-LOCO also yields higher fidelity: masking its top-k features drops model performance more and masking bottom-k features drops it less, compared to L-SV, LIME, CSHAP, and the FDT baseline.", "questions": "*Scalability & Runtime*\n\n1. Issue: LOCO’s leave one covariate approach requires fitting p + 1 models per cluster (or approximations thereof).\n2. Request: Please report wall clock training and inference time (and memory usage) for R-LOCO vs. L-SV/LIME on the Diabetes and California datasets and discuss scaling behavior as p grows.\n\n*Automatic Region Validation*\n\n1. Issue: Cluster purity is critical for correctness (Theorem 4.1) but cluster-straddling degrades performance (App C).\n2. Request: Please integrate a purity metric (e.g., variance threshold on LOCO vectors) to automatically detect and refine impure clusters. The goal is to answer: under what quantitative purity threshold does performance sharply degrade?\n\n*High-Dimensional / Correlated Features*\n\n1. Issue: All controlled experiments assume independent features. As the authors acknowledge real datasets often exhibit strong correlations. I am unsure how much of this correlation is depicted in the 2 datasets considered.\n2. Request: Have you tested R-LOCO on synthetic correlated feature settings (e.g., Gaussian multivariate with block covariance) to gauge robustness? How do correlations affect clustering fidelity and TP/FP rates?\n\n*Choice of Distance Metric*\n\n1. Issue: Euclidean distance in LOCO space may underperform when features have different variance scales.\n2. Question: This maybe another issue for high dimensional data. Have you compared Mahalanobis or cosine distances for clustering? Could adaptive metric learning improve region coherence?\n\n*Applicability*\n\n1. Issue: It remains unclear how R-LOCO generalizes to unstructured data or non-additive model parts (e.g., CNN feature maps).\n2. Question: Are there specific use cases/applications that benefit from the computational overhead of the proposed method?", "limitations": "The paper does not explicitly discuss limitations in the main paper. However, there is an answer regarding limitations in the checklist. I recommend adding one in the main paper that discusses:\n    • Clustering assumptions: Requirement that clusters align with true functional regions; sensitivity to hyperparameters.\n    • Computational overhead: Multiple model fits for LOCO and clustering steps.\n    • Model/domain scope: Currently validated on tabular, piecewise-linear or tree-based models; adaptation to deep networks or non-tabular data is future work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Quality**\n\n*Strengths:*\n\n1. Theoretical rigor: Full proofs for Theorems 2.2 and 4.1 appear in Appendices A and B, with clear statement of assumptions (piecewise linearity, independence, clustering inclusion).\n2. Comprehensive experiments: Controlled benchmarks include 50 runs per setting, error quantiles for non-important features, and ablations over four clustering algorithms and hyperparameters (K = 2,4,8,20; damping = 0.5,0.6,0.9) demonstrating stability.\n3. Real-world validation: Utilizes public datasets (Diabetes, California), high-performing XGBoost models, and two separate fidelity metrics (top-k and bottom-k masking).\n\n*Weaknesses:*\n\n1. Clustering reliance: Performance depends heavily on cluster purity (Appendix C). It is unclear how the method performs in in more high-dimensional data where feature overlap is higher. I was unable to find if automated purity checks are utilized in the paper\n2. Compute cost: LOCO requires re-fitting models per region; no detailed runtime or memory analysis is given, limiting assessment of scalability to high-dimensional or large-scale data. \n3. Applicability: LIME and shap values are applicable across domains (extensions to large-scale vision datasets as well). Due to reliance on clustering, I am unsure of the proposed method's applicability.\n\n**Clarity**\n\n*Strengths:*\n\n1. Logical flow: Paper moves cleanly from theoretical critique (Sec. 2–4) to controlled experiments (Sec. 5) and real data (Sec. 6)\n2. Notation consistency: Feature sets, clusters, and Shapley/LOCO notation remain consistent throughout.\n\n*Weaknesses:*\n\n1. Dense appendices: Key implementation details (e.g., distance metrics, damping choices) are relegated to appendices, forcing readers to flip back and forth. Fig 6 (which is in appendix) should be in the main paper.\n2. Acronym overload: Readers must track R-LOCO, R-LOCOTC, R-LOCOIC, CSHAP, FDT consider consolidating or emphasizing defaults.\n\n**Significance**\n\n*Strengths:*\n\n1. Bridges local/global gap: R-LOCO offers a general recipe to turn any global explainer into a regional/local one, addressing a central challenge in XAI.\n2. Actionable insights: Demonstrates that clustering in importance space (vs. input space) yields markedly better fidelity, guiding future explainer design.\n\n*Weaknesses:*\n\n1. Domain scope: Focuses exclusively on tabular, piecewise-linear, or tree-based models; applicability to vision, NLP, or deep nets remains unclear.\n\n**Originality**\n\n*Strengths:*\n\n1. Novel theoretical critique: Per my awareness, the paper is the first to formally prove failures of L-SV under ideal conditions for piecewise models.\n\n*Weaknesses:*\n\n1. Building blocks: Clustering algorithms and LOCO are existing techniques; R-LOCO’s novelty lies in their combination rather than entirely new algorithms.", "was_revised": true, "final_justification": "I am increasing my score to 4 and my confidence to 3. The additional results are along expected lines with the proposed method suffering due to scaling the number of features. However, for some applications (as noted by the authors in the rebuttal) it might be justifiable. The lack of more complicated synthetic settings in Section 5 is still a concern.", "rebuttal": "We thank the reviewer for their comments and the opportunity to clarify our contributions. Below, we address the specific points raised.\n\n> Q: Request: Please report wall clock training and inference time (and memory usage) for R-LOCO vs. L-SV/LIME on the Diabetes and California datasets and discuss scaling behavior as *p* grows.\n\n**A**: Thank you for this helpful suggestion. We have now conducted a detailed runtime and memory usage analysis for R-LOCO, LIME, and SHAP on both the California and Diabetes datasets.\n\nWe find that R-LOCO is significantly faster at inference time than the baselines. While our method incurs additional cost during training, due to clustering and model fitting, it performs inference with minimal overhead, as only the cluster assignment is required per sample. We observe a similar pattern for memory usage. \n\nIn terms of scalability, R-LOCO’s  **training** computational cost grows approximately linearly with the number of features. This behavior is on par with widely-used approaches such as SHAPLoss and variants (Explaining by Removing: A Unified Framework for Model Explanation). Note that it needs to be done only once. Furthermore, R-LOCO can be optimised for tree-based models using approaches such as Projected Forests (SHAFF: Fast and consistent SHApley eFfect estimates via random Forests), where a single model can be reused to efficiently compute multiple conditional expectations.\n\nWe believe these additions substantially strengthen the contribution of our paper.\n\n#### California Dataset\n| Method   | Train (s) | Infer (s) | Total (s) | Train (MB) | Infer (MB) | Peak (MB) |\n|----------|-----------|-----------|-----------|-------------|-------------|-----------|\n| R-LOCO   | 33.521    | 0.032     | 33.553    | 1687.9      | 8.6         | 2743.0    |\n| LIME     | 0.038     | 34.684    | 34.722    | 0.5         | 2.3         | 1.0       |\n| SHAP     | 0.002     | 20.698    | 20.700    | 0.0         | 0.8         | 6.8       |\n\n#### Diabetes Dataset\n\n| Method   | Train (s) | Infer (s) | Total (s) | Train (MB) | Infer (MB) | Peak (MB) |\n|----------|-----------|-----------|-----------|-------------|-------------|-----------|\n| R-LOCO   | 1.611     | 0.010     | 1.621     | 12.2        | 0.5         | 1.7       |\n| SHAP     | 0.002     | 1.433     | 1.436     | 0.0         | 145.0       | 16.7      |\n| LIME     | 0.008     | 0.838     | 0.846     | 0.2         | 1.3         | 0.4       |\n\n---\n\n> Q: Please integrate a purity metric (e.g., variance threshold on LOCO vectors) to automatically detect and refine impure clusters. The goal is to answer: under what quantitative purity threshold does performance sharply degrade?\n\n**A**: Thank you for the suggestion. While we report results with several clustering parameters to give a detailed analysis, in our current implementation, we include a knee-point strategy to choose the number of clusters. We also support any clustering class with `fit` and `predict` methods, enabling integration with methods that include automatic cluster validation.\n\n---\n\n> Q: Have you tested R-LOCO on synthetic correlated feature settings (e.g., Gaussian multivariate with block covariance) to gauge robustness? How do correlations affect clustering fidelity and TP/FP rates?\n\n**A**: We have deliberately avoided introducing correlation in the synthetic experiments because our goal is to evaluate feature importance under the assumption of independence, where ground truth importance can be unambiguously defined.\n\nIn correlated settings, disentangling the effects of statistical dependence from true functional importance becomes challenging, making it difficult to compute TP/FP rates in a principled way. \n\nThat said, our experiments on real-world datasets inherently include correlations, and we evaluate performance there using perturbation-based evaluation as there is no ground truth available in the real-world setting.\n\n---\n\n> Q: Have you compared Mahalanobis or cosine distances for clustering? Could adaptive metric learning improve region coherence?\n\n**A**:  As noted in lines 210–212 of the main paper, R-LOCO supports any similarity function during inference, including those used by the clustering algorithm. We have experimented with cosine distances in place of the Mahalanobis, and found minimal impact on overall conclusions.\n\nNonetheless, we agree that in some settings, particularly in domain-specific structures, careful metric selection could yield improvements. Our implementation supports passing a custom distance function and clustering algorithm, giving users full control over this aspect.\n\n---\n\n> Q: Are there specific use cases/applications that benefit from the computational overhead of the proposed method?\n\n**A**:  A significant majority of real-world machine learning applications, particularly in critical sectors like finance, healthcare, retail, and insurance, are in majority built on structured, tabular data. Many applications on tabular data can afford moderate computational overhead in exchange for more faithful and locally accurate explanations. \n\nMoreover, methods with similar computational profiles, such as SHAPLoss and variants (see *Explaining by Removing* paper) are already used in practice. R-LOCO offers additional interpretability benefits by structuring explanations into coherent regions, which is often desirable in real-world workflows where more localised decisions are important.\n\n---\n\n> Q: However, there is an answer regarding limitations in the checklist. I recommend adding one in the main paper that discusses: • Clustering assumptions: Requirement that clusters align with true functional regions; sensitivity to hyperparameters. • Computational overhead: Multiple model fits for LOCO and clustering steps. • Model/domain scope: Currently validated on tabular, piecewise-linear or tree-based models; adaptation to deep networks or non-tabular data is future work.\n\n**A**:  Thank you for this important suggestion. We agree that these limitations should be more explicitly stated in the main paper and will revise accordingly.\n\nAs noted in lines 227–229, we provide a theoretical analysis of the impact of imperfect or contaminated clusters on the fidelity of R-LOCO explanations. Additionally, Section 5.4 and Table 2 present an empirical analysis of how clustering affects performance.\n\nWhile R-LOCO requires model refitting per feature, similar to methods like SHAPLoss, this overhead is incurred only during training. Inference is highly efficient. Moreover, for tree-based models (commonly used in tabular data), the training cost can be significantly reduced using techniques such as Projected Forests (SHAFF: Fast and consistent SHApley eFfect estimates via random Forests).\n\nIndeed, our method is currently validated on tabular data, as are most feature removal-based explanations. Extending R-LOCO to deep learning models or non-tabular modalities is a promising direction for future work.\n\nWe will make these limitations more explicit in the final version of the paper to ensure clarity and transparency."}, {"review_id": "hbqDJkI9Ba", "reviewer": "Reviewer_nEVM", "summary": "The paper under review tackles the timely and important problem of providing principled, interpretable and trustworthy feature attributions for predictions of black-box models. The authors argue that the widely used methods LIME and Local Shapley Values (LSV) can assign importance to features that do not locally influence the model's output, even when the features are independent. Inspired by the fact that global importance methods like Leave Out COvariates (LOCO) do not suffer from this problem, they propose to provide local feature importance in the form of global feature importance scores for a restriction of the predictor to a specific region of the input space. They exploit the fact that the LOCO scores for a feature is formulated as  an expectation over inputs of the gap in performance with respect to a best predictor that does not have a access to that feature. They explore different clustering methods to identify regions that enjoy low variance in performance gaps, effectively leading to sets of inputs for which the expectation is a good approximation. They first evaluate their approach in a controlled setting where the data generating processes are explicitly defined by pieces, providing evidence that their approach succeeds at retrieving more accurately the variables that effectively appear on each piece while less often identifying variables that do not, compared to LIME and LSV. They further use their approach on two real world datasets to compare its fidelity towards observed performance drops with two other regional approaches.", "questions": "Could you provide examples of the regions identified by R-LOCO using AffinityPropagation and include a discussion on the interpretability vs \"clustering quality\" trade-off? Such a discussion should be included in subsection 5.4 in my opinion.\n\nIn section 6, it seems to me that R-LOCO and the two other baselines provide feature importance on each identified region. How were these multiple feature importance views used to obtain a single variable ranking used in Figure 1 and 2? If a single ranking is indeed used, why not compare to LOCO in this evaluation?\n\nAs a follow up question, what would be the importance ranking provided by R-LOCO with true clusters for the first order model used in the paper? Since $X_6$ has no importance on both regions would it be ranked after $X_1,\\ldots, X_4$? If so, it seems that by chosing carefully the paramaters, the performance of $f_{-6}$ can be made arbitrarily low, which seems to me like an undesirable property.\n \nIn definition 2.1, is the whole function required to be continuous? While the name refer to $m$-continuity, no requirement is stated about this. Also this terminology ($m$-CPWL) is introduced, but no reference is made elsewhere in the paper if I'm not mistaken. What is the point of introducing this definition?\n\nIn the proof of Theorem 4.1, it is assumed that $\\hat{f}_{0},\\hat{f}_{j}$ are estimated for each region, which seems to contradict lines 191-192 that present them as estimators of population minimizers. Could you clarify this point? In your implementation, is the clustering done using local or global minimizers?\n\nLine 215, is the boundary set $\\hat{A}_{K+1}$ used in practice? What local attribution is computed in this case?\n\nLine 247, what criterion is used for clustering in input space?\n\nLine 281, It seems to me that this claim does not necessarily hold. Could you please elaborate on this or provide a reference for this fact?\n\nLines 277-278, could you provide formulas for the ground truth attributions you use, in particular for the model with interactions?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well written and proposes an interesting perspective on local feature importance by drawing inspiration from global feature importance. Experiments are interesting and well chosen to provide the reader with a good perspective on the challenges of feature attributions and the contributions of the paper.  \nIn a sense, the method can be described as identifying a classifier $\\kappa:\\mathcal{X}\\to [1,\\ldots,m]$ and estimate the global variable importance scores LOCO $\\Psi^k$ for the restrictions on the regions $A_k=\\{x\\in\\mathcal{X} : \\kappa(x)=k\\}$. The local variable importance for an input $x$ is then given by $\\Psi^{\\kappa(x)}$. While $\\Psi^k$ is well justified (at least in the case of independent features and in absence of interactions) for the predictor restricted on $A_k$, the method does not account for variable importance of the classifier $\\kappa$. In extreme cases (for a fine enough partition), this approach would result in a constant predictor on each region (and so $\\Psi^k=\\vec{0}$ for all $k$) while the classifier $\\kappa$ could be arbitrary complex. In this extreme case, R-LOCO would present null variable importance for all inputs, even though the predictor may be arbitrarily complex and actually rely on several specific variables, which can arguably be viewed as influencing the global model's output. I believe that the interpretability of the classifier $\\kappa$, and thereby the complexity of the partition, is a key component of such an approach, which is not discussed in the paper. It would appear to me that any approach which relies on partition of the input space needs to deal with a trade-off. While partitions with a large number of small regions will lead to smaller variance, they are also harder to interpret. \n\nMoreover, if a feature may not be important locally (like $X_6$ in the first order model), I would argue that it is an important feature when it comes to R-LOCO feature importance, since R-LOCO feature importance (say with true clusters) at input $X$ depends on the sign of $X_6$. I believe that in many applications this importance of $X_6$ cannot be overlooked. If the importance of the variables on which depends the clustering is viewed as a separate problem by the authors, I find that the paper lacks a discussion on how to reconcile this phenomenon with the local feature importance towards a global understanding of the model.\n\nRelated to the method and the evaluation, I find that some important information is missing (see questions) which makes it difficult to understand the advantages and limitations of the proposed approach.\nIn particular, it seems to me that the evaluation on real world datasets is focusing on global feature importance where as the proposed approach is specifically designed for local feature importance.", "was_revised": true, "final_justification": "Good paper addressing important questions about explainability of black box models. The main issue of the proposed approach resides in the lack of interpretability of the regions which potentially hinders the overall usefulness of the final explanations.", "rebuttal": "We thank the reviewer for their comments and the opportunity to clarify our contributions. Below, we address the specific points raised.\n\n> Q: Could you provide examples of the regions identified by R-LOCO using AffinityPropagation and include a discussion on the interpretability vs \"clustering quality\" trade-off? Such a discussion should be included in subsection 5.4.\n\n**A**: Thank you for the suggestion. We agree that illustrating the identified regions will improve our analysis. However, most clustering algorithms only assign group labels without offering interpretability. In response, we fit a decision tree to predict the cluster labels, providing an interpretable proxy for the regions identified.\n\nInterestingly, we observed that R-LOCO correctly identifies the important regions by first splitting on the sign of X6, in contrast to clustering in the input space. This insight will be incorporated into Subsection 5.4 to better illustrate the interpretability vs clustering quality trade-off as suggested. See the regions found below:\n\nThe region for R-LOCO in feature importance space is:\n```\n|--- X6 <= 0.00\n|   |--- X5 <= -0.86\n|   |   |--- class: 0\n|   |--- X5 >  -0.86\n|   |   |--- class: 0\n|--- X6 >  0.00\n|   |--- X4 <= -0.56\n|   |   |--- class: 1\n|   |--- X4 >  -0.56\n|   |   |--- class: 0\n```\nand the region for R-LOCO in input space is:\n```\n|--- X5 <= -0.01\n|   |--- X2 <= -0.00\n|   |   |--- class: 0\n|   |--- X2 >  -0.00\n|   |   |--- class: 1\n|--- X5 >  -0.01\n|   |--- X6 <= 0.05\n|   |   |--- class: 2\n|   |--- X6 >  0.05\n|   |   |--- class: 3\n```\n---\n\n> Q: In Section 6, how were the multiple feature importance views combined into a single ranking for Figure 1 and 2? Why not include LOCO in the comparison?\n\n**A**: In this experiment, we were interested in comparing the “local” methods by perturbing each instance based on its absolute instance-wise importance. Since real-world data lacks ground-truth explanations, we rely on perturbation-based proxies that operate at the instance level. Using LOCO would result in the same perturbation across all instances, which does not align with the local evaluation setup.\n\nHowever, we agree it is a valid point, and we will include LOCO as a baseline in the revised version. In our experiments, it performed less well than R-LOCO.\n\n---\n\n> Q: As a follow up question, what would be the importance ranking provided by R-LOCO with true clusters for the first order model used in the paper? Since $X_6$ has no importance on both regions would it be ranked after \n$X_1, \\cdots, X_4$? If so, it seems that by chosing carefully the paramaters, the performance of \n can be made arbitrarily low, which seems to me like an undesirable property.\n\n**A**: Comparing the importance of the splitting variable to other features is not straightforward. We excluded it from our evaluation to avoid introducing bias as the splitting variable can be viewed as globally important. Instead, we focus on variables that are either locally important (as per our definition) or not important at all (i.e., functionally and statistically independent).\n\nTo clarify further, we conducted a more fine-grained analysis (See proof in rebuttal of reviewer GsiJ). For the first-order model, assuming $R_A = {X_6 \\leq 0}$ and $R_B = {X_6 > 0}$ we explicitly computed the feature importance variable \n\n- For $\\( X \\in R_A \\)$:\n  $$\n  \\gamma_A(X) = \\left(X_1^2, X_2^2, 0, 0, 0, 0.25(X_1 + X_2 - X_3 - X_4)^2 \\right)\n  $$\n- For $\\( X \\in R_B \\)$:\n  $$\n  \\gamma_B(X) = \\left(0, 0, X_3^2, X_4^2, 0, 0.25(X_1 + X_2 - X_3 - X_4)^2 \\right)\n  $$\n\nAveraging these expressions over their respective regions yields equal importance for the relevant features in each region:\n- For $\\( X \\in R_A \\)$:\n  $$\n  E_{R_A}[\\gamma_A(X)] = \\left(1/12,1/12, 0, 0, 0, 1/12 \\right)\n  $$\n- For $\\( X \\in R_B \\)$:\n  $$\n  E_{R_B}[\\gamma_B(X)] = \\left(0, 0, 1/12, 1/12, 0, 1/12 \\right)\n  $$\n\n We will include this analysis in the appendix.\n\n---\n\n> Q: In Definition 2.1, is the function required to be continuous? Also, the term \"-CPWL\" is introduced but not used again.\n\n**A**: The definition was adapted from the cited papers. However, in our specific setting, we focus on each m function being affine. The definition was meant to introduce the class of piecewise linear functions used in the following theorem. We will revise the text\nto make it clearer and use the affine function instead of continuity. Thank you for pointing this out. \n\n---\n\n> Q: In the proof of Theorem 4.1, it is assumed that $\\hat{f}{0},\\hat{f}{j}$ are estimated for each region, which seems to contradict lines 191-192 that present them as estimators of population minimizers. Could you clarify this point? In your implementation, is the clustering done using local or global minimizers?\n\n**A**: Thank you for pointing this out. Clustering is performed using global minimisers.\n\nAs we mentioned in Figure 6 (which we will move into the main paper), once regions are identified, in principle, any global method can be applied within each region.\n\nThe proof uses regional estimators for $f_{-j}(X)$. However, as we said in L567, the other case can be easily deduced. In fact, using the global minimiser $\\( f_{-j}(X) = \\mathbb{E}[f(X) \\mid X_{-j}] \\)$ still satisfies the proof’s conditions, which requires that it does not depend on variables outside the corresponding model used for the observation $X$. This condition is met whenever $X_j$ is locally important: removing this variable in the condition does not alter the switching variable, so both $f(x)$ and $f_{-j}(X)$ evaluate to the same region-specific affine function. We'll revise the explanation for clarity.\n\n---\n\n> Q: Line 215 – Is the boundary set used in practice? What local attribution is computed there?\n\n**A**: As mentioned, the boundary set can be treated as a separate group. However, such cases are very rare in practice. Typically, ties can be broken by randomly assigning the sample to one of the valid regions. We will clarify this in the final version.\n\n---\n\n> Q: Line 247 – What criterion is used for clustering in input space?\n\n**A**: We use the same clustering algorithm and criterion as for the feature importance. We just swap the input matrix with the feature importance matrix.\n\n---\n\n> Q: Line 281 – The claim seems questionable (LOCO values being between (0, 1). Could you elaborate or provide a reference?\n\n**A**: Thank you for pointing this out. LOCO is often linked to the variance explained (e.g., Sobol indices); its values are positive, and when properly normalised, typically fall within a defined range. We will clarify this and cite sources such as: \"Feature Importance: A Closer Look at Shapley Values and LOCO.\"\n\n---\n\n> Q: Lines 277–278 – Could you provide the formula for ground-truth attributions, especially for the interaction model?\n\n**A**: Yes, we use the absolute product of feature values and their corresponding coefficients as the ground-truth measure of importance. We do not account for directionality in this evaluation.\n\nFor example, for a model\n\n$\nf(X) = \\sum_i f_i(X_i) + \\sum_{i < j} f_{i,j}(X_i, X_j) + \\dots,\n$\n\nthe total contribution of feature $\\( X_k \\)$ at point $\\( X \\)$ is the sum of all components involving \\( X_k \\). In our interaction model:\n\n$\n\\text{Contribution of } X_1 = \\text{Contribution of } X_2 = 3 \\sqrt{3} X_1 X_2\n\nWe compute true/false positives using the absolute value of the contributions."}, {"review_id": "89VpZc1o2F", "reviewer": "Reviewer_GsiJ", "summary": "This paper introduces a local importance method R-LOCO (Regional Leave Out Covariates), which tries to address limitations of SHAP and LIME. To improve the reliability of these methods, the authors first cluster the feature importance space, and then apply global importance methods to each cluster. The simulations and real data analysis shown the methods improved the previous local feature importance methods.", "questions": "1. In most of the literature, local importance typically refers to the importance assigned to a single prediction. In that sense, can R-LOCO be understood as calculating global importance for clusters?\n2. The choice of the 10th and 95th percentiles as bound for the non-important features attributions in Section 5.1 is unconventional. Are there any considerations for not using 5th and 95th percentiles?\n3. For the German credit datasets with 20 features, is there any reason only 19 features are shown in Figure 5 in Supplementary Material E?\n4. Could you provide some intuition for why clustering in the feature importance space works better than clustering in input space as demonstrated in Section 5. Specifically, what causes the reduced variance of R-LOCO in Section 5.4?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1. The paper identified some limitations in local importance methods in SHAP and LIME.\n2. The paper proposes a method that combines clustering and LOCO to bridge local and global importance methods.\n3. In simulation and real-world datasets, R-LOCO outperformed SHAP and LIME in terms of both TP and FP.\n\nWeakness:\n\n1. The authors only considered the case that the population can be clustered. However, it’s unclear how R-LOCO performs when the population is homogenous. In such cases, I would expect clustering to be less effective. I think it could be a good sensitive analysis, particularly if its performance remains comparable to Global LOCO. This could help demonstrate R-LOCO’s robustness.\n2. The paper didn’t clearly state how the model determines which features are non-important. If the means are all non-zero as shown in Section 5.1, it’s confusing how are negatives identified. A clearer description of the decision rule would improve readability.\n3. The “Mean Performance Drop” in Figure 1 and Figure 2 is negative, I believe it should be positive instead when important features are perturbed. \n4. For better clarity, it’s better to at least mention the dimension of 2nd-order interaction model in Section 5.1. \n\nTypos and formatting:\n\n1. Missing period in line 377.\n2. Figure 1 and Figure 2 which are associated with Section 6 are placed on page 7, but Figure 3 associated with Section 5.4 is placed on page 8. I think it could be rearranged for better readability.", "was_revised": true, "final_justification": "All the issues are solved.", "rebuttal": "We thank the reviewer for their comments and the opportunity to clarify our contributions. Below, we address the specific points raised.\n\n---\n\n> Q: Could you provide some intuition for why clustering in the feature importance space works better than clustering in input space as demonstrated in Section 5? Specifically, what causes the reduced variance of R-LOCO in Section 5.4?\n\n**A**: Consider the model  \n\n$\nf(X) = (X_1 + X_2)\\cdot 1_{X_6 \\le 0} + (X_3 + X_4)\\cdot 1_{X_6 > 0},\n$\n\nClustering in the feature importance space (generated by R-LOCO with squared loss conformity score) yields a more robust and functionally meaningful separation of the model's two operational regions than clustering in the original input space. Below, we demonstrate this by analysing the decision rules associated with clustering in each space.\n\n#### Part 1: Clustering in Input Space\n\nThe model defines two regions based on the value of $X_6$:\n\n- **Region A ($R_A$):** $\\( X_6 \\le 0 \\)$\n- **Region B ($R_B$):** $\\( X_6 > 0 \\)$\n\nThe effectiveness of clustering depends on the geometric separation between the point clouds of $R_A$ and $R_B$. We analyse this by computing their theoretical centroids:\n\nThe centroid of Region A is  \n\n$\nC_A = \\mathbb{E}[X \\mid X \\in R_A] = (0, 0, 0, 0, 0, -0.5),\n$\n\nand the centroid of Region B is  \n$C_B = \\mathbb{E}[X \\mid X \\in R_B] = (0, 0, 0, 0, 0, 0.5)$.\n\n**Proof:**  \nFor $i \\in \\{1, ..., 5\\}$, $X_i$ is independent of $X_6$, so  \n$\\( \\mathbb{E}[X_i \\mid X_6 \\le 0] = \\mathbb{E}[X_i] = 0 \\)$.  \n$\\( X_6 \\sim \\text{Uniform}[-1, 0] \\Rightarrow \\mathbb{E}[X_6 \\mid X_6 \\le 0] = -0.5 \\)$.\n\n\n**Conclusion for Part 1:**  \n\nAlthough the centroids are distinct, the difference lies only in $X_6$, such that any variability in the other five dimensions $X_1,\\dots, X_5$ introduces noise, making geometric separation in input space unreliable for clustering, as we can show by considering the decision rule:\n\n$x \\in R_A$ if $d(x,C_A)^2-d(x,C_B)^2<0$. \n\nThis rule is the one that underlies K-means clustering, where each observation is attributed to the region with the closest centroid. Expanding this gives:\n\n$d(x,C_A)^2-d(x,C_B)^2 = -2 \\langle x, (C_A-C_B) \\rangle + \\Vert C_A \\Vert^2 - \\Vert C_B \\Vert^2$,\n\nSo the clustering decision rule reduces to\n\n $x \\in R_A \\Leftrightarrow -2 <x,(C_A-C_B)> + \\Vert C_A \\Vert^2 - \\Vert C_B \\Vert^2 < 0 $ or $ \\langle x, (C_A-C_B) \\rangle > 0$\n\nThis rule is similar to the classical linear SVM $sign(\\langle x,(C_A-C_B) \\rangle +b)$ (where +1 means $R_A$, and -1 means $R_B$) with normal vector $(C_A-C_B)=(0,0,0,0,0,-1)$. Thus, only $X_6$ is used for discriminating between the two groups (and roughly speaking $sign(X_6)$). Although the centroids are distinct in the $X_6$ dimension,  noise from the other five features (which are irrelevant to the decision boundary) hinders reliable clustering in input space. We show in part 2 that the clustering in R-LOCO Importance Space gives much better separation. \n\n---\n\n#### Part 2: Clustering in R-LOCO Importance Space\nEach point $X$ is mapped to an importance vector  \n\n$\n\\gamma(X) = (\\Delta_1, \\dots, \\Delta_6),\n$\n\nwhere   $\\Delta_j(X) = (f(X) - f_{-j}(X))^2,$  and the LOCO predictor is $f_{-j}(X) = \\mathbb{E}[f(X) \\mid X_{-j}]$.\n\nFor $X \\in R_A$, the importance vector is  \n\n$\n\\gamma_A(X) = \\left(X_1^2, X_2^2, 0, 0, 0, 0.25(X_1 + X_2 - X_3 - X_4)^2 \\right).\n$\n\n---\n\n**Proof:**  \nIn Region A, $f(X) = X_1 + X_2$.\n\n-  $f_{-1}(X) = \\mathbb{E}[X_1 + X_2 \\mid X_{2..6}] = \\mathbb{E}[X_1] + X_2 = X_2 \\Rightarrow \\Delta_1 = X_1^2$\n- $f_{-2}(X) = \\mathbb{E}[X_1 + X_2 \\mid X_{1,3..6}] = X_1 + \\mathbb{E}[X_2] = X_1 \\Rightarrow \\Delta_2 = X_2^2$\n- For $j \\in \\{3,4,5\\}$, $\\Delta_j = 0$\n- $f_{-6}(X) = 0.5(X_1 + X_2) + 0.5(X_3 + X_4) \\Rightarrow$\n\n  $\n  \\Delta_6 = \\left(X_1 + X_2 - 0.5(X_1 + X_2 + X_3 + X_4) \\right)^2 = 0.25(X_1 + X_2 - X_3 - X_4)^2\n  $\n---\n\nAs previously, we compute in the feature space the decision rule for clustering the feature vectors by computing the centroids.\n\nThe centroid of Region A’s importance vectors is  \n\n$\nC_{\\gamma,A} = \\left(\\tfrac{1}{3}, \\tfrac{1}{3}, 0, 0, 0, \\tfrac{1}{3} \\right).\n$\n\n---\n\n**Proof:**  \nSince all  $X_i \\sim \\text{Uniform}[-1,1]$:  \n$\\( \\mathbb{E}[X_i^2] = 1/3 \\)$, and all cross terms vanish due to independence and zero mean.\n\n---\n\nBy symmetry, for $\\( X \\in R_B \\)$:  \n\n$\n\\gamma_B(X) = \\left(0, 0, X_3^2, X_4^2, 0, 0.25(X_1 + X_2 - X_3 - X_4)^2 \\right)\n$\n\nand the centroid is  \n\n$\nC_{\\gamma,B} = \\left( 0, 0, \\tfrac{1}{3}, \\tfrac{1}{3}, 0, \\tfrac{1}{3} \\right).\n$\n\n---\n\n**Conclusion for Part 2:**  \n\nAs before, the decision rule is \n\n$sign(<\\gamma(x), (C_{\\gamma,A}-C_{\\gamma,B})> +b)$,\n\n where +1 means $R_A$, and -1 means $R_B$ and \n$C_{\\gamma,A}-C_{\\gamma,B}= \\left(\\tfrac{1}{3}, \\tfrac{1}{3}, -\\tfrac{1}{3}, -\\tfrac{1}{3}, 0 \\right)$. Thus, \n\n $x \\in R_A$ if $sign(<\\gamma(x), C_{\\gamma,A}-C_{\\gamma,B})>0$.\n\nThe features used for each cluster $R_A, R_B$ are only based on the local important features, unlike input space clustering, the importance space reveals separation across multiple dimensions $\\( \\Delta_1, \\Delta_2, \\Delta_3, \\Delta_4 \\)$. More precisely, \n\n- **Region A**  \n  If a point is in Region A, $f(\\mathbf{X})$ depends on $\\(X_{1}\\)$ and $\\(X_{2}\\)$.  \n  Thus, $\\Delta_{1}$ and $\\Delta_{2}$ will be large, while $\\Delta_{3}$ and $\\Delta_{4}$ will be zero.  \n  The expression will be positive.\n\n- **Region B**  \n  If a point is in Region B, $f(\\mathbf{X})$ depends on $\\(X_{3}\\)$ and $\\(X_{4}\\)$.  \n  Thus, $\\Delta_{3}$ and $\\Delta_{4}$ will be large, while $\\Delta_{1}$ and $\\Delta_{2}$ will be zero.  \n  The expression will be negative.\n\nThis captures fundamentally different patterns of feature usage, exactly what R-LOCO is designed to uncover.\n\n---\n\n**Final Conclusion:**  \nWhile clustering in input space gives only information on the globally important variable, R-LOCO reveals a multi-dimensional separation aligned with the model's logic. It transforms the problem from finding a geometric partition to identifying consistent feature relevance patterns.\n\n---\n\n> Q: The paper didn’t clearly state how the model determines which features are non-important. If the means are all non-zero as shown in Section 5.1, it’s confusing how negatives are identified. A clearer description of the decision rule would improve readability.\n\n**A**: Lines 276–278 explain that we focus solely on variables directly involved in the computation of the final prediction after the condition. We use the absolute product of feature values and their corresponding coefficients as a measure of importance.\n\nWe are not concerned with directionality here. Our primary test is whether a method assigns importance to independent and unused variables, not whether it captures sign.\n\nFor example, for a model\n\n$\nf(X) = \\sum_i f_i(X_i) + \\sum_{i < j} f_{i,j}(X_i, X_j) + \\dots,\n$\n\nthe total contribution of a feature $\\( X_k \\)$ at point $\\( X \\)$ is the sum of all components involving $\\( X_k \\)$.  In our interaction model:\n\n$\n\\text{Contribution of } X_1 = \\text{Contribution of } X_2 = 3 \\sqrt{3} X_1 X_2.\n$\n\nAs we are not interested in directionality, we take the absolute value of the contributions for computing TP and FP.\n\nWe will clarify this in the revision.\n\n---\n\n> Q: The authors only considered cases where the population can be clustered. What happens when the population is homogeneous? Could a sensitivity analysis help validate R-LOCO’s robustness?\n\n**A**: Our main goal is to demonstrate a clear failure case for traditional baselines. Piecewise models are the most natural example where locality breaks down, hence our focus on them.\n\nHowever, our real-world example, where the population is likely not clusterable, shows that R-LOCO remains robust and effective compared to existing methods.\n\n---\n\n> Q: Why use the 10th and 95th percentiles for non-important feature attributions in Section 5.1? Why not 5th and 95th?\n\n**A**: No particular reason, we verified that the conclusions hold using the 5th and 95th percentiles. We will update the manuscript accordingly.\n\n---\n\n> Q: In the German Credit dataset (20 features), why are only 19 shown in Figure 5 (Supplementary Material E)?\n\n**A**: This was a mistake. The last feature was accidentally omitted while improving figure visibility in Inkscape.\n\n---\n\n> Q: The “Mean Performance Drop” in Figures 1 and 2 is negative. Shouldn’t it be positive when important features are perturbed?\n\n**A**: Apologies for the confusion. We used the error, not accuracy. The base error is used as the baseline.\n\n---\n\n> Q: For clarity, the dimension of the 2nd-order interaction model in Section 5.1 should be mentioned.\n\n**A**: We will include this detail. However, note that only the variables actually used in the model's definition are considered.\n\n---\n\n> Q: In most of the literature, local importance typically refers to the importance assigned to a single prediction. In that sense, can R-LOCO be understood as calculating global importance for clusters?\n\nYes."}, {"review_id": "9CC3SGcEKd", "reviewer": "Reviewer_v4Qd", "summary": "The authors first discuss a major limitation of the vanilla-SHAP local explanation method : when the model is piece-wise linear over regions $A_1, A_2, \\ldots$, the local importance of a feature in $A_k$ can be polluted by model behavior in other regions, leading to counterintuitive interpretations.\n\nTo address this limitation, it is proposed to first compute a point-wise loss increase $\\Delta_j(X, Y)$ when removing feature $j$ from the dataset, cluster these local Importance vectors in $C$ groups, and then provide local feature importance to new points based on which cluster it is closest to.\n\nThe method is assessed on toy examples and 2 tabular datasets.", "questions": "## Questions\n- In section 5.2, it is not clear how \"product of feature values and their corresponding coefficients\"\napplies to the non-piecewise linear models (i.e. 2nd order models with and without interactions).\nWhat measure of importance is used as ground truth for these two functions? This clarification is required for me to increase my score because the paper has a large emphasis on ground-truth evaluations.\n\n- The model $f_{-j}$ used in R-LOCO is fitted on the whole dataset minus feature $j$. However, the proof of **Theorem 4.1** requires that the model $f_{-j}$ is **refitted** only using data that lands in a specific cluster $C_k$. This assumption is not part of the premise of the theorem nor is it discussed in the main text. It is only discussed on the caption of Figure 6 from the appendix. I think the premise of the theorem should be rewritten to include this assumption. \n\n## Counter-example\nI do not think that the R-LOCO approach is well motivated by focusing on piece-wise linear models. To prove my point, I propose a toy example where R-LOCO seemingly fails. I would like the authors to address or repudiate it.\n\nLet $X_1,X_2\\sim U(-1, 1)$ and $f(X_1, X_2) = X_2 \\mathbb{1}(X_1\\geq 0) - X_2\\mathbb{1}(X_1< 0)$. From what I understand of the paper, an ideal regional explanation method should identify the two regions $\\mathbb{1}(X_1<0)$ and $\\mathbb{1}(X_1\\geq 0)$ and report the linear model coefficient in each region (or at least feature importance consistent with these coefficients). \nHowever, applying clustering in the feature importance space (instead of the input space) is not able to recover the two regions. Indeed, since the conditional expectation is the optimal predictor under the squared loss, we have\n- $f_{-2}(x) = \\mathbb{E}[f(X)|X_1=x_1] = 0$\n- $f_{-1}(x)=\\mathbb{E}[f(X)|X_2=x_2]=0$.\n\nThe corresponding LOCO effects using $Y=f(X)$ and the squared loss as the conformity score are\n\n- $\\Delta_1(X, f(X)) = (f(X)-f_{-1}(X))^2 = f(X)^2$\n- $\\Delta_2(X, f(X)) = (f(X)-f_{-2}(X))^2 = f(X)^2$\n\nNow consider the map $\\gamma: [-1, 1]^2 \\rightarrow \\mathbb{R}^2$ defined as $X\\mapsto (\\Delta_1(X, f(X)), \\Delta_2(X, f(X)))=(f(X)^2, f(X)^2)$. Since $f(X_1, X_2) = -f(-X_1, X_2)$, we have that $\\gamma(X_1, X_2) = \\gamma(-X_1, X_2)$. That is, the two regions $\\mathbb{1}(X_1<0)$ and $\\mathbb{1}(X_1\\geq 0)$ in input space completely overlap when mapped through $\\gamma$. Consequently, no clustering algorithm applied in the LOCO space could recover the ground-truth regions.\n\nNote that this counter-example is also applicable CSHAP (i.e. clustering on the L-SV). Therefore, this critique is not specific to R-LOCO but more to any clustering method that is applied on the feature importance space instead of the input space.", "limitations": "The authors clearly highlight that the mathematical results all assume that the model is piece-wise linear. This assumption is not too restrictive seeing as it covers Tree-based methods (piece-wise constant) and ReLU networks.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well written and a breeze to read from start to finish.\n- The technical sections 2, 3, and 4 are very clear.\n- The limitation of SHAP highlighted by **Theorem 2.2** is an important contribution to the community.\n- The experiments on toy data and models highlight the benefits of reporting regional explanations instead of fully local or global.\n\n## Weaknesses\n- Regional explainability is an emerging subject and there are existing methods that the current paper is not citing. For instance, the Effector package [1] (and the methods within) are not cited. Since [1] is still a preprint, I cannot be too severe on its exclusion in the experiments. Adding a citation + discussion would \nbe sufficient.\n\n- The notion of \"local feature importance\" is not consistent within the paper. For instance, at line 123, it is stated that the feature local importance of a piece-wise linear model is the coefficient used in the region. At line 277, it is instead stated that the \"product of feature values and their corresponding coefficients\" is the local importance. These are two different importance measures and the correct one should be clarified directly after definition 2.1.\n\n- Although I think the weakness of SHAP highlighted by **Theorem 2.2** motivates well the need for a regional explanation method, I do not think it motivates R-LOCO especially well. To explain why, I refer to the **Question** section below where I present a toy example where R-LOCO fails to recover ground-truth regions on a piece-wise linear model, and ask the authors to respond to this counter-example. This counter-example puts into question the emphasis on piece-wise linear models as a motivation for the approach.\n\n\n[1] Effector: A Python package for regional explanations. https://arxiv.org/pdf/2404.02629v1", "was_revised": true, "final_justification": "I have increased my score to \"borderland accept\" in light of the rebuttal.", "rebuttal": "We sincerely thank the reviewer for their thorough evaluation and insightful comments. This feedback is invaluable for improving the quality and clarity of our work. Below, we address each point in detail.\n\n---\n\n> Q: On the Definition of Local Importance: how our definition of local importance: “the product of feature values and their corresponding coefficients”, applies to non-piecewise linear models (e.g., second-order models with or without interactions), and what measure of importance is used as ground truth for these models.\n\n**A**: The correct definition for our experiments is indeed the “product of feature values and their corresponding coefficients.” We will revise the text following Definition 2.1 to ensure this definition is applied consistently throughout the paper.\n\nFor the non-piecewise linear models in Section 5.2, we define the ground-truth feature contribution using a standard functional decomposition, similar to a functional ANOVA.  \n\nFor a model $f(X)$ that can be decomposed into main effects and interactions:\n\n$f(X) = \\sum_{i} f_i(X_i) + \\sum_{i < j} f_{i,j}(X_i, X_j) + \\dots,$\n\nthe total contribution of a feature $X_k $ at a point $X$ is the sum of all components that include $X_k$. We will clarify this in the final version.\n\nFor example, in our interaction model, the contributions are:\n\n$\n\\text{Contribution of } X_1 = \\text{Contribution of } X_2 = 3 \\sqrt{3} \\, X_1 X_2.\n$\n\nHowever, as we are not interested in directionality, we take the absolute value of the contributions for computing TP and FP.\n\n---\n\n> Q: On the LOCO Model Fit within a Specific Cluster\n\n**A**: As noted in Figure 6 (where we provide details), once the regions are identified, a practitioner can apply any global importance method of choice to characterise them. In our experiments, the different approaches perform similarly. We plan to move Figure 6 into the main text and highlight this part more clearly. \n\nRegarding the proof of the theorem, while we presented it using regional estimators, we also noted that the version using global minimisers can be easily derived. In fact, using the global minimiser $\\( f_{-j}(X) = \\mathbb{E}[f(X) \\mid X_{-j}] \\)$ still satisfies the proof’s conditions, which requires that the estimator $f_{-j}(X)$ does not depend on variables outside the corresponding linear model used for the observation $X$. This condition is met whenever $X_j$ is locally important: removing this variable in the condition does not alter the switching variable, so both $f(x)$ and $f_{-j}(X)$ evaluate to the same region-specific affine function. We'll revise the explanation for clarity.\n\n---\n\n> Q: On the Counter-Example\n\n**A**: We thank the reviewer for this excellent and insightful counter-example. The original counter-example uses the model:\n\n$\nf(X_1, X_2) = X_2 \\cdot \\text{sgn}(X_1)\n$\n\nwith a squared-loss conformity score. This leads to an importance vector of: $(X_2^2, X_2^2)$ for all points, making the two regions $\\( X_1 < 0 \\)$ and $\\( X_1 \\ge 0 \\)$ impossible to separate. This example underscores a key point we discuss in the paper (L389-391): the feature importance representation used for clustering is crucial, and motivates a promising direction for future work on enriching these representations to capture different facets of model behaviour.\n\nBefore addressing this issue, we would like to emphasise that even in this example, if R-LOCO were to find an incorrect cluster or return just one cluster, the model would still avoid attributing importance to unused and independent variables, in contrast to SHAP, which is the main point of our paper.\n\nNow, regarding the importance of $X_2$. The clustering algorithm does not “see” \\( X_1 \\) at all; it only sees a single value $\\( X_2^2 \\)$ that determines the point’s position in the importance space. Consequently, the algorithm partitions data based on $\\( X_2^2 \\)$, and may forms clusters like:\n\n- **Cluster 1 (Low Importance):** Points where $\\( X_2^2 \\)$ is small (i.e., $\\( X_2 \\)$ is close to 0).\n- **Cluster 2 (High Importance):** Points where $\\( X_2^2 \\)$ is large (i.e., $\\( X_2 \\)$ is close to -1 or 1).\n\nThis is still a valid description of the model’s behaviour with respect to $\\( X_2 \\)$, as it reveals regions where the magnitude of importance is homogeneous.\n\n---\n\n#### **Adressing the Core Issue:** Enhanced Importance Vectors for Clustering\n\nThe issue arises because squared loss is sign-agnostic: squaring discards the direction of the effect, which is crucial in distinguishing the two regions in the counter-example.  \n\nAs said in (L389-391), we could, for example, enrich the feature importance vectors to improve the clustering. In those lines, we suggested incorporating higher-order LOCO (e.g., removing subsets of features to compute LOCO). However, in this specific case, a more informative dimension would be the **signed effect**:\n\n$\n\\delta_j(X) = f(X) - f_{-j}(X)\n$\n\nIn the counter-example, this yields:\n\n$\n\\delta_1^b(X) = \\delta_2^b(X) = f(X)\n$\n\nSo the full feature importance vector becomes $\\gamma$:\n\n$$\n\\gamma(X) = (f(X)^2, f(X)^2, f(X), f(X))\n$$\n\n- For $\\( X_1 < 0 \\): f(X) = -X_2  → (X_2^2, X_2^2, -X_2, -X_2) $\n- For $\\( X_1 \\ge 0 \\): f(X) = X_2  → (X_2^2, X_2^2, X_2, X_2) $\n\n\nIn this signed importance space, the two regions are now distinct and separable. For $\\( X_2 \\neq 0 \\)$, points in $\\( X_1 < 0 \\)$ and $\\( X_1 \\ge 0 \\)$ are mapped to opposite sides of the origin, allowing a standard clustering algorithm (e.g., K-Means) to easily distinguish the two groups.\n\n---\n\n> Q: On Citation of [1]\n\n**A**: We thank the reviewer for bringing the Effector package [1] to our attention. This is indeed relevant prior work, and we will add an appropriate citation and discussion in the revised manuscript."}]

Author rebuttal:

Meta-review: This submission first identifies failures of two popular feature attribution methods, Local Shapley Values (L-SV) and LIME, in finding locally important features. Motivated by these failures, it then proposes R-LOCO, a hybrid local/global explanation method that partitions the feature space into regions with homogeneous feature importance characteristics and then applies a global attribution method to each region.

Strengths identified by the reviewers are as follows:
- Valuable identification of failures of L-SV and LIME, for even a piecewise-linear model.
- R-LOCO bridges local and global feature importance, providing an interesting perspective on the former by drawing on methods for the latter.
- R-LOCO demonstrates higher fidelity than L-SV, LIME, etc. on real data.
- The paper is well-written and easy to read.

Weaknesses:
- Reviewers were concerned that the regions produced by R-LOCO may be complex and uninterpretable, hindering overall interpretability. While reviewers mostly accepted the argument in the author rebuttal that the complexity of regions is a problem for future work, they feel (and I support them as area chair) that the issue should be discussed more in the current paper. Specifically:
    - The number of clusters appears to be the main way to control region complexity.
    - In their rebuttal to Reviewer nEVM, the authors showed a decision tree fit to approximate the clusters/regions. This tree as well as additional trees should be included in the paper along with the accuracies of approximation, perhaps in a trade-off with the number of clusters.
    - The role of the feature importance space (as shown in the rebuttal to Reviewer v4Qd's counterexample) should also be discussed.
- Computational cost: R-LOCO requires fitting $O(p)$ models where $p$ is the number of features. The author rebuttal shows that this training cost is mitigated because inference is fast, having only to assign points to clusters. The rebuttal (to a different point, see below) also suggests that one only needs to fit $p$ models using the whole dataset and not for each cluster, which would further mitigate the concern.
- There was confusion about important points such as the definition of "local feature importance" and whether feature importance is estimated using regional (per cluster) or global estimators (using the whole dataset), among other clarifications and corrections. The rebuttal seems to have addressed these and the clarifications should of course be added to the paper. The region vs. global estimator question is particularly important for the point about computational cost.

All reviewers became positive toward the submission during the discussion period and agreed that it should be accepted. I support this consensus. At the same time, I emphasize that this consensus is dependent on the resolution of some concerns by the rebuttal and the corresponding recommended changes, as summarized above.
Decision: Accept (poster)