Title: Regional Explanations: Bridging Local and Global Variable Importance
Venue: NeurIPS 2025
Accepted: yes
Keywords: variable importance

shapley values

lime

feature importance

explainability

interpretability

leave-one-out importance
Abstract: We analyze two widely used local attribution methods, Local Shapley Values and LIME, which aim to quantify the contribution of a feature value $x_i$ to a specific prediction $f(x_1, \dots, x_p)$.  Despite their widespread use, we identify fundamental limitations in their ability to reliably detect locally important features, even under ideal conditions with exact computations and independent features. We argue that a sound local attribution method should not assign importance to features that neither influence the model output (e.g., features with zero coefficients in a linear model) nor exhibit statistical dependence with functionality-relevant features. We demonstrate that both Local SV and LIME violate this fundamental principle. To address this, we propose R-LOCO (Regional Leave Out COvariates), which bridges the gap between local and global explanations and provides more accurate attributions. R-LOCO segments the input space into regions with similar feature importance characteristics. It then applies global attribution methods within these regions, deriving an instance's feature contributions from its regional membership. This approach delivers more faithful local attributions while avoiding local explanation instability and preserving instance-specific detail often lost in global methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors first discuss a major limitation of the vanilla-SHAP local explanation method : when the model is piece-wise linear over regions $A_1, A_2, \\ldots$, the local importance of a feature in $A_k$ can be polluted by model behavior in other regions, leading to counterintuitive interpretations.\n\nTo address this limitation, it is proposed to first compute a point-wise loss increase $\\Delta_j(X, Y)$ when removing feature $j$ from the dataset, cluster these local Importance vectors in $C$ groups, and then provide local feature importance to new points based on which cluster it is closest to.\n\nThe method is assessed on toy examples and 2 tabular datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is well written and a breeze to read from start to finish.\n- The technical sections 2, 3, and 4 are very clear.\n- The limitation of SHAP highlighted by **Theorem 2.2** is an important contribution to the community.\n- The experiments on toy data and models highlight the benefits of reporting regional explanations instead of fully local or global.\n\n## Weaknesses\n- Regional explainability is an emerging subject and there are existing methods that the current paper is not citing. For instance, the Effector package [1] (and the methods within) are not cited. Since [1] is still a preprint, I cannot be too severe on its exclusion in the experiments. Adding a citation + discussion would \nbe sufficient.\n\n- The notion of \"local feature importance\" is not consistent within the paper. For instance, at line 123, it is stated that the feature local importance of a piece-wise linear model is the coefficient used in the region. At line 277, it is instead stated that the \"product of feature values and their corresponding coefficients\" is the local importance. These are two different importance measures and the correct one should be clarified directly after definition 2.1.\n\n- Although I think the weakness of SHAP highlighted by **Theorem 2.2** motivates well the need for a regional explanation method, I do not think it motivates R-LOCO especially well. To explain why, I refer to the **Question** section below where I present a toy example where R-LOCO fails to recover ground-truth regions on a piece-wise linear model, and ask the authors to respond to this counter-example. This counter-example puts into question the emphasis on piece-wise linear models as a motivation for the approach.\n\n\n[1] Effector: A Python package for regional explanations. https://arxiv.org/pdf/2404.02629v1\n\nQUESTIONS:\n## Questions\n- In section 5.2, it is not clear how \"product of feature values and their corresponding coefficients\"\napplies to the non-piecewise linear models (i.e. 2nd order models with and without interactions).\nWhat measure of importance is used as ground truth for these two functions? This clarification is required for me to increase my score because the paper has a large emphasis on ground-truth evaluations.\n\n- The model $f_{-j}$ used in R-LOCO is fitted on the whole dataset minus feature $j$. However, the proof of **Theorem 4.1** requires that the model $f_{-j}$ is **refitted** only using data that lands in a specific cluster $C_k$. This assumption is not part of the premise of the theorem nor is it discussed in the main text. It is only discussed on the caption of Figure 6 from the appendix. I think the premise of the theorem should be rewritten to include this assumption. \n\n## Counter-example\nI do not think that the R-LOCO approach is well motivated by focusing on piece-wise linear models. To prove my point, I propose a toy example where R-LOCO seemingly fails. I would like the authors to address or repudiate it.\n\nLet $X_1,X_2\\sim U(-1, 1)$ and $f(X_1, X_2) = X_2 \\mathbb{1}(X_1\\geq 0) - X_2\\mathbb{1}(X_1< 0)$. From what I understand of the paper, an ideal regional explanation method should identify the two regions $\\mathbb{1}(X_1<0)$ and $\\mathbb{1}(X_1\\geq 0)$ and report the linear model coefficient in each region (or at least feature importance consistent with these coefficients). \nHowever, applying clustering in the feature importance space (instead of the input space) is not able to recover the two regions. Indeed, since the conditional expectation is the optimal predictor under the squared loss, we have\n- $f_{-2}(x) = \\mathbb{E}[f(X)|X_1=x_1] = 0$\n- $f_{-1}(x)=\\mathbb{E}[f(X)|X_2=x_2]=0$.\n\nThe corresponding LOCO effects using $Y=f(X)$ and the squared loss as the conformity score are\n\n- $\\Delta_1(X, f(X)) = (f(X)-f_{-1}(X))^2 = f(X)^2$\n- $\\Delta_2(X, f(X)) = (f(X)-f_{-2}(X))^2 = f(X)^2$\n\nNow consider the map $\\gamma: [-1, 1]^2 \\rightarrow \\mathbb{R}^2$ defined as $X\\mapsto (\\Delta_1(X, f(X)), \\Delta_2(X, f(X)))=(f(X)^2, f(X)^2)$. Since $f(X_1, X_2) = -f(-X_1, X_2)$, we have that $\\gamma(X_1, X_2) = \\gamma(-X_1, X_2)$. That is, the two regions $\\mathbb{1}(X_1<0)$ and $\\mathbb{1}(X_1\\geq 0)$ in input space completely overlap when mapped through $\\gamma$. Consequently, no clustering algorithm applied in the LOCO space could recover the ground-truth regions.\n\nNote that this counter-example is also applicable CSHAP (i.e. clustering on the L-SV). Therefore, this critique is not specific to R-LOCO but more to any clustering method that is applied on the feature importance space instead of the input space.\n\nLIMITATIONS:\nThe authors clearly highlight that the mathematical results all assume that the model is piece-wise linear. This assumption is not too restrictive seeing as it covers Tree-based methods (piece-wise constant) and ReLU networks.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a local importance method R-LOCO (Regional Leave Out Covariates), which tries to address limitations of SHAP and LIME. To improve the reliability of these methods, the authors first cluster the feature importance space, and then apply global importance methods to each cluster. The simulations and real data analysis shown the methods improved the previous local feature importance methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. The paper identified some limitations in local importance methods in SHAP and LIME.\n2. The paper proposes a method that combines clustering and LOCO to bridge local and global importance methods.\n3. In simulation and real-world datasets, R-LOCO outperformed SHAP and LIME in terms of both TP and FP.\n\nWeakness:\n\n1. The authors only considered the case that the population can be clustered. However, it’s unclear how R-LOCO performs when the population is homogenous. In such cases, I would expect clustering to be less effective. I think it could be a good sensitive analysis, particularly if its performance remains comparable to Global LOCO. This could help demonstrate R-LOCO’s robustness.\n2. The paper didn’t clearly state how the model determines which features are non-important. If the means are all non-zero as shown in Section 5.1, it’s confusing how are negatives identified. A clearer description of the decision rule would improve readability.\n3. The “Mean Performance Drop” in Figure 1 and Figure 2 is negative, I believe it should be positive instead when important features are perturbed. \n4. For better clarity, it’s better to at least mention the dimension of 2nd-order interaction model in Section 5.1. \n\nTypos and formatting:\n\n1. Missing period in line 377.\n2. Figure 1 and Figure 2 which are associated with Section 6 are placed on page 7, but Figure 3 associated with Section 5.4 is placed on page 8. I think it could be rearranged for better readability.\n\nQUESTIONS:\n1. In most of the literature, local importance typically refers to the importance assigned to a single prediction. In that sense, can R-LOCO be understood as calculating global importance for clusters?\n2. The choice of the 10th and 95th percentiles as bound for the non-important features attributions in Section 5.1 is unconventional. Are there any considerations for not using 5th and 95th percentiles?\n3. For the German credit datasets with 20 features, is there any reason only 19 features are shown in Figure 5 in Supplementary Material E?\n4. Could you provide some intuition for why clustering in the feature importance space works better than clustering in input space as demonstrated in Section 5. Specifically, what causes the reduced variance of R-LOCO in Section 5.4?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper under review tackles the timely and important problem of providing principled, interpretable and trustworthy feature attributions for predictions of black-box models. The authors argue that the widely used methods LIME and Local Shapley Values (LSV) can assign importance to features that do not locally influence the model's output, even when the features are independent. Inspired by the fact that global importance methods like Leave Out COvariates (LOCO) do not suffer from this problem, they propose to provide local feature importance in the form of global feature importance scores for a restriction of the predictor to a specific region of the input space. They exploit the fact that the LOCO scores for a feature is formulated as  an expectation over inputs of the gap in performance with respect to a best predictor that does not have a access to that feature. They explore different clustering methods to identify regions that enjoy low variance in performance gaps, effectively leading to sets of inputs for which the expectation is a good approximation. They first evaluate their approach in a controlled setting where the data generating processes are explicitly defined by pieces, providing evidence that their approach succeeds at retrieving more accurately the variables that effectively appear on each piece while less often identifying variables that do not, compared to LIME and LSV. They further use their approach on two real world datasets to compare its fidelity towards observed performance drops with two other regional approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written and proposes an interesting perspective on local feature importance by drawing inspiration from global feature importance. Experiments are interesting and well chosen to provide the reader with a good perspective on the challenges of feature attributions and the contributions of the paper.  \nIn a sense, the method can be described as identifying a classifier $\\kappa:\\mathcal{X}\\to [1,\\ldots,m]$ and estimate the global variable importance scores LOCO $\\Psi^k$ for the restrictions on the regions $A_k=\\{x\\in\\mathcal{X} : \\kappa(x)=k\\}$. The local variable importance for an input $x$ is then given by $\\Psi^{\\kappa(x)}$. While $\\Psi^k$ is well justified (at least in the case of independent features and in absence of interactions) for the predictor restricted on $A_k$, the method does not account for variable importance of the classifier $\\kappa$. In extreme cases (for a fine enough partition), this approach would result in a constant predictor on each region (and so $\\Psi^k=\\vec{0}$ for all $k$) while the classifier $\\kappa$ could be arbitrary complex. In this extreme case, R-LOCO would present null variable importance for all inputs, even though the predictor may be arbitrarily complex and actually rely on several specific variables, which can arguably be viewed as influencing the global model's output. I believe that the interpretability of the classifier $\\kappa$, and thereby the complexity of the partition, is a key component of such an approach, which is not discussed in the paper. It would appear to me that any approach which relies on partition of the input space needs to deal with a trade-off. While partitions with a large number of small regions will lead to smaller variance, they are also harder to interpret. \n\nMoreover, if a feature may not be important locally (like $X_6$ in the first order model), I would argue that it is an important feature when it comes to R-LOCO feature importance, since R-LOCO feature importance (say with true clusters) at input $X$ depends on the sign of $X_6$. I believe that in many applications this importance of $X_6$ cannot be overlooked. If the importance of the variables on which depends the clustering is viewed as a separate problem by the authors, I find that the paper lacks a discussion on how to reconcile this phenomenon with the local feature importance towards a global understanding of the model.\n\nRelated to the method and the evaluation, I find that some important information is missing (see questions) which makes it difficult to understand the advantages and limitations of the proposed approach.\nIn particular, it seems to me that the evaluation on real world datasets is focusing on global feature importance where as the proposed approach is specifically designed for local feature importance.\n\nQUESTIONS:\nCould you provide examples of the regions identified by R-LOCO using AffinityPropagation and include a discussion on the interpretability vs \"clustering quality\" trade-off? Such a discussion should be included in subsection 5.4 in my opinion.\n\nIn section 6, it seems to me that R-LOCO and the two other baselines provide feature importance on each identified region. How were these multiple feature importance views used to obtain a single variable ranking used in Figure 1 and 2? If a single ranking is indeed used, why not compare to LOCO in this evaluation?\n\nAs a follow up question, what would be the importance ranking provided by R-LOCO with true clusters for the first order model used in the paper? Since $X_6$ has no importance on both regions would it be ranked after $X_1,\\ldots, X_4$? If so, it seems that by chosing carefully the paramaters, the performance of $f_{-6}$ can be made arbitrarily low, which seems to me like an undesirable property.\n \nIn definition 2.1, is the whole function required to be continuous? While the name refer to $m$-continuity, no requirement is stated about this. Also this terminology ($m$-CPWL) is introduced, but no reference is made elsewhere in the paper if I'm not mistaken. What is the point of introducing this definition?\n\nIn the proof of Theorem 4.1, it is assumed that $\\hat{f}_{0},\\hat{f}_{j}$ are estimated for each region, which seems to contradict lines 191-192 that present them as estimators of population minimizers. Could you clarify this point? In your implementation, is the clustering done using local or global minimizers?\n\nLine 215, is the boundary set $\\hat{A}_{K+1}$ used in practice? What local attribution is computed in this case?\n\nLine 247, what criterion is used for clustering in input space?\n\nLine 281, It seems to me that this claim does not necessarily hold. Could you please elaborate on this or provide a reference for this fact?\n\nLines 277-278, could you provide formulas for the ground truth attributions you use, in particular for the model with interactions?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies flaws in commonly used local explanation methods: Local Shapley Values (L-SV) and LIME, even under idealized assumptions of independent features and exact computation. The authors prove via Theorem 2 that L-SV can assign nonzero importance to features outside the true active region of a piecewise-linear model, and similarly show LIME failures under the same setup. To remedy this, they propose R-LOCO (Regional Leave Out COVariates), which:\n\n1. Computes global LOCO importance for each observation.\n2. Embeds instances in the “feature-importance space” defined by their LOCO vectors.\n3. Clusters these embeddings into K sets (variants in table are based on different clustering schemes).\n4. Assigns each point the region’s averaged LOCO scores as its local explanation.\n\nThey prove in Theorem 4.1 that if a cluster lies entirely within a true region, R-LOCO recovers exact local attributions for piecewise-linear models. Through controlled simulations with ground-truth explanations (first and second-order additive and interaction models), R-LOCO achieves true-positive rates up to 97–98% with false-positive rates below 4%, outperforming L-SV and LIME (TP 30–66%, FP 34–70%). In real-world tests on the Diabetes and California datasets (using XGBoost), R-LOCO also yields higher fidelity: masking its top-k features drops model performance more and masking bottom-k features drops it less, compared to L-SV, LIME, CSHAP, and the FDT baseline.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality**\n\n*Strengths:*\n\n1. Theoretical rigor: Full proofs for Theorems 2.2 and 4.1 appear in Appendices A and B, with clear statement of assumptions (piecewise linearity, independence, clustering inclusion).\n2. Comprehensive experiments: Controlled benchmarks include 50 runs per setting, error quantiles for non-important features, and ablations over four clustering algorithms and hyperparameters (K = 2,4,8,20; damping = 0.5,0.6,0.9) demonstrating stability.\n3. Real-world validation: Utilizes public datasets (Diabetes, California), high-performing XGBoost models, and two separate fidelity metrics (top-k and bottom-k masking).\n\n*Weaknesses:*\n\n1. Clustering reliance: Performance depends heavily on cluster purity (Appendix C). It is unclear how the method performs in in more high-dimensional data where feature overlap is higher. I was unable to find if automated purity checks are utilized in the paper\n2. Compute cost: LOCO requires re-fitting models per region; no detailed runtime or memory analysis is given, limiting assessment of scalability to high-dimensional or large-scale data. \n3. Applicability: LIME and shap values are applicable across domains (extensions to large-scale vision datasets as well). Due to reliance on clustering, I am unsure of the proposed method's applicability.\n\n**Clarity**\n\n*Strengths:*\n\n1. Logical flow: Paper moves cleanly from theoretical critique (Sec. 2–4) to controlled experiments (Sec. 5) and real data (Sec. 6)\n2. Notation consistency: Feature sets, clusters, and Shapley/LOCO notation remain consistent throughout.\n\n*Weaknesses:*\n\n1. Dense appendices: Key implementation details (e.g., distance metrics, damping choices) are relegated to appendices, forcing readers to flip back and forth. Fig 6 (which is in appendix) should be in the main paper.\n2. Acronym overload: Readers must track R-LOCO, R-LOCOTC, R-LOCOIC, CSHAP, FDT consider consolidating or emphasizing defaults.\n\n**Significance**\n\n*Strengths:*\n\n1. Bridges local/global gap: R-LOCO offers a general recipe to turn any global explainer into a regional/local one, addressing a central challenge in XAI.\n2. Actionable insights: Demonstrates that clustering in importance space (vs. input space) yields markedly better fidelity, guiding future explainer design.\n\n*Weaknesses:*\n\n1. Domain scope: Focuses exclusively on tabular, piecewise-linear, or tree-based models; applicability to vision, NLP, or deep nets remains unclear.\n\n**Originality**\n\n*Strengths:*\n\n1. Novel theoretical critique: Per my awareness, the paper is the first to formally prove failures of L-SV under ideal conditions for piecewise models.\n\n*Weaknesses:*\n\n1. Building blocks: Clustering algorithms and LOCO are existing techniques; R-LOCO’s novelty lies in their combination rather than entirely new algorithms.\n\nQUESTIONS:\n*Scalability & Runtime*\n\n1. Issue: LOCO’s leave one covariate approach requires fitting p + 1 models per cluster (or approximations thereof).\n2. Request: Please report wall clock training and inference time (and memory usage) for R-LOCO vs. L-SV/LIME on the Diabetes and California datasets and discuss scaling behavior as p grows.\n\n*Automatic Region Validation*\n\n1. Issue: Cluster purity is critical for correctness (Theorem 4.1) but cluster-straddling degrades performance (App C).\n2. Request: Please integrate a purity metric (e.g., variance threshold on LOCO vectors) to automatically detect and refine impure clusters. The goal is to answer: under what quantitative purity threshold does performance sharply degrade?\n\n*High-Dimensional / Correlated Features*\n\n1. Issue: All controlled experiments assume independent features. As the authors acknowledge real datasets often exhibit strong correlations. I am unsure how much of this correlation is depicted in the 2 datasets considered.\n2. Request: Have you tested R-LOCO on synthetic correlated feature settings (e.g., Gaussian multivariate with block covariance) to gauge robustness? How do correlations affect clustering fidelity and TP/FP rates?\n\n*Choice of Distance Metric*\n\n1. Issue: Euclidean distance in LOCO space may underperform when features have different variance scales.\n2. Question: This maybe another issue for high dimensional data. Have you compared Mahalanobis or cosine distances for clustering? Could adaptive metric learning improve region coherence?\n\n*Applicability*\n\n1. Issue: It remains unclear how R-LOCO generalizes to unstructured data or non-additive model parts (e.g., CNN feature maps).\n2. Question: Are there specific use cases/applications that benefit from the computational overhead of the proposed method?\n\nLIMITATIONS:\nThe paper does not explicitly discuss limitations in the main paper. However, there is an answer regarding limitations in the checklist. I recommend adding one in the main paper that discusses:\n    • Clustering assumptions: Requirement that clusters align with true functional regions; sensitivity to hyperparameters.\n    • Computational overhead: Multiple model fits for LOCO and clustering steps.\n    • Model/domain scope: Currently validated on tabular, piecewise-linear or tree-based models; adaptation to deep networks or non-tabular data is future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}