Title: Probabilistic Stability Guarantees for Feature Attributions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Explainability

Interpretability

Feature Attributions

Safety

Robustness
Abstract: Stability guarantees have emerged as a principled way to evaluate feature attributions, but existing certification methods rely on heavily smoothed classifiers and often produce conservative guarantees. To address these limitations, we introduce soft stability and propose a simple, model-agnostic, sample-efficient stability certification algorithm (SCA) that yields non-trivial and interpretable guarantees for any attribution method. Moreover, we show that mild smoothing achieves a more favorable trade-off between accuracy and stability, avoiding the aggressive compromises made in prior certification methods. To explain this behavior, we use Boolean function analysis to derive a novel characterization of stability under smoothing. We evaluate SCA on vision and language tasks and demonstrate the effectiveness of soft stability in measuring the robustness of explanation methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a notion of soft-stability for explanations: Given an explanation and a classifier prediction from masking features not included in the explanation, this captures the *probability* that the classifiers predictions are unchanged when masking using the set of explanations which include up to radius-$r$ more features than the original (where $r$ is a hyperparameter). This is an improvement over previously used hard-stability, which strictly labels an explanation as stable or unstable at a given radius. The authors demonstrate both qualitatively and quantitatively how their relaxed version of stability yields improvements over hard-stability. They propose an estimator for both hard and soft stability, for which they prove formal guarantees. Finally, the authors link stability and smoothing: smoother classifiers enjoying better worst-case stability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is well written and visually appealing, with clear (and well-formatted) theorems and figures. \n* The problem of providing a relaxed version of hard stability is important for obtaining a more nuanced understanding of trusting explanations.\n* The estimator yields guarantees now for the classifier $f$ instead of smoothed $\\hat{f}$, as in MuS. \n* The experimental results are convincing\n  * The SCA bound is indeed non-trivial at larger radii, whereas the MuS bound is. \n  * Smoothing does seem to generally increase stability\n\n**Weaknesses**\n\n* You introduce explanations of feature $i$ as $\\alpha_i \\in \\mathbb{R}$ and hence total explanations as $\\alpha \\in \\mathbb{R}^N$ in section 2.1. However, you later move analysis to $\\alpha \\in \\{0, 1\\}^n$. You explain in line 66-67 that explanations can be converted into binary vectors via selecting the top-k. Two comments on this\n  * (Minor) You never explicitly state that this is indeed what you are doing for the rest of the paper. Although that may be obvious to some, it would be beneficial to do so.\n  * Such a step requires selecting some $k$ to use, but it is not made clear what $k$ is chosen or how. I can see such a choice being consequential.  For example, I imagine larger $k$ correlates with increased stability at all radii.\n* There are two missing citations that would be beneficial for the paper\n  * [1] This is a highly cited work on smoothing in interpretability, proposing the use of gaussian noise to smooth explanations. You mention such a practice in contrast to your discrete smoothing, and adding the citation there or in your related works section seems natural to me.\n  * [2] This work has many similarities to yours. They propose *explainer astuteness* which takes in a radius $r$ and threshold $\\lambda$ and returns the probability that explanations of points $x’$ in the $r$-ball around the point of interest $x$ are within $\\lambda d(x,x’)$ of explanation for $x$. As a probability, explainer astuteness takes values from $0$ to $1$ and is thus a *soft measure of stability*. The authors also provide a link between model smoothness and their measure of stability: lower bounding explainer astuteness with the probabilistic lipschitzness [3] of the model: a relaxed version of local lipschitz. \n\n\n[1] Smilkov, Daniel, et al. \"Smoothgrad: removing noise by adding noise.\" arXiv preprint arXiv:1706.03825 (2017).\n\n[2] Khan, Zulqarnain Q., et al. \"Analyzing explainer robustness via probabilistic lipschitzness of prediction functions.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\n[3] Mangal, Ravi, et al. \"Probabilistic lipschitz analysis of neural networks.\" International Static Analysis Symposium. Cham: Springer International Publishing, 2020.\n\nQUESTIONS:\nThe first two questions are related to the weaknesses above\n\n* Am I correct in my interpretation above that you are selecting some $k$ to go from real valued to binary explanations? If not, what are you doing to get the binary explanations? If you are: How do you select $k$? Is it a function of $r$, or chosen independently? \n* While I mention many similarities with [1] I believe that your work still has novelty. Can you explain the similarities and differences with [1]? What does your method bring to the table that [1] does not? One difference is that the astuteness measure in [1] uses the $r$-ball around a point as opposed to your discrete masking. However major similarities, such as both providing a soft stability measure, remain. Would the method in [1] and this method provide similar results at similar radii? I.e., would their stability rankings for different methods agree? To this end, a variant of Figure 6 including the explainer astuteness measure from [1] could be valuable. \n* In line 177 you define $\\gamma$ as the “distance to the decision boundary”. Is this the distance of $x$ to the boundary? Or is it related to the distance of $x$ masked with $\\alpha$ or $\\alpha’$ to the boundary?\n\nBased on the answers to these questions, I will consider raising my score by 1 point. \n\n[1] Khan, Zulqarnain Q., et al. \"Analyzing explainer robustness via probabilistic lipschitzness of prediction functions.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2024.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

comments: SUMMARY:
The paper presents a discrete probabilistic‐verification framework that offers robustness guarantees for feature attributions. By considering soft stability guarantees, the method yields tighter bounds than the hard guarantees produced by multiplicative smoothing. The results show higher stability rates across several attribution techniques.

STRENGTHS AND WEAKNESSES:
### Strengths
- The paper is well written, the claims, theorems and propositions are sufficiently proved.
- The unified framework for certifying both hard and soft stability guarantees is interesting and improve on previous work.
- Enhancing probabilistic verification techniques for explainability is important to strengthen safety assurances in safety-critical systems.

### Weaknesses
- The paper offers little to no discussion on how stabilizing feature attributions might improve adversarial robustness.
- Related works in discrete randomized smoothing, e.g. [1], [2], has not been considered for comparison and/or evaluation.

[1] Hierarchical randomized smoothing
[2] Localized randomized smoothing for collective robustness certification

QUESTIONS:
- Within the results of Figure 9, MuS with $\lambda = 0.125$ yields better results than SCA for the RoBERTa model, why is this the case? How would it perform with $\lambda = 0.1$? 
- Would it be possible to expand the results to a couple of other datasets both in vision and in language?
- Would it be possible to expand the results to include an evaluation against adversarial attacks?

LIMITATIONS:
It is not clear if for lower $\lambda$ values MuS is equal to SCA.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nRecognizing the limitations of existing stability guarantees, in this manuscript, the author(s) proposed a stability certification algorithm based on soft stability, extending prior work to the probabilistic setting. And through empirical experiments, the author(s) demonstrated the effectiveness of the proposed method. The manuscript was mainly motivated by the central idea of additive features.\n\n---\n\nSTRENGTHS AND WEAKNESSES:\nIn my opinion, the strengths and weaknesses of this manuscript are as follows:\n\n## Strengths:\n\n1) The proposed algorithm introduces a soft stability-based certification framework, extending prior deterministic stability guarantees to a probabilistic setting. \n2) The algorithm offers stronger and more flexible stability guarantees compared to existing algorithms.\n3) The author(s) implemented empirical experiments to support the effectiveness of proposed algorithm.\n\n---\n\n## Weaknesses:\n\n1) While the author(s) claimed that using experiments validated the finds, the absence of accompanying code will diminish the credibility of the results and makes reproducibility more difficult.\n\n2) Some experimental details/explanations are not clear; for example, how to determine the value of $\\lambda$. For more details, please see Section ***Questions***\n\n3) The manuscript contains some formatting inconsistencies and issues.\n\n\n---\n\nQUESTIONS:\nThe manuscript looks interesting, theoretical results are mathematically derived and empirical experiments are performed to verify these results. After reviewing this manuscript, I have the following questions/comments, I am looking forward to the responses from the author(s). Thanks. \n\n1. In Figure 2, the stability analysis is presented only for correctly classified instances. It would be valuable to understand how the hard and soft stability metrics behave on misclassified samples. Are there any interesting results when the algorithm is applied to incorrectly classified images? If such experiments were conducted, it would be great if the author(s) could include or discuss them, as they could provide further insight into the algorithm’s robustness and limitations.\n\n2. In Line 76, the author(s) mentioned that two similar attributions yield the same prediction. And in the manuscript, the author(s) gave two ways including overlapping and subset/superset (i.e., additive perturbations in the manuscript) feature sets. For comparison, could you please illustrate any other ways (If there are) and cite the related work?\n\n3. In Figure 3, the notion of similarity is central to the discussion. Could the author(s) clarify how similarity is defined? Just from the observation, or other actual metrics? Also, even if there is much overlapping, it will not be significant (maybe the cat nose patch is more important than others) if the overlapping are not key features, then it will make the stability very different also be reasonable and obvious.\n\n4. In Line 221, about the value of $\\lambda$, could we use the theoretical results obtained in the manuscript explain/guide it?\n\n5. In Figure 5, could you please explain the cross point and the changing among yellow and green in the 3rd subfigure, yellow and red in the 4th subfigure? And why is there a sudden drop in the last subfigure? Understanding these patterns might help clarify the behavior and limitations of the proposed method under varying conditions.\n\n6. Some other tiny issues:\n\n(1) Please avoid repeatedly spelling out terms after their abbreviations have been introduced, such as \"Stability Certification Algorithm (SCA) \" is fully written multiple times in Lines 48, 108, 131, … after the abbreviation is introduced.\n\n(2) Ensure consistency in reference formatting. There are inconsistencies such as, refs. [59] vs. [60], refs. [62] vs. [64], etc.\n\n---\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles certifying the robustness of feature-attribution explanations when additional features are added. It introduces soft stability, a probabilistic metric quantifying the share of perturbations that leave the prediction unchanged, and presents the Stability Certification Algorithm (SCA), a simple, model-agnostic sampling method that statistically guarantees both soft and hard stability. Leveraging new Boolean analysis, the authors prove that mild random-mask smoothing improves stability without accuracy loss, and confirm these insights on vision and language models with multiple attribution methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n- The paper is very well-written and easy to follow.\n- SCA is simple and model-agnostic.\n- The authors show that SCA can provably certify soft and hard stability.\n- The authors show that *mild* smoothing can boost stability without the severe accuracy loss of prior certifiers, a well-known usability bottleneck.\n- The authors provide a Boolean-analysis result that supplies a principled explanation for why mild smoothing helps, moving beyond purely empirical claims.\n- Experiments reportedly span both vision and NLP tasks, suggesting breadth.\n\n## Weaknesses\n\nThe main weakness is that the paper's evaluation leaves several gaps that weaken its practical impact. \n\n### No human validation\n\nThe paper does not test whether the proposed stability score aligns with human judgements of explanation quality, leaving practical relevance unverified. I do not expect authors to address the human validation during the rebuttal, but justification for omitting it would strengthen the paper.\n\n### Incomplete baseline coverage\nThe study omits empirical comparisons with other evaluations for feature attribution methods, e.g., [1 - 4]. Again, I do not expect authors to do this comparison during the rebuttal, but it would be nice to explicitly tell why these baselines are beyond the scope of the paper.\n\n### Missing runtime evidence\n\nSCA requires $O\\left(1 / \\varepsilon^2\\right)$ samples (Theorem 3.1). For tight tolerances, this could be prohibitively slow, but the authors report no wall-clock time, FLOP count, or memory usage for SCA and other baselines.\n\n### No mask-time sensitivity study\n\nAll experiments fix the explanation to the top 25 \\% of features, yet attribution methods differ widely in sparsity. Without varying this threshold, we cannot know whether SCA still beats MuS at 10 \\% or 40 \\%, nor can practitioners learn how to pick $k$ for their own tasks.\n\n[1] Hooker, Sara, et al. \"A benchmark for interpretability methods in deep neural networks.\" Advances in neural information processing systems 32 (2019).\n\n[2] Fel, Thomas, et al. \"How good is your explanation? algorithmic stability measures to assess the quality of explanations for deep neural networks.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.\n\n[3] Salih, Ahmed, et al. \"Investigating explainable artificial intelligence for mri-based classification of dementia: a new stability criterion for explainable methods.\" 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2022.\n\n[4] Faber, Lukas, Amin K. Moghaddam, and Roger Wattenhofer. \"When comparing to ground truth is wrong: On evaluating gnn explanation methods.\" Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2021.\n\nQUESTIONS:\n- Why restrict to additive, uniformly sampled supersets $\\Delta \\_r$? Could SCA be extended to certify robustness to feature removals or structured perturbations (e.g., contiguous text spans or image patches)?\n\n- Practitioners must pick $\\varepsilon, \\delta$ (to decide how many explanations to draw), $r$, and possibly $\\lambda$. Do you have heuristics or default settings that balance certificate strength, runtime, and accuracy loss?\n\nLIMITATIONS:\nAlthough the limitations are not crucial, it is important that authors mention them:\n\n- SCA needs $O\\left(1 / \\varepsilon^2\\right)$ forward passes per input; for small $\\varepsilon$ the wall-time and memory could be prohibitive\n\n- Stability is certified only for additive feature inclusions drawn uniformly. Real edits often involve removals or structured, correlated changes, leaving those scenarios uncertified.\n\n- Certificates are framed by ( $\\varepsilon, \\delta$ ) while any non-zero failure probability may be unacceptable in safety-critical settings demanding zero risk.\n\n- The theory assumes a computable distance to the decision boundary $\\gamma$ (eq. 5) while estimating even loose lower bounds is non-trivial for deep networks, so the tightness of the bounds is uncertain.\n\n- The method certifies stability only for a single input-explanation pair. It offers no distribution-level or global guarantee akin to Lipschitz bounds.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}