Title: OrdShap: Feature Position Importance for Sequential Black-Box Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Interpretability

Explainability

XAI

Shapley
Abstract: Sequential deep learning models excel in domains with temporal or sequential dependencies, but their complexity necessitates post-hoc feature attribution methods for understanding their predictions. While existing techniques quantify feature importance, they inherently assume fixed feature ordering — conflating the effects of (1) feature values and (2) their positions within input sequences. To address this gap, we introduce OrdShap, a novel attribution method that disentangles these effects by quantifying how a model's predictions change in response to permuting feature position. We establish a game-theoretic connection between OrdShap and Sanchez-Bergantiños values, providing a theoretically grounded approach to position-sensitive attribution. Empirical results from health, natural language, and synthetic datasets highlight OrdShap's effectiveness in capturing feature value and feature position attributions, and provide deeper insight into model behavior.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Shapley-value-based feature attribution method designed to explain the importance of a feature's position within sequential data. To address the high computational cost of Shapley value estimation, the authors propose a least-squares approximation technique. Experimental results using inclusion/exclusion evaluation demonstrate that the proposed method, OrdShap-VI, produces more faithful explanations compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* The paper addresses a critical limitation in existing feature attribution methods for deep learning models. Most current approaches overlook the role of a feature’s sequential position and focus solely on perturbing feature values. By explicitly modeling positional importance, the proposed method fills a notable gap in the interpretability literature for sequential data.\n\n* The methodology is grounded in a solid theoretical framework, leveraging Shapley values and least-squares approximation to ensure both theoretical soundness and computational feasibility.\n\nWeaknesses:\n\nThe primary concern lies in the evaluation strategy of the proposed method.\n\n* Although the method is designed for sequential data, the experiments are limited to a narrow set of domains. While the EHR dataset is sequential in nature, it belongs to a specific subdomain (medical records). The evaluation would be more convincing if the method were tested on standard time-series benchmarks from broader application areas, such as forecasting or regression tasks in finance, weather domains.\n\n* The NLP evaluation is conducted on the IMDB sentiment classification dataset, which is a relatively simple binary classification task. It would be valuable to assess the scalability and robustness of the method in more complex settings—e.g., multi-class classification with thousands of labels, or tasks involving long-context inputs such as those seen in modern large language models (e.g., inputs exceeding 100k tokens). It remains unclear whether the method maintains its sensitivity to positional importance when the input length becomes very large or when the task complexity increases.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-15"}

comments: SUMMARY:
OrdShap, a feature attribution method designed specifically for sequential black-box models, is introduced in this work. It's motivated by observation that existing works like SHAP conflate the importance of a feature's value with the importance of its position within a sequence. A SHAP extension is proposed linking feature-and-position attribution to Sanchez-Bergantinos values, and permutes features in subsets of values. Attributions of value-importance and position-importance are summary metrics from OrdShape. Experiments are done on synthetic data, EHR data, and IMDB to demonstrate the approach.

STRENGTHS AND WEAKNESSES:
* The connection to Sanchez-Bergantinos values to XAI is novel and provides theoretical grounding for the approach. 
* The paper overall is well-written, motivated and provides good theoretical and empirical proofpoints.
* Even though acknowledged, the computational cost is absolutely immense: d! * 2^d. LSq-approx is better, but still requires sampling K subsets and L permutations, making it significantly more expensive than KernelSHAP. This also as a result limits the applicability to long sequences (e.g., thousands of doc tokens).
* OrdShap-PI is ultimately just a single linear coefficient. It's likely ultimately an oversimplication and connects weakly to the non-parametric, game-theoretic concept from the theoretical motivation.

QUESTIONS:
* How many permutations were used for experiments in Tab. 2?
* Is there an empirical analysis on the sensitive of the approach to choices of K and L for approximation?
* Is this primarily memory or time bound setup -- what is the maximum sequence length d that's practical?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops OrdShap, a method for attributing predictions of sequential models to both the value and the position of each input feature. Existing attribution approaches typically assume a fixed feature ordering, which makes it difficult to separate the attribution of a feature's value from its position in the sequence. OrdShap addresses this by quantifying how predictions change when the positions of features are permuted.\nThe method incorporates permutations with the familiar ablations for Shapley values, which the authors connect to Sanchez-Bergantiños values from game theory. It produces two scores per feature: one for its value (OrdShap-VI) and one for its position (OrdShap-PI). The paper also proposes efficient approximation algorithms for computing these scores, using random draws of permutations and feature subsets, and a least squares formulation to calculate attributions for all features at once. Experiments on healthcare, language, and synthetic datasets show that OrdShap can distinguish between value and position effects in model predictions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is well written and clear. It studies an important problem, and while I am not an expert on attribution methods, it seems like the problem has not been properly addressed before.\n\nWeaknesses:\n\nI think that some motivation for the necessity of the specific permutation-based method would've been beneficial. I will clarify what I mean by that under the \"questions\" part of the review.\nAnother small comment is that the evaluation of the order attribution is not entirely clear to me. While the evaluation with Inclusion and Exclusion AUC is pretty intuitive and common for the value attribution, I think the evaluation with permutations at section 6.1 should be better motivated and explained. There is no scalar metric compared between ordshap and and the baselines, and it is also unclear how one should compare it with methods that do not attribute the order at all.\n\nQUESTIONS:\nTo expand on my comment from \"weaknesses\":\n\n1) Many models use inputs such as positional encodings to denote a feature's place in the sequence. Why is a simple baseline that adds an input with the position of the feature and runs shap on this extended set of {features, positional inputs} not applicable to the problem? If it is applicable, should the paper be comparing to this?\n2) While order of features is an important aspect, isn't it that in most applications, such as the medical example given in the paper, the timing of the feature is important and not just its order? That is, we would have a tuple (t, x) with the time of a medical event/token, and its value, and what matters is not only if a medication was given before/after some token like a lab test, but how much after/before. The paper mentions irregular intervals, says the method can be extended, and points to the appendix. But from the appendix I did not really understand how this extension to such attributions is performed and what properties it has.\n\nSince I am unfamiliar with the literature on methods like TIME and PoSHAP, I would be glad to learn how these works and the current one under review address these questions.\n\nSome smaller comments:\n1) Figure 1 is not very easy to understand for a reader that starts reading the paper. It is unclear why the raised predicted probability is a desired outcome and what is it an alternative to. A figure with a clearer story may be helpful for many readers that go through the intro.\n2) Section 2 says \"While these attributions methods can be applied to sequential models, most assume feature independence\". What does \"independence\" mean here? Is it statistical independence, or some kind of other independence? I guess that KernelSHAP has some independence assumptions, but I was not aware that most shapley-based baselines make statistical independence assumptions.\n3) According to theorem 1, OrdShap-VI has the desirable properties that are common in Shap based methods, does OrdShap-PI have any such guarantees as well?\n\nLIMITATIONS:\nThe authors have appropriately discussed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nin this paper the authors introduce OrdShap, a novel feature attribution method designed for sequential deep learning models - these often require understanding of both feature values and their positions within input sequences and OrdShap proposes a novel approach to tackle this. Traditional attribution methods assume a fixed feature order, conflating the effects of feature values and their sequence positions. OrdShap addresses this problem by quantifying how model predictions change when feature positions are permuted, thus disentangling the effects of feature values and positions which ultimately shows superior performance. The method is grounded in game theory, specifically relating to Sanchez-Bergantinos values (which the authors do a good job of introducing and motivating) and provides a structured representation of feature importance based on position. Empirical evaluations on health, natural language, and synthetic datasets demonstrate that the proposed method, OrdShap, is effective in capturing both feature value and position attributions, offering deeper insights into model behavior. The paper also proposes efficient algorithms to approximate OrdShap, addressing computational challenges inherent in Shapley-based methods. The experimental evaluations show benefits in terms of both interpretability as well as superior performance to several benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written and the motivation is clear. The authors give a nice introduction and the technical preliminaries section is well written. One thing I really like about this work is that authors are very pedagogical about their approach and explanations. There is consistently a rigorous theoretical explanation and alongside it a intuitive explanation or a motivating example showcasing what it means. The sanchez and Bergantinos values are also well motivated and if it is the case that they have not been investigated in XAI literature (in which I am not well versed but could not find many works online combining the two), I think the idea is neat and clear. \n\nRegarding the introductory example of predicting LOS, could the authors give an example of the \"medical tokens\" and how does it look in practice, maybe in appendix in a small section? Although I really appreciate the figure 2 and it made the example much clearer, I think this would be the cherry on top that would make the example perfectly clear. For many of us who have been working with LLMs the word token and its meaning is now a bit contrived, I think this would possibly clear any remaining confusion that the reader might have. \n\nI would once again like to emphasize my satisfaction with the authors' ability to straightforwardedly explain the concepts, as I found the Table 1 and the example given with Bag, Gloves and Hats very intuitive and nice to motivate the need for using OrdShap. \n\nAlthough the method is clear, the question of computational expense naturally arises but the authors did a good job of both presenting it as the technical limitations of the paper and did a good job in producing sampling and least-squares approximations. I am not too sure if the least-squares approximation can give any downsides and what are its weaknesses, and I would appreciate if the authors could comment a bit more on this but in general I was satisfied with both approaches that authors proposed (5.1 and 5.2). Regarding experiments, I was convinced with the benefit of using OrdShap and was happy that authors gave execution time results as well as presented their details so detailedly. However, looking at Table 7 in the supplementary material I was a bit surprised with how much slower the OrdShap methods are and I think that the authors should have highlighted this in the main paper as it is extremely slower. Although it is mentioned in the limitations and conclusion section, the authors a bit misleadingly mention that they have mitigated this limitation through sampling and approximation. Also, I think that the significance results should be reported in the main paper and not the supplementary material - especially as (unless I am mistaken), most of the results for OrdShap-VI in the e.g., Inclusion AUC are not statistically significantly different to the performance of second and third best performing method (e.g., IG and KS). \n\nOverall I think it is a well written paper that contributes to XAI with clear motivation and applications but there are few weaknesses that I think the authors should address and comment upon. I think that the authors open a nice alleyway for algorithms that can disentangle value and position importance of each feature in a sequence and introducing the SB values is definitely something that can be important/useful for XAI, but after examining the appendix and the statistical (in)significance of some results that come at an expense of much longer computation and execution times, I was reluctant to give a larger score.\n\nQUESTIONS:\nPlease see strengths/weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}