OrdShap: Feature Position Importance for Sequential Black-Box Models
Authors: Davin Hill
Brian L. Hill
Aria Masoomi
Vijay S Nori
Robert E. Tillman
Jennifer Dy
Venue: NeurIPS 2025 poster ()
Keywords: Interpretability
Explainability
XAI
Shapley
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We propose a novel feature attribution method that disentangles attributions based on a feature's value and its position within a sequence.

Abstract: Sequential deep learning models excel in domains with temporal or sequential dependencies, but their complexity necessitates post-hoc feature attribution methods for understanding their predictions. While existing techniques quantify feature importance, they inherently assume fixed feature ordering — conflating the effects of (1) feature values and (2) their positions within input sequences. To address this gap, we introduce OrdShap, a novel attribution method that disentangles these effects by quantifying how a model's predictions change in response to permuting feature position. We establish a game-theoretic connection between OrdShap and Sanchez-Bergantiños values, providing a theoretically grounded approach to position-sensitive attribution. Empirical results from health, natural language, and synthetic datasets highlight OrdShap's effectiveness in capturing feature value and feature position attributions, and provide deeper insight into model behavior.

OrdShap: Feature Position Importance for Sequential Black-Box Models
Davin Hill" Northeastern University
Brian L. Hill" Age Bold
Aria Masoomi Northeastern University
Vijay S. Nori Optum AI
Robert E. Tillman Optum Al
Jennifer Dy Northeastern University
Abstract
Sequential deep learning models excel in domains with temporal sequential de- pendencies, but their complexity necessitates post-hoc feature attribution methods for understanding their predictions. While existing techniques quantify feature importance, they inherently assume fixed feature ordering - conflating the effects of (1) feature values and (2) their positions within input sequences. To address this gap, we introduce OrlShap, novel attribution method that disentangles these effects by quantifying how model's predictions change in response permut- ing feature position. We establish a game-theoretic connection between OrdShap and Sanchez- Berganttios values, providing a theoretically grounded approach position-sensitin attribution. Empirical results from health, natural language, and synthetic datasets highlight OrdShap's ss effectiveness in capturing feature value and feature position attributions, and provide deeper insight into model behavior.
Introduction
As complex and opaque deep learning mod- els are increasingly used in high-stakes ap- plications, is important to understand the factors that contribute to their predictions.
Trajectories days
Driginal Sequince
Pummatdd Sequanie LOS days
Parmated Sceverce LOS x days
Predicted Probabiltt
Feature attribution methods are widely used approach that seeks to quantify the sen- sitivity of model predictions to changes in individual input features, thus generating scores that represent that feature's s impor- tance or relevance [5, 89]. In particular, local methods generate separate scores for each given input sample; this helps users to understand individual model predictions rather than relying solely global attribu- tion scores. Many recent feature attribution methods have adapted the Shapley Value framework [65] from the field of coopera- tive game theory [43, 75]. Shapley-based methods have demonstrated their utility in wide range of domains [70, 28, 9] and sat- isfy a number of theoretical properties [43].
Histogram of Predictions
Redicod Prdo0att
0:39
Orignnl Sopanns
Denstly
Sequence Length
Figure Predicting hospital Length-of-See (LOS) 3 days for patient using sequence of medical tokens, representing tests and medications (App- A.2). Traditional feature attribution methods capture model sensitivity token values. However, we observe that permuting token order has significant effect on the predicted risk over time (Left) and final prediction (Right), even when token values are unchanged.
Corresponding author: Email: hill.davi @ @northeastem.edu Work done while at Optum AI
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A) Input
(B) Evaluate
(C) Summarize
Tokenized Sequence
Sample Sample
importance OroShapkVi
Importance OrdShap=ff
Position
1
2
3
Tott Emiprratu
Medt Acatuminapee
Chobotend
Figure 2: Overview of OrdShap on a medical example. (A) OrdShap takes a sequence of features (i.e. tokens) as input, then (B) evaluates the black-box model while sampling different token permutations (i.e. reordering tokens) and subsets. (C) We summarize attributions due to token value and token position using OrdShap-VI and OrdShap-PI, respectively. Positive OrdShap-Vi indicates that the presence of the token in the sequence increases the probability of LOS2 3. Positive OrdShap-Pl indicates that probability of LOS2 3 increases when the token appears later in the sequence; high magnitude indicates high model sensitivity to token position.
In this work we focus sequential data, where the samples consist of sequence of features, such as text, time-series, or genetic sequencing. Deep learning models such as Transformers [80] and Recurrent Neural Networks [62] have been shown to be highly effective sequential data [83] and are able to capture the sequential dependency between data features. These models make predictions based not only the specific feature values in each sample, but also the order in which they occur.
While existing feature attribution methods can be applied to sequential models, they inherently assume fixed feature ordering. In particular, most approaches generally derive attributions by measuring the sensitivity of predictions to small changes in feature values, and otherwise disregard the effeets of feature order [26]. However, permuting feature order, while leaving feature values unchanged, can have a significant impact on the model's 's prediction. For example, when predicting hospital length-of-stt -f-y from longitudinal Electronic Health Record (EHR) data' [59, 41, 55, 27], simply permuting the timestep of a test result can inflate the predicted risk even when the test result values unchanged (Fig. 1), resulting in widely varying predictions. Since existing feature attribution methods assume a fixed feature ordering, they effectively conflate the effects of 1) the feature's 's value and 2) the feature's 's relative position within the sequence.
In this work we propose OrdShap, a local feature attribution approach that quantifies model sensitivity to both feature values and feature positions in sequential data (Fig. 2). To the best of our knowledge, OrdShap is the first method to disentangle these two effects. This disentanglement valuable in critical domains such as health, where the timing of medical events be important as their occurrence. For example, when analyzing patient trajectories, OrdShap can identify whether an elevated lab value is significant due to its magnitude or because it occurred when the patient was first admitted a distinction that cannot be captured with existing methods.
particularly
Main Contributions:
We propose OrdShap, a local feature attribution framework for sequential models. We further propose OrdShap-VI VI and OrdShap-PI, which quantify the Value Importance (VI) and Position Importance (PI) of each feature sequence, respectively.
We establish a game-theoretical connection between OrdShap and the Sanchez- Berganttno (SB) value [63], which satisfies several desirable axioms but has not been utilized for feature attribution.
We propose two algorithms efficiently approximate OrdShap.
Empirical results from health, natural language, and synthetic datasets show that OrdShap is able to capture how feature ordering affects model prediction.
2 Related Works
Feature Attribution Methods. variety of post-hoe feature attribution methods have been proposed
[26, 88], many of which can be applied to sequential models. Existing methods have used feature masking [60, 43, 17], model gradients [4, 66, 76, 69], or self-attention weights [86, 1, 22, 13]. Lundberg and Lee [43] adapted Shapley Values [65] for feature attributions, which has been extended
We provide additional background EHR data in App. A.2.

---

to number of Chapley-based approaches [75, 44, 21, 33). While these attributions methods can be applied to sequential models, many assume feature independence [43, 39]. Frye al. [23] and Wang et al. [82] investigate additional constraints to enforce sequential dependency. Second-order methods [44, 77, 46, 32, 79] can also be used to detect used detect feature dependencies.
Feature Attribution for Sequential Models. Several methods have also been proposed specifically for sequential models [51, 89, 78], both on unstructured and structured datasets. TransSHAP [36] and MSP [72] calculate feature attributions for unstructured text using feature masking. TimeSHAP [6] adapts KernelSHAP [43] to Recurrent Neural Networks. Meng et al. [47] applies realistic perturbations to test samples using generative model. TIME [71] calculates the change in model loss when permuting features between samples. While these methods are specific for sequential models, they do not explicitly estimate the importance of feature position or order. The most closely related work PoSHAP [20], which a global method that averages KernelSHAP [43] attributions over each position index for given test set of samples. In contrast, the proposed method OrdShap is local attribution method that distentangles the effects of within-sample feature permutations.
3 Technical Preliminaries and Background
Let X be dataset, where each sample E X is sequence (æ), (2,, ..,,..,, .,,)). We refer to elements I, E x as features of x; depending on the application, these can represent (e.g.) tokens, sensor measurements, word embeddings, and can be real or vector-valued. Given a sample N = d) be the set of feature indices in ær. Consider a trained prediction model f that takes samples from X input. We refer to f as the black-box model that we want to explain. We assume that f has real-valued output; in the multi-class setting, we take the model output for single selected class. Let Gg be the symmetric group on S S N with permutations mapping corresponding to each a E G5 using the same symbol, i.e. (i) maps i E S to another element j E S. For example,
X, let
Gg. We denote the bijective
write given E G (1.233,4) using one-line notation:
(a(1), (3),
(3, 1,
We can similarly write the inverse permutation a² which maps back to the previous ordering:
σ -0 (1), '(2), 1(3), (44) = 2,
An inversion is a pair (a(i), such that < and oii) > oj). We denote og as the unique permutation within Gg AA N with 0 inversions, i.e. the identity permutation. Let Ø,i,t (o())...., ol - 1), of (l......((()))) position E. The goal feature attribution is to assign a scalar value to each feature E N representing its relevance to the model output. In this work, we focus on the Shapley value framework; we introduce Shapley values in $3.1 and discuss how they are used for feature attribution in $3.2.
denote the permutation o E 65y()) with inserted at
3.1 Shapley Values and Extensions
The Shapley value [65] is game- theoreetc method to attribute the contributions of individual "players" to cooperative game. Let P(N) - R be set function, i.c. the characteristic function, that represents the "payoff" assigned to a subset of players S C N, where P(N) denotes the power set of N. The Shapley value is then the unique attribution for (N, that satisfies several desirable axioms: Efficiency, Symmetry, Null Player, and Additivity (App. A.4).
(NN - |S| 1)!(S))) [USU - x(S)
(N
Σ SCN(O)
(3)
NI
The notation |. indicates set cardinality. Intuitively, the Shapley value averages the marginal
contribution of feature i (i.e. v (S -v()))>(s))) v(S)) over the all subsets S S N \ {i). Note that Eq. 3 does not assume any player ordering. A number of extensions have been proposed in this direction [53, 63]; in particular, Sanchez and Bergantiñoss [63] propose a value that also captures permutations for each subset S C N. Let w : Ω IR denote a generalized characteristic function, where Ω the set of permutations a E 6 VS C N. We can then define the Banchez-Bergaätinng (SB) value:
151+1
-NNI- - 1)!
d SB (N,w) = Σ
Σ
Σ
[w<o,s,,) - w(o)]
IN((SS+
SCNV(1 8ESS
3

---

The SB value has been investigated in the context of network theory [18, 49] and other game theory applications [48], though it has not been investigated in the XAI literature. Similar to the Shapley value, the SB value is the unique value to satisfy the axioms of F Symmetry"" Null Player*, and A which have been adapted for ordered coalitions (App. A.5).
3.2 Shapley Values for Feature Attribution
Shapley values have become widely adopted in machine learning as principled approach for calculat- ing feature attributions in black-box models [61]. Specifically, we define features as "players" in cooperative game, then define a characteristic function v/,z that maps a given sample x E X. with features N S ablated, to the output of model f. Several such functions have been investigated in the literature [43, 75, 14], differing mainly in their ablation mechanisms. Feature ablation is generally performed by masking ablated features with some value. For example, we can replace features N \ S by sampling a distribution X, where X' can be the data distribution or a reference distribution.
(5) We omit the subscripts f.æ: when clear from context. In contrast, the SB value have not been investigated in the context of feature attribution. In the next section we discuss challenges with the SB value and introduce novel framework for attributing importance feature positions.
(S) = Ex (f') 2 = vi E S]
4 Disentangling Feature Position Importance
Assume that we have sample z E X and a prediction model f whose output f(x) depends on the ordering of features in x. Our goal is to generate an attribution for each feature in x based on I) the feature's value, and 2) the feature's position within the sequence. For convenience, we assume that each feature can be permuted to any position within the sequence; in App. B.1 we extend this assumption for applications with fixed subsequences of features or irregular time intervals. We first introduce novel characteristic function to allow feature permutations ($4.1). We then define OrdShap in $4.2. Approximation methods are addressed in $5.
4.1 Quantifying the Effects of Feature Ordering
We propose to quantify the model's output when permuting each feature across different positions in the sequence. Intuitively, the Shapley value evaluates the model f for every possible subset of features S S N. For each subset S C N, we also want to evaluate ff when each feature E S is permuted to every position é N. This poses challenge when applying permutations Gg to the sequence v, since the features N \ S are not permuted. More formally, for given feature i, there exist positions e E N such that:
EHor E Gg -1i)=
EHoe Gs ((i)= ='3l
(6)
SCN
SCN
EE
VES
The proof is detailed in App. C.I. Therefore, feature is not permuted uniformly across positions. For example, in the singleton set S = (i). the only permutation a E Gg is o therefore i is not permuted to positions f E N S. We therefore define the characteristic function @ PPN) GN - R, which
instead rearranges x for permutation a E GG then ablates the features in \ 5:
Iz VEE VENNS
f (S, Ex f(x') 202-00
(7)
Note that Eq. 7 is generalization of Eq. 5, reducing to the latter under the identity permutation. Concretely, " N = v. recovers the model output for sample x under the original order.
4.2 OrdShap: Shapley Value with Positional Conditioning
We now define a new attribution N θ - & x IRd, where θ is the space of functions W. Definition 1. (OrdShap) Given set of players N = {1....,d} and function w P(N) x G + R, then the OrdShap values e a)) for each player i E and position E N is defined as follows:
(1S) - 1)!(IN) - SSD) (=((, - ((S\(i).o))
The (N,(NW = Σ Σ
(8)
(IN) 1)!N!
SCN GEEN iES (1)

---

Definition I yields d x d matrix, where each row corresponds to a feature i E N, and each column to position & N. Each element "h represents the importance feature conditioned on being permuted to position E in the sequence. The diagonal elements "/, correspond to the importance of each feature its original position, with the remaining features randomly permuted. Thus, OrdShap provide a structured representation of how feature position influences the model output.
Next, while OrdShap offer a detailed view of the position-specific importance, we propose two summary measures: importance related to feature value (OrdShap-VI) and position (OrdShap-PI).
OrdShap- VL. We can disentangle the effect of feature's 's value from its position by averaging over all possible positions where that feature might appear (i.c., averaging over a (i)). This marginalization yields single attribution score for each feature that reflects only the feature's 's contribution due to its value, separate of any positional dependency within the sequence.
RN, = |NV
Σ (, PEN
(9)
This can be interpreted as the Shapley value averaged over all permutations of the features in x. Under certain conditions, Eq. 9 has an additional interpretation the SB value for feature i:
Theorem 1. Given a set of players N = (l,...,d} and characteristic function ß P(NN x GN F. there exists a corresponding function w f2 - E. representing is averaged over the permutations functiooo
a E G which contain a given π E G5 AS C N. Then 7/( ()) = (N w) is the unique value to satisfy the SB axioms of Efficiency, Symmetry, Null Player, and Additivity (App. A.5).
The proof is detailed in App. C.2. Further, if we assume that permuting ablated features, which represent noninformative features, to different positions within x has effect the model output, then % corresponds directly to the SB value for feature i.
OrdShap-PI. We propose summary measure that quantifies the average change in "/ as the feature position l varies, by approximating the positional effects through linear model. Intuitively, OrdShap-PI provides an interpretable answer to the question: is given feature more important it appears later in the sequence, and by how much more? The OrdShap-PI B; is estimated by minimizing the squared error between the centered feature importance V/. 7% and the centered positions f f, where i = ΣjEN //|NV|
arg min [[e- 2)3, - (N..(N,, v) - = &
(10)
This results in attribution whose magnitude indicates the strength of the positional effect, and whose sign indicates whether the model output increases (3, > 0) or decreases (3; < 0) as feature i is permuted to later positions in the sequence. Together, the OrdShap-V VI values capture the feature importance when averaging over all positions, and OrdShap-Pl reflects the average change in model output as a feature's 's position deviates from its mean position. In practice, full OrdShap matrix for detailed positional effects on model predictions, or use the OrdShap- VI and OrdShap-PI Pl values for summarized attributions for feature value and position.
leverage the
4.3 Comparison: Shapley, OrdShap, OrdShap- VI, and OrdShap-PI
In Fig. we present a toy example illustrating the differences between the introduced measures. Items are drawn from a pile with replacement, each with a value defined in Table 1. In particular, note that bags have no inherent value, however, any items drawn before drawing a bag are discarded, and therefore have value 0. Only pairs of gloves, and not individual gloves, have value. We investigate a sample sequence of six items, =[Hat, Hat, Hat, Bag, R-Glove, R-Glove], with combined value of (2 = O.
Table 1: Payoffs for the toy example.
Item
Value
Bag. L-Glove, R-Glove
if drawn after
Hat
"Bag". otherwise
if drawn after "Bag", otherwise O.
Pair of Gloves
We observe that Shapley Values ($3.2) yields zero attribution for each feature, as the method does not capture order-dependent effects. contrast, the OrdShap values (Fig. 3B) indicate that several features have non-zero importance but they are highly affected by sequence position. In particular, the "Bag" feature (red) contributes less when appears later, as fewer items can follow it. The three "Hat" features (blue, orange, green) have equal, increased value in later positions, while the two "R-Glove"
5

---

(A)
Shapley Values
(B) OrdShap Values
(C)
OrdShap-VI
(D)
OrdShap-PI
Hat 0.0
Hat
0.5
0.2
Hat 0.0
Hat
05
02
Hat
0.0
PAt
Hat
0.5
0.2
Bag
Bag
1.5
-0.6
R-Glove
0.0
R Glove
0.0
R-Glove
0.0
0.
1.0
15
2.0
2
Position
0.0
05
1.0
1.5
2.0
-1.0
-0.5
0.0
0.5
1.0
Figure 3: Toy example illustrating the differences between (A) Shapley Values, (B) OrdShap, (C) OrdShap-Vi VI, and (D) OrdShap-PL. Values are calculated on the sample [Hat, Hat, Hat, Bag, R-Glove, R-Glove], with characteristic function defined in $4.3.
features yield no value, they never form pair with a "I-GGlove" feature. The OrdShap-Vi values (Fig. 3C) provide the average contribution across all permutations, isolating feature importance from positional effects. The OrdShap-P1 (Fig. 3D) then quantifies position importance; the strongly negative coefficient for "Bag" indicating reduced value in later positions. summary, OrdShap shows that "Hat" and "Bag" do contribute to the sequence, but their impact is masked by their positional dependency.
OrdShap Approximation
While OrdShap provides way to quantify importance due to feature value and position, the for- mulation in Eq. is generally infeasible due to the 2d subsets and d! permutations. We therefore introduce model-agnostic, fixed-sample algorithms for approximating OrdShap. $5.1 introduces a sampling-base approach that approximates "/a,- $5.2 introduces a least-squares approach for directly estimating OrdShap-VI and OrdShap-PI.
5.1 Sampling Approximation
We use permutation sampling approach [74] to approximate ",,,(N, () with a fixed sample size (detailed in App. B.3, Alg. 1). Our approach samples subsets $ C N and permutations &N. then evaluates the marginal contribution to model output when feature is included in For a given l E N we filter permutations such that o (i)
5.2 Least-Squares Approximation
While the sampling algorithm in $5.1 estimates each (N, using a fixed number of samples, we can further improve efficiency by directly approximating OrdShap- VI and OrdShap-PI PI using leaar-s (I.S) approach. compared with the sampling algorithm, the LS approach reduces the number of calls to the model f. since attributions are calculated for all features simultaneously.
Definition 2. (Least-Squares Approximation of OrdShap) Given a set of players N = {l..... ... d} and
characteristic function is PPN) x GN - R. then OrdShap-V VI and OrdShap-PI the solutions Oi 3i, respectively, Vi E N for the following minimization problem:
mii
ΣΣ H(SS))
Σ LES
04 +
Σ iES
0
-1) (3(8 -0(,,))5((0,65)
SCN OEGN
S40,N
Σ
Σ G(N,o)+ (11)
8.l.
ai
N!
NEN
dEEN
The coefficient μ applies weighting based on subset size. Per Thm. (proof details in App. C.3), setting this coefficient to
-1
((())))(()) recovers the SB value for the optimal a coefficients.
Theorem 2. Given of players = {1, {1....,dy d], characteristic function ß P(N) GN RR
and weighting p(s) ( (1)) then the coefficients a1...., a minimizing Eq. 11 are SB values for the game (N,w), w), where is the corresponding function defined in Thm. I.
|N

---

DeepLift
KameSHAP
LIME
LOCO
Random
OrdShap-VI
Avo. Meddl Oulputt (Prabablitn)
EICU Mortality
MIMICIII Mortalit
EICU-LOS
MIMICIII-LOO
IMDB
0.95
0.90
0.82
0.90
0.80
0.85
0.80
0.87
0.80
0.79
0.80
0.75
0.76
0.75
3.0
0.5
1.0
0.0
0.5
1.5
0.0
0.5
1.0
0.0
0.5
1.5
0.0
6.5
14
% Features Permuted
5. Features Permuled
Sc Features Permutee
5. Features Perm-uted
% Features Permuted
Figure 4: Evaluation of OrdShap-Pl (blue) by permuting an increasing number of features and calculating the change in the predicted probability of the predicted class (higher is better). On average, permutating features according the OrdShap-PI attributions increases or maintains the model's prediction, in contrast to existing methods. AUC calculations and error bars are provided in App. E.I.
The optimal / coefficients then represent the linear approximation of the average change in Y, as the position l changes, i.e. the OrdShap-PI values. We now approximate Eq. 11 using fixed number of samples. Let L/(G x) and (NN(N) be the uniform distributions over GN and PNNN), N, we draw K samples Si..... SK ((P(N)) and KI. samples
respectively. For a given K, ......., KL ~ L((x)). ). We take advantage of two key properties to improve efficiency. First, from the proof of Thm. 2, we can calculate a independently of B3. Second, extending a result from [63], we establish the following corollary (proof details in App. C.4):
Corollary 2.1. Let v be a characteristic function s.t. vi() = T&T Enees W(S,o) VS N. Then the OrdShap- VI value "/s is equal to the Shupley value under v, L.e. 7: = di D).
This result allows solve for a separately using the KernelSHAP algorithm [43]; the solution to is then used to calculate B. Let A ((a) E {0, (0,1kkxa and A KKLid Lxd be defined as follows:
(a) 1
1s, (i)
A =s (a) ( 2)
(12)
where "mod" denotes the modulo operator and 1 denotes the indicator function. Additionally, let W(o) E [0, _] and W*() E [0, 1| be diagonal weighting matrices, with diagonal elements Wr(a) = p(S:), w = u(S, mod L) and all other elements equal to Let F(a) E R
F(0) R. represent the model outputs, with elements F((") 1, Et (S, olK+s)
(2, ) and F(() / (S: mad L, oz) (2, a ). It then follows that a, has the solution:
a (1) (A (A
(13)
3 = (4l0))110 [1 (A()))TW(O)
F(0)
(14)
Similar to Lundberg and Lee [43], we enforce the constraint in Eq. by eliminating a feature j in Eq. 13 then solving aj using the constraint. The algorithm is summarized in App. B.3 Alg, 2. In practice, we can improve efficiency by reducing the number of features to evaluate: for example, including a preliminary feature selection step, combining features into interpretable groups (similar to superpixels [60]), or omitting irrelevant features.
In summary, naively evaluating Eq. 8 incurs O(d! 55) complexity, where 8y is the cost of evaluating the model f. We reduce this to OdKL + dl LL) with the sampling algorithm, and ( de K d3) with the LS algorithm. In practice, the LS algorithm is typically faster than the Sampling algorithm since function evaluations are often the performance bottleneck. However, the LS algorithm only calculates OrdShap-PI and OrdShap- VI values, therefore the Sampling algorithm required to calculate the full OrdShap value matrix in Def. 1.
6 Experiments
We empirically evaluate OrdShap on its ability to identify the importance of given sample's feature ordering. In $6.1 we quantitatively evaluate OrdShap against existing attribution methods. In $6.2
7

---

Table 2: Evaluation of OrdShap- VI using Inclusion AUC (higher is better) and Exclusion AUC (lower is better) metrics, with standard error in parentheses. The berf-peen explainer for each model bolded. The associated eurves used to generate these results are provided App. E.I.
Metric
Explainer
EICU-LOS
EICU-Mort
MIMICII-LOSS
MIMICIIlMMrt
IMDB
DL
0.906 (0.010)
0,804 (0.013)
0.893 (0.011)
0,854 (0.012)
0.784 (0.016)
IG
0.903 (0.010)
0,803 (0.013)
0.896 (0.011)
0,859 (0.012)
0.791 (0.016)
KS
0.904 (0.010)
0.801 (0.014)
0.898 (0.011)
0,855 (0.012)
0,863 (0.011)
Inclusion
LIME
0,866 (0.012)
0.776 (0.016)
0.885 (0.012)
0,804 (0.015)
0.859 (0.012)
1
LOCO
0.904 (0.010)
0.799 (0.014)
0.894 (0.011)
0.854 (0-012)
0.855 (0.012)
Random
0.812 (0.014)
0.769 (0.016)
0.851 (0.014)
0.705 (0.017)
0.797 (0.016)
OrdShap-VI VI
0.913 (0.010)
0.809 (0.013)
0.899 (0.011)
0.862 (0.011)
0.866 (0.012)
DL
0.614 (0.025)
0.727 (0.022)
0.817 (0.018)
0,484 (0.025)
0,855 (0.012)
IG
0.629 (0.024)
0.728 (0-021)
0.795 (0-018)
0.481 (0.025)
0.867 (0.013)
KS
0.626 (0.024)
0.730 021)
0.805 (0.019)
0.485 (0.025)
0.766 (0-018)
Exclusion
LIME
0.771 (0.017)
0.750 (0.018)
0.840 (0.016)
0.591 (0.022)
0.804 (0.016)
AUC
LOCO
0.618 (0.025)
0.732 (0.021)
0.803 (0.019)
0.495 (0.025)
0.784 (0.017)
Random
0.821 (0.014)
0.759 (0.017)
0.866 (0.013)
0,694 (0.018)
0.849 (0.014)
OrdShap-Vi VI
0.573 (0.028)
0.724 022)
0.788 (0.020)
0.472 (0.025)
0.779 (0.021)
we investigate OrdShap on a synthetic dataset. In $6.3 we qualitatively compare OrdShap with KemelSSHA. In $6.4 we present execution time results. All experiments were performed on an internal cluster using AMD 7302 16-Core processors and NVIDIA A100 GPUs.
Datasets and Models. We evaluated OrdShap on two EHR datasets (MIMICII [34] and EICU [57]) and natural language dataset (IMDB [45]). Full dataset and model details are provided in App. D. MIMICIII and EICU are large-scale studies for Intensive Care Unit (ICU) patient stays. Both datasets include sequences of clinical events for each patient stay, such as administered lab tests, infusions, and medications. We follow Hur et al. [31] for data preprocessing and tokenization, then train BERT classification models [19] on two tasks for each dataset: 1) Mortality and 2) Length-of-Stay 2 3 days (LOS). We also include the IMDB dataset, containing over 50,000 movie reviews, on a sentiment analysis task. We apply a pretrained DistilBERT model [64] using the Huggingface library [85], then evaluate the relevance of each sentence towards the predicted sentiment of the entire movie review.
the OrdShap Least-Squares algorithm ($5.2) due to computational constraints. We compare OrdShap to a variety of attribution methods: KernelSHAP (KS) [43], LIME [60], and LOCO [40] are perturbation-basel approaches. We additionally compare against gradient-based approaches: Integrated Gradients (IG) [76], and DeepLIFT (DL.) [66]. We also include a baseline, Random, consisting of attributions drawn from (0, 1)". Additional detail on the implemented methods are provided in App. D.2.
Explainers. Unless otherwise stated,
6.1 Quantitative Evaluation
We compare OrdShap to a variety of attribution methods to evaluate how (1) OrdShap-PP -P captures feature position importance, and (2) OrdShap-V VI calculates an attribution that incorporates position information. For all explainers, we generate attributions for 200 test samples which are evaluated using quantitative metrics.
(1) Position Importance. The OrdShap-Pi attributions quantify both the magnitude and direction of positional impact on feature importance with respect to model predictions. Therefore, to evaluate OrdShap-PI, Pl, we systematically permute samples according to the generated attributions and measure the change in model output. Specifically, we select subset of features based attribution magnitude, then permute features toward the beginning (negative attributions) or end (positive attributions) of the sequence. We then measure the model output probability for the predicted class, P(Y Ø)æ), where ă represents the permuted sample and ğ is the predicted class under the original sample æ. We compare competing methods by similarly permuting features according to their attributions.
Results are presented in Fig. As expected, we observe that permuting features according to OrdShap-Pl generally increases the model output across feature subsets. Interestingly, the model output remains relatively flat for IMDB when permuting features; this suggests that the model is able to accurately predict sentiment regardless of sentence order. In contrast, competing methods fail to capture the importance of feature ordering. Permuting features according to traditional attributions either does not affect (EICU-LOS) negatively affects (all other models) affects model output.
8

---

(A) Token t;
Value vt.))
VI
Pl
Legend
(B)
OrdShap
(C) 1.0
A B
0.1i2 +2
Acbribution Va ves
0.12
/
/
OrdShwp-i
C
2
/
x
8.0
0.0
D
x
x
A,
C.C
-,,,
0.5
fncolineur (f) sigmoid
(E/2 v(t))
-0.05
0.00
0.05
-1.0
OrdShap-P1
K5
LIME
LOCO
DL
Figure 5: Attributions for a synthetic dataset and model Snonlinear. (A) Token values; tokens are assigned Value Importance (VI) and/or Position Importance (PI) with respect sequence index i. (B) OrdShap- VI and OrdShap-PP attributions are able to separate the different tokens based on VI and PI effects. (C) In contrast, attributions from existing methods cannot distinguish between the different tokens since VI and PI effects are entangled.
(2) Value Importance. OrdShap-V VI attributions quantify a feature's contribution to model predictions independent of positional effects, making them robust across different sequence orderings. To evaluate this property, we extend the Inclusion/Exclusion AUC (IncAUC/ ExcAUC) [33] metrics to incorporate feature permutations (details provided in App. D.3). More concretely, for each sample we rank features by attribution scores and iteratively select the top k% of features. These features are then either masked from the original sample (ExcAUC) or retained in an otherwise completely masked sample (Un) Importantly, we add a permutation step that reorders features in the masked sample, We then measure agreement (post-hoc accuracy) between the model's predictions under the masked, permuted samples and original sample, averaging over 10 permutations for each kt then summing the area under the resulting curve for different l values. We additionally implement similarly modified iAUCIdAUC metric [56], which directly evaluates the model output rather than post-hoc accuracy.
IncAUC/ExcAUC results are shown in Table 2 (iAUC/dAUC results in App. E. Table 5). We observe that OrdShap-V VI outperforms all competing methods on IncAUC, and performs well on ExcAUC, only second to KemelSHAA on the IMDB dataset. This supports our observation from (1), that the DistilBert model predictions are only mildly affected by sentence order in IMDB, Overall the results indicate that OrdShap-V VI provides more robust attribution under feature permutation, and better captures each feature's s value importance independent of positional context.
6.2 Attribution Disentanglement on Synthetic Data
We create synthetic dataset to evaluate the disentanglement of value and position importance.
We sample 200 sequences ((0)) n E (1,..., 200}, of length 10, where each element (!m)) is a token sampled uniformly from the set (A, C, D, A, B, CJ. We define two models: funear (t) = v(4) v(4) and Snealinear (4) = sigmoid ((ieaar (1)). where v(t,) is the value of token 4 (Fig. 5A).
E
Fig. 5 shows the results for fnoulincar- We observe that the associated OrdShap- VI and OrdShap-Pl (Fig. 5B) follow the expected VI and PI values from Fig. 5A. In particular, tokens A, B, A, B exhibit nonzero OrdShap-PI values, which indicates dependency
feature position. In contrast, tokens exhibit excibit OrdShap-PI x O, indicating no positional dependency. In addition, OrdShap- VI and OrdShap-Pl attributions are able fully separate the 7 tokens according to their value and position importance. This contrasts with traditional attribution methods (Fig. 5C), in which value and position importance are entangled together, and thus the tokens are unable to be separated.
C.D.C
The results for fínearr are shown in App. E.3 Fig. 9. The additive nature of ftinear results in constant OrdShap attributions for all samples in the dataset. However, existing metrics (Fig, 9C) still unable to disentangle the effects of value and position importance.
6.3 Case Study on Medical Tokens
In Fig. 6A we further investigate the LOS 2 3 prediction sample from Fig. this sample, the model predicts LOS < 3 days, therefore positive attributions indicate important features with respect to LOS < 3 days. Several key observations emerge. Many medication tokens have high OrdShap-Vi VI values, which suggests that these medications correspond to LOS < 3 prediction. In particular, Potassium Chloride generally used to treat potassium deficiency, a relatively minor condition that does not generally require long hospital stay. OrdShap-PI (negative with low magnitude) indicates

---

Table Execution time results, in seconds, averaged over 200 samples. Standard deviation results are provided in parentheses. We compare existing attribution methods with OrdShap. calculated using the Least Squares (I.S) and Sampling (S) algorithms. Note that competing methods cannot disentangle importance due to feature value and position.
Explainer
EICU-LOS
BICU-Mort
MIMICII-LOS
MIMICIII- Mort
IMDB
DL
0,02 (0.03)
0.01 (0.03)
0.02 (0.03)
0.01 (0.03)
0.08 (0.03)
0,13 (0.25)
0.04 (0.07)
0.10 (0.08)
0.03 (0.04)
0.22 (0.08)
KS
0.69 (0.08)
0.21 (0.08)
0.68 (0.10)
0.17 (0.06)
1.85 (1.88)
LIME
0,14 (0.04)
0.11 (0.06)
0.16 (0.07)
0.12 (0.03)
0.14 (9.05)
LOCO
0.57 (0.18)
0.38 (0.09)
(0.16)
0.26 (0.11)
0.06 (0.04)
OrdShap (LS)
29.29 (1.30)
5.53 (0.30)
27.33 (4.41)
5.49
54.61 (72.16)
OrdShap (S)
2470.73 (085.0)
984.03 (264.0)
1369.00 (721.7)
565.51 (288.9)
680.23 (440.9)
that this token is less predictive of LOS < 3 (and conversely, more predictive of LOS 2 3) when it occurs later in the sequence. We observe that, indeed, KerneISHAP attributions are relatively low compared with OrdShap-V VI for this token due to its position, however KerneISHAP fails to distentangle the positional effect.
Interestingly, Bedside Glucose Test is assigned a high KemelSHAP value, but OrdShap indicates that this to- ken's importance is mainly positional; having this test later in the sequence is indicative of longer LOS. This follows intuition, as Bedside Glucose common test that does not typically indicate patient's condition. Again, KernelSHAP assigns this token a high attribu- tion value since it appears within the last quartile of the original sequence, but cannot distinguish between the positional and value importance.
(A)
Medication
OrdShap Attributions
0.06
Potassium Chloride mEqimi
OrdShap
0.02
0.00
Base Excess Test
Bedsidd Glucosa Test
-0.04
11.25
0.75 -0.50 0.25
OrdShap-PI
6.4 Execution Time Evaluation
(B)
KerneiSHAP Attributions
We provide exccution time results for OrdShap and com- peting methods in Table 3. Both OrdShap algorithms are generally slower than competing methods; this is this cost of calculating positional attributions, which requires evaluating the black-box model over multiple permutations of the input sequence. As expected, the the Least Squares algorithm (LS) significantly improves computational efficiency compared with the Sampling algorithm (S), and therefore should be the preferable algorithm when the does require calculating the entire matrix.
0.05
0.00
Base Excess Test
Bedside Glucose Test
Potasolum Chioride mEg/ml
Figure 6: Case study for the sample from Fig. 1. (A) OrdShap-PI and OrdShap-VI attributions. (B) KernelSHAP attributions.
7 Limitations and Conclusion
While many feature attributions methods exist for sequential deep learning models, existing methods assume fixed feature ordering when explaining model predictions. Our work addresses this gap by introducing OrdShap, a novel approach that uses permutations to quantify the importance of feature's position within the input sequence. Regarding limitations, Shapley-based methods are often computationally expensive due to the summation over 2d coalitions; OrdShap additionally requires averaging over d! permutations. We address this limitation through sampling and approximation ($5), and leave further improvements for future work. In particular, many algorithms have been proposed for improving the efficiency of Shapley value approximation (App. A.3) which could possibly be extended to approximate OrdShap. Additionally, while we theoretically establish a relationship between OrdShap-V VI and Banchez-Bergantin values in this work (Thm. 1), OrdShap-PI requires different axiomatization based on positional importance, which we leave for future work.
10

---

Acknowledgements
This work was supported in part by the following grants from the National Heart, Lung, and Blood Institute: NIH 2132HL007722-44, UOI U01HL089856, 5R01HL167072, and 1R01HL171213-02. We thank the NeurIPS reviewers and metaereviewers for their helpful comments and feedback.
References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020. Association for Computational Linguistics. doi: 1220 385.
[2] David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. CoRR, abs/ 0/5/1886.080499 2018.
[3] Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning, pages 272 221. PMLR, 2019.
[4] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robers Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10, 2015.
[5] Alejandro Barredo Arrieta, Natalia Dodz-Rod Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58: 82-115, 2020. ISSN 1566-2535. doi: 10.
[6] João Bento, Pedro Saleiro, André F. Cruz, Mário A.T. Figueiredo. and Pedro Bizarro. Time- SHAP: Explaining recurrent models through sequence perturbations. Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, KDD '21, pages 2565-2573, New York, NY, USA, 2021, Association for Computing Machinery. ISBN 978-1- 4503-8332-5. doi: 10. 10.1145/344554..344 177754464.4666666
[7] Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and Aggregating Feature-based Model Explanations. arXiv:2005.000 00063 [cs, stat], May 2020.
[8] Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O' O'Reilly Media, Inc.", 2009.
[9] Philippe Bracke, Anupam Datta, Carsten Jung, and Shayak Sen. Machine learning explainability finance: An application to default risk analysis. 2019.
[10] Michael Bücker, Gero Szepannek, Alicja Gosiewska, and Przemyslaw Biecek. Transparency, auditability, and explainability of machine learning models in credit scoring. Journal of the Operational Research Society, 73(1):70-90, 2022.
[11] Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the Shapley value based on sampling. Computers operations research,
2009.
[12] A Charnes. B Golany, M Keane, and J Rousseau. Extremal principle solutions of games in characteristic function form: Core, Chebychev and Shapley value generalizations. Econometrics of planning and efficiency, pages 123-133, 1988.
[13] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 782-791, 2021.
[14] Hugh Chen, Ian C Covert, Scott M Lundberg, and Su-In Lee. Algorithms to estimate Shapley value feature attributions. Nature Machine Intelligence, -00,,,, 2023.
11

---

[15] Justin Cosentino, Babak Behsaz, Babak Alipanahi, Zachary R. McCaw. Davin Hill, Tae-Hwi Schwantes- An, Dongbing Lai, Andrew Carroll, Brian D. Hobbs, Michael H. Cho, Cory Mclean, and Farhad Hormozdiari. Inference of chronic obstructive pulmonary disease with deep learning on raw spirograms identifies new genetic loci and improves risk models. Nature Genetics, April 2023. ISSN 1546-1718.
doi: 10.1038/s41 (a0.041118888022---11 023-01337--
[16] Ian Covert and Su-In Lee. Improving kernelSHAP: Practical shapley value estimation using linear regression. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of the 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 3457-3465. PMLR, April 2021.
[17] Ian C Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. The Journal of Machine Learning Research, 22(1):9477-9566 2021.
[18] Mónica del Pozo, Conrado Manuel, Enrique Gonzalez-Arangienna, and Guillermo Owen. Centrality in directed social networks. A game theoretic approach. Social Networks, 33(3): 191-200, 2011.
[19] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4171-41181 2019.
[20] Quinn Dickinson and Jesse G. Meyer. Positional SHAP (PoSHAP) for Interpretation of machine learning models trained from biological sequences. PLOS Computational Biology, 18(1):1-24, January 2022. doi: fo..a//jommlll.cbb.o00000
[21] Gabriel Erion, Joseph D Janizek, Pascal Sturmfels, Scott M Lundberg, and Su-In Lee. Improving performance of deep learning models with axiomatic attribution priors and expected gradients. Nature machine intelligence, 3(7):620 631, 2021.
[22] Kawin Ethayarajh and Dan Jurafsky. Attention flows are shapley value explanations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 49-54, Online, August 2021. Association for Computational Linguistics. doi: 1188653311/222 acl-short.8.
[23] Christopher Frye, Colin Rowat, and Ilya Feige. Asymmetric Shapley Values: Incorporat- ing causal knowledge into model-agnostic explainability. Advances in Neural Information Processing Systems, 33:1229-1223 2020.
[24] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. SHAP-IQ: Unified approximation of any-order shapley interactions. Advances in Neural Information Processing Systems, 36:11515-11551 2023.
[25] Fabian Fumagalli, Maximilian Muschalik, Patrick Kolpaczki, Eyke Hüllermeier, and Barbara Hammer. KerneISHAP-IQQ Weighted least square optimization for shapley interactions. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 14308-14342. PMLR, 2024.
[26] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. survey of methods for explaining black box models. Acm Computing Surveys, 51(5), August 2018. ISSN 0360-0300. doi: 10.1 15/124//33600
[27] Brian Hill, Melikasadat Emami, Vijay Nori, Aldo fordova-Palomert Robert Tillman, and Eran Halperin. CHIRon: A generative foundation model for structured sequential medical data. In Deep Generative Models for Health Workshop NeurIPS 2023, 2023.
[28] Davin Hill, Max Torop, Aria Masoomi, Peter Castaldi, Edwin K. Silverman, Sandeep Bodduluri, Surya P Bhatt, Taedong Yun, Cory McLean, Farhad Hormozdiari, Jennifer Dy, Michael Cho, and Brian D Hobbs. Deep learning utilizing suboptimal spirometry data to improve lung
12

---

function and mortality prediction in the UK biobank. medRxiv the preprint server for health sciences, 2023. doi: 0..1101/2023.04.282223.3 23.2899777.
[29] Davin Hill, Aria Masoomi, Max Torop, Sandesh Ghimire, and Jennifer Dy. Boundary-aware un- certainty for feature attribution explainers. In International Conference on Artificial Intelligence and Statisties, pages 55 63. PMLR, 2024.
[30] Davin Hill, Joshua Bone, Aria Masoomi, Max Torop, and Jennifer Dy. Axiomatic explainer globalness via optimal transport. In The 28th International Conference on Artificial Intelligence and Statistics, 2025.
[31] Kyunghoon Hur, Jiyoung Lee, Jungwoo Oh, Wesley Price, Younghak Kim, and Edward Choi. Unifying heterogeneous electronic health records systems via text-based code embedding. In Conference Health, Inference, and Learning, pages 183-203. PMLR, 2022.
[32] Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. Explaining explanations: Axiomatic feature interactions for deep networks. The Journal of Machine Learning Research, 22(1)4688744400 2021.
[33] Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. FastSHAP: Real-time shapley value estimation. In International Conference on Learning Representations, 2022.
[34] Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng. Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, a freely accessible critical care database. Scientific data, 3(1):1-9, 2016.
[35] Zulqarmain Q Khan, Davin Hill. Aria Masoomi, Joshua T Bone, and Jennifer Dy. Analyzing explainer robustness via probabilistic lipschitzness of prediction functions. In International Conference Artificial Intelligence and Statistics, pages 1378-1386. PMLR, 2024.
[36] Enja Kokalj, Blaž Skrlj, Nada Lavraë, Senja Pollak, and Marko Robnik-Sikoja. BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, pages 16-21, Online, April 2021. Association for Computational Linguistics.
[37] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:2009.07896 2020.
[38] Patrick Kolpaczki, Maximilian Muschalik, Fabian Fumagalli, Barbara Hammer, and Eyke Hüllermeier. SVARM-IQ: Efficient approximation of any-order Shapley interactions through stratification. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors, Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pages 3520-3528. PMLR, 2024.
[39] I. Elizabeth Kumar, Suresh Venkatasubramanian. Carlos Scheidegger, and Sorelle A. Friedler. Problems with Shapley-value based explanations as feature importance measures. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML' '00. JMLR.org, 2020.
[40] Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman. Dnstribution-frre predictive inference for regression. Journal the American Statistical Associ- ation, 10955-33:004411111 2018
[41] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrish- nan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Transformer for Electronic Health Records. Scientific Reports, 10(1):7155, April 2020. ISSN 2045-2322. doi: 10. 10.1088/s41588 190-000--22--0----.....
BEHRT:
[42] Scott Lundberg and Su-In Lee. A Unified Approach to Interpreting Model Predictions. Technical arXiv, November 2017. URI. attt:://arxie http///wwvv.orgaa///////
Report arXiv: 1.XXiv1705.00 705.07007774 07874. arXiv: 1705.07774 [cs, stat] type: article.
13

---

[43] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[44] Scott M Lundberg, Gabriel G Erion, and Su-In Lee. Consistent individualized feature attribution for tree ensembles. arXiv preprint arXiv: /00.:..//..000/ 088803888, 2018.
[45] Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, 2011.
[46] Aria Masoomi, Davin Hill, Zhonghui Xu, Craig P. Hersh, Edwin K. Silverman, Peter J. Castaldi, Stratis Ioannidis, and Jennifer Dy. Explanations of black-box models based on directional feature interactions. In 10th International Conference on Learning Representations, ICLR 2022. OpenReview.net, 2022.
[47] Han Meng, Christian Wagner, and Isaac Triguero. Explaining time series classifiers through meaningful perturbation and optimisation. Information Sciences, 645:119334, 2023, ISSN 0020-0255. doi: 10.
[48] Rodolfo Metulini and Giorgio Gnecco. Measuring players' importance in basketball using the generalized Shapley value. Annals of Operations Research, 325(1):441-465 2023.
[49] Tomasz P. Michalak, Piotr L.. Szczepański, Talal Rahwan, Agata Chrobak, Simina Brånzei, Michael Wooldridge, and Nicholas R. Jennings. Implementation and computation of a value for generalized characteristic function games. ACM Transactions on Economics and Computation. 2(4), October 2014. ISSN 2167-8375, doi: 10. 10.1145/2665007.
[50] Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for shapley value estimation. Journal of Machine Learning Research, 23(43): 46, 2022.
[51] Edoardo Mosca, Ferenc Szigeti, Stella Tragianni, Daniel Gallagher, and Georg Groh. SHAP- based explanation methods: A review for NLP interpretability. In Proceedings of the 29th International Conference Computational Linguistics, pages
2022.
[52] An-phi Nguyen and María Rodríguez Martínez. On quantitative aspects of model interpretability. arXiv preprint arXiv:2007.07584, 2020.
[53] Andrzej S. Nowak and Tadeusz Radzik. The shapley value for n-Person games in generalized characteristic function form. Games and Economic Behavior, 6(1):150-161, 1994. ISSN 0899-8256. doi: (9000//amee.194.100000
[54] Lars Henry Berge Olsen and Martin Jullum. Improving the sampling strategy in KemneISHAP, 2024.
[55] Chao Pang, Xinzhuo Jiang, Krishna S. Kalluri, Matthew Spotnitz, RuiJun Chen, Adler Perotte, and Karthik Natarajan. CEHR-BERYS Incorporating temporal information from structured EHR data to improve prediction tasks. In Subhrajit Roy, Stephen Pfohl, Emma Rocheteau, Girmaw Abebe Tadesse, Luis Oala, Fabian Falck, Yuyin Zhou, Liyue Shen, Ghada Zamzmi, Purity Mugambi, Ayah Zirikly, Matthew B. A. McDermott, and Emily Alsentzer, editors, Proceedings of Machine Learning for Health, volume 158 of Proceedings of Machine Learning Research, pages 239-260. PMLR. December 2021.
[56] Vitali Petsiuk, Abir Das, and Kate Saenko. RISE: Randomized Input Sampling for Explanation of Black-box Models. arXiv: / 1806.07421 [cs], September 2016 88
[57] Tom J. Pollard, Alistair W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. The eICU Collaborative Research Database, freely available multi-center database for critical care research. Scientific Data, 5(1):880178 September 2018. ISSN 2052-4463. doi: 10.1038/sdata.201 2018..17..
14

---

[58] Khansa Rasheed, Adnan Qayyum, Mohammed Ghaly, Ala Al-Fugaha, Adeel Razi, and Ju- naid Qadir. Explainable, trustworthy, and ethical machine learning for healthcare: survey. Computers in Biology and Medicine, 149: 106043, 2022.
[59] Laila Rasmy, Yang Xiang, Ziqian Xie, Cui Tao, and Degui Zhi. Med-BERT: Pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction, NPJ digital medicine, 4(1):66, 2021.
[60] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should I Trust You?": Explain- ing the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pages 1135-1 1135-1144, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978-1-4503- 4232-22. doi: 10. 09.114///939672.29397777
[61] Benedek Rozemberezki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. The shapley value in machine learning. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI- 22, pages 5572-5579. International Joint Conferences on Artificial Intelligence Organization, July 2022. doi: 10.24963/ijcai. 2022/778. 20227777..
[62] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propppappgggiinggg errors. nature, (3888))) 536, 1986.
[63] Estela Sanchez and Gustavo Bergantiñios. On values for generalized characteristic functions. 19:229-234, 1997.
[64] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
[65]
Lloyd S Shapley et al. A value for n-person games. 1953.
[66] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences.. In Doina Precup and Yee Whye Tch, editors, Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1-453333. PMLR, August 2017.
[67] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods. In Proceedings of the AAAI/ACM Conference AI, Ethics, and Sociery, AIES '20, pages 180-186, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978- 1-053-77100-- doi: 10.1 10.1145/3355227.335588
[68] Dylan Slack, Sophie Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable Post hoc Explanations Modeling Uncertainty in Explainability. In Neural Information Processing Systems (NeurIPS), 2021.
[69] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: Removing noise by adding noise. arXiv preprint 1rXiv:1706.0382. 2017.
[70] Matthew Smith and Francisco Alvarez. Identifying mortality factors from Machine Learning using Shapley values-a case of COVID19. Expert Systems with Applications, 176:1 14832, 2021.
[71] Akshay Sood and Mark Craven. Feature importance explanations for temporal black-box models. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8):8351-8360, June 2022. doi: 10. 16.1/a9/aai, v36.8.20800.
[72] Joel Stremmel, Brian L. Hill, Jeffrey Hertzberg, Jaime Murillo, Llewelyn Allotey, and Eran Halperin. Extend and explain: Interpreting very long language models. Antonio Parziale, Monica Agrawal, Shalmali Joshi, Irene Y. Chen, Shengpu Tang, Luis Oala, and Adarsh Sub- baswamy, editors, Proceedings of the 2nd Machine Learning for Health Symposium, volume 193 of Proceedings of Machine Learning Research, pages 218-258. PMLR, November 2022.
15

---

[73] Erik Strumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. The Journal of Machine Learning Research, 11:1-18, 2010.
[74] Erik Strumbelj and Igor Kononenko. Explaining Prediction Models and Individual Predictions with Feature Contributions. Knowledge and Information Systems, 41(3):647-666 665, December 2014. ISSN 0219-1377. doi: 10.1007//10115-013-07777
[75] Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9269-9278. PMLR, July 2020.
[76] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic Attribution for Deep Networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pages 3319-3328. JMLR.org, 2017.
[77] Mukund Sundararajan, Kedar Dhamdhere, and Ashish Agarwal. The shapley taylor interaction index. In International Conference on Machine Learning, pages 9259-9268. PMLR, 2020.
[78] Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti. Explainable Al for time series classification: A review, taxonomy and research directions. IEEE access: practical innovations, open solutions, 10: 10:1000001007224, 2022. doi: 10.1 100///CCESS
[79] Max Torop, Aria Masoomi, Davin Hill, Kivane Kose, Stratis loannidis, and Jennifer Dy. SmoothHess: ReLU network feature interactions via stein's lemma. Advances in Neural Information Processing Systems, 36, 2024.
[80] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[81] Giorgio Visani. Enrico Bagli, Federico Chesani, Alessandro Poluzzi, and Davide Capuzzo. Statistical stability indices for LIME: Obtaining reliable explanations for machine learning models. Journal of the Operational Research Society, 73(1)=11-101, 2022. doi: 10.1080/ 01605682.2020. 1865846.
[82] Jiaxuan Wang, Jenna Wiens, and Scott Lundberg. Shapley flow: A graph-based approach to interpreting model predictions. In International Conference on Artificial Intelligence and Statistics, pages 721-729. PMLR, 2021.
[83] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma. Junchi Yan, and Liang Sun. Transformers in time series: A survey, arXiv preprint 1xXi::2202.07125. 2022.
[84] Brian Williamson and Jean Feng. Efficient nonparametric statistical inference on population feature importance using Shapley values. In International Conference on Machine Learning, pages 10282-10291. PMLR, 2020.
[85] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: State-of-the-art natural language processing. arXiv preprint arXiv: 1XX0:1911000771, 2019.
[86] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048-2057. PMLR, 2015.
[87] Taedong Yun, Justin Cosentino, Babak Behsaz, Zachary R. McCaw, Davin Hill, Robert Luben, Dongbing Lai, John Bates, Howard Yang, Tae-Hwi Schwantes-An, Yuchen Zhou, Anthony P. Khawaja, Andrew Carroll, Brian D. Hobbs, Michael H. Cho, Cory MeLean, and Farhad Hormozdiari. Unsupervised representation learning on high-dimensional clinical data improves genomic discovery and prediction. Nature Genetics, 5668):1604-16 August 2024. ISSN 1546-17 1718. doi: 10.
16

---

[88] Yu Zhang, Peter Tiño, Aleš Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence, 5(5):726-742, 2021.
[89] Ziqi Zhao, Yucheng Shi, Shushan Wu, Fan Yang. Wenzhan Song, and Ninghao Liu. Interpreta- tion of time-series deep models: A survey, arXiv preprini arXiv:2305. 14582, 2023.
17

---

A Background
A.1 Societal Impact
As machine learning models are increasingly used for decision-making in high-impact domains [58, 15, 28, 87, 10, 9]. it is important develop explainable Al (XAI) methods to improve understanding of model predictions. XAI methods help to increase model transparency and reliability, which allows for informed decision-making when using the prediction models. The proposed method, OrdShap, addresses a critical gap in explainable AI for sequential models by quantifying how feature ordering influences model predictions.
However, we acknowledge that improved explainability tools could potentially be misused to reverse- engineer proprietary models reconstruct private information from models trained medical data. Additionally, like all attribution methods, OrdShap provides simplified approximations of complex model behavior that may create overconfidence in model predictions. Practitioners should these XAI methods to in conjunction with human judgment and combine them with other safeguards. For example, recent works have developed uncertainty quantification methods [68, 29, 16], as well as metrics to evaluate explainer stability [81], robustness [2, 35, 67], and complexity [30, 52, 7J. These additional tools can be used mitigate the possible negative societal effects of XAI.
A.2 Machine Learning with Electronic Health Record (EHR) Data
Many of the examples and experiments in the main text use electronic health record (EHR) data, which consist of clinical events that occur during patient's 's interactions with hospitals clinics. Clinical events can include medical diagnoses (c.g. ICD-9 codes), prescribed medications, medical procedures, laboratory tests. Each sample in the EICU and MIMICII datasets (see App- D) contains the history for an individual patient, which consists of lists of these clinical events over time. Recent works [59, 41, 55, 27, 31] have explored tokenizing these clinical events for use with Transformers in various prediction tasks. While there are a variety of approaches, most convert each unique clinical event separate token, which we refer Therefore, the processed EHR datasets in the main text consist of tokenized samples, where each sample represents separate patient, and each token represents clinical event.
medical tokens in the manuscript.
Below we provide examples medical tokens. The MIMICIII and EICU datasets include medications, laboratory tests, and infusions.
Medications:
Sodium Chloride 0.9% IV
Insulin Lispro (Human) 100 unit/mL
Dextrose 50% Solution
Hydrocortisone Sodium Succinate PF 100 mg
Magnesium Sulfate 50% Injection Solution
Laboratory Tests:
total bilirubin
platelets x 1000
WBC 1000 [White blood cell count]
MPV [Mean Platelet Volume]
Glucose
Infusions:
Normal Saline 20K
Epinephrine
Sodium Bicarbonate mL/hr
Propofol
Isoproterenol
18

---

(A) Tokenized Sequence Input
(B) Tokenized Sequence with Irregular Intervals and Grouping
Position
2
3
+
5
6
Pasition
3
9
Tokens
A2
X3
K4
Kg
%
Ag
Grouping
+6
Tokens
Ag
Xg
Position
+-8
Position
+8
Figure Extension OrdShap to alternative position indexing. (A) A sequence of features, presented as tokens. We assume that each token can be permuted to each position. For convenience, this is the assumed formulation in the main text. (B) Alternative sequence of tokens, with tokens separated into permutation-invariann groups and occurring independent, irregular time indexing.
A.3 Background: Shapley Value Approximation
Shapley-based methods are often prohibitively expensive to compute exactly in the general case; many methods been proposed to address this challenge [14]. Some algorithms take advantage of model-specific properties compute Shapley values, such as for linear models [43, 74], trees [44], or neural networks [43, 21, 3]. Model-agnostic methods generally sample number of subsets < 2 random. These stochastic approaches include permutation sampling methods [73, 74, 11] and also least-squares methods [12, 43). Several works have further investigated and improved both sampling [50] and least-squares approaches [16, 54, 84). An alternate line of work has also investigated Shapley approximation approaches for higher-order attributions [38, 24, 25].
A.4 Shapley Value Axioms [65]
Efficiency. ΣiEN 6; v(V).
Symmetry. Define Vn as the characteristic function v where the players are permuted according to
permutation G. For any player and permutation
GN (N, va) (N,, v).
Null Player, player is considered a null player in the game (N, v) if for every C N (i}. v(S) = v(S U {i)). The null player axiom states that for every null player i, CN v) = 0.
Additivity. Let v, be different characteristic functions. Let (v + ()( represent the character- istic function (SS) + i(S) AS C N. Then for any player i, 6, + D) = @. w) (N,
A.5 Sanchez-Berganttion Value Axioms [63]
Efficiency # EiEN (N,W)) TNT. Eoesy w(o).
Symmetry* Two players i.j are symmetric in the game (N,w) if w(a...) w(a.),) for all S C {i,j}, E Gg. and (1....,|S] 1}. The symmetry axiom states that for any symmetric players i, 4SB (N, = (N)
Null Player* A player is considered null player in the game (N,w) w) if w()) w(a for all S N (1) o E 6s. and / E (l...., | + 1}. The null player axiom states that for every null player i, d (N,))=O.
Additivity Let w.W be two different characteristic functions. Let (w + (o) represent the characteristic function w(o) + w(o) Yo S S N. Then for any player i, 0² (N, w + w)
(S, w) + 6gB (N,
19

---

Algorithm OrdShap Sampling Algorithm
Input: : Data sample x RR" Model f; Number Samples K,
Output E Ra
7 7-60
for E (I.....,d]
for E {1, (l...., L do
// Sample Permutations
Sample permutation 9i ~ U(GN)
for (1,.. {} do
// Sample Subsets
Sample permutation oı L(6N))
+ of
A - (5, (S) [i]. oz)
T (0) + % (1) A
end
end
end
Return TR
B
Additional Method Details
B.1 Generalization to Alternative Position Indexing
In the main However, in some applications, (1) features may be naturally grouped into cermuutatooninvariin or (2) features may occur at irregular time indices (Fig. 7B). For example, in EHR data, a sequence of patient tokens may be grouped into individual visits; in this case, user may want to enforce the tokens within each visit to be permutation invariant. Visits may also occur irregular time indices, therefore permuting tokens between visits should not always result in the same change in position.
assume that each feature be permuted any position in the sequence (Fig. 7A).
sets,
OrdShap can be extended accommodate these constraints while maintaining the same framework and theoretical properties. Intuitively, the position index used in the main text can be mapped to any other index, including time. Under this new mapping, for given feature, OrdShap-PI would represent position importance with respect to units of time rather than position index. OrdShap-Vi VI would correspondingly represent value importance with respect to the mean time of the sequence.
More concretely, let N {I..... [ be the original token indices and let be the grouped time indices (e.g., visit-level indices). We define mapping function : N - G' that associates each token with its corresponding group. As illustrated in Fig. 7B, we might have = (1,...,9) 9) and G = (1,3, 9}. We can then rewrite Eq. to account for the new constraints:
(IS) 1)111) ISD! (o(S,)) -5 -(S (1),))]
(N = Σ
Σ OEEN (()=
(15)
(IN) - T)!N
SCN
EES
Note that in Eq. e5 E G rather than E N from the original definition of therefore, this results in a x |G| matrix rather than d x matrix. addition, Eq. 15 still evaluates on each permutation GN however permutations within each group are effectively averaged together, resulting in desired within-group permutation invariance. Therefore, after this extension, the remainder of the framework and algorithm remain unchanged with position indices & G representing grouped positions rather than individual position indices.
B.2 OrdShap Approximation: Sampling Algorithm
In Algorithm 1, we present the sampling algorithm as described $5.1. We define a function e that takes an ordering E G and a position i (1.....N)) and returns the set of elements in that ordered before i; i.c. ij (G) < (i) *3.
20

---

Algorithm 2 OrdShap Least-Squares Algorithm
Input: Data sample I R²; Model f; Number Samples K, Output a E IR" , E R
A(a), W( wh,) plo), F(A) a. -
// Initialize variables
Sample permutations oi...., OKL ~ (()))) Sample subsets S1,..., SK U(P(N))
// Sample Permutations
// Sample Subsets
8 (0, E Et TNT ESeN1
// Compute the baseline
for E (L...., KL3 for j E (l...., d3 do A + 1l s: W., (()) -
end
W
p(S: mod 1.)
F. Wa. Ss mod L, oz) à
// Evaluate under permutations and subsets
end
for
E {1, {1,.... K} do
for E (1,...... d} do
A + 1s, (i)
end
w((a)
=(S))
((a) T, E1-11 +1
end
p(a)
pla) pla ((0)
// Remove feature enforce the constraint.
a2
< (A (A (w p(o)
// Eq. 13
an EE (NN, σi) (00 Σz=2a;
// Enforce constraint.
[ (1 plm)
// Eq. 14
Return 8
B.3 OrdShap Approximation: Least-Squares Algorithm
In Algorithm 2, we present the sampling algorithm as described $5.2.
21

---

C Theoretical Details
C.1 Proof of Permutation Non-uniformity.
We want to show that, for a given feature i, there exist positions e,e' E N such that:
Ha E Gs (1) - + Σ
1o e Gs (1) [')l
(16)
SCN
iES
Proof. We up proof by contradiction. Assume that the following equation holds for all e, E N:
Σ
e Gs: -())
Σ SCN
fo Gs: 0 [
(17)
SCN
iES
SES
Combining components in the summations:
Σ
[loe 6s (1) -1) - e : :0) =0
(18)
YES
We fix e = i, i.c. the position of feature i. We also fix l' = j for given j E N \ {i}. Therefore, we can rewrite the equation follows:
Σ SCN
l0 E Gs: : -0) = 0l - 10 Gs: (0) er)
IES
=0
JES
la E Gg 6
1o foo G5 :3:0 = =0 (19)
ESS
>0
Note that the first summation in Eq. reduces to zero, since the permutations within Gg where S contains both and j will uniformly permute feature i to positions and L' In the second summation, the first term is strictly positive, since i is always in the S. However, since & S, there exist no permutations a E Gg that permute feature to position f.. Therefore, the second term in the second summation will always be zero. This forms contradiction, which completes the proof.
C.2 Proof of Theorem 1.
Proof. We want to show that, given the value (NN, there exists corresponding characteristic function w such that (NV 5) = o. (N.w).
We first introduce the relevant notation. Given permutations æ E Gs and a E G for S C N, we define ((T, o) the number of pairwise disagreements between и and a:
((T, = (((((())(())) : =(1) < <(i) A > a())
(20)
Let I( = = GN <(T, = 0} represent the set all permutations
GN such that there are 0 pairwise disagreements between and Let T(T) (i E N: i E w) denote the set containing the elements permutation æ. We define the characteristic function w + R. where Ω the set of permutations Gg VS N:
w(T)) = ([T(T))) [N]!
Σ (T(()),)
(21)
Note that for given 6s. [[(NN, +)
The characteristic function represents w averaged over permutations in G with the relative order of elements æ fixed, and elements masked.
|N
TST
22

---

For given permutation E Gs, let a denote the permutation o with removed, i... o_; (oi,..., k-- 101...,,k-11... 1+-1.0k+1........ (ss such that o (i). Additionally, let tii represent or with inserted index i.c. + (o k- ...
We now begin the proof. From Eq. 8:
1 ISD! (=(5,)) - ((S)(o))))
(,) Σ Σ
(IN1 - 1)
SEN iES
(i)=t
Averaging over (I,...,,IN.. |N
N
IN|
1
Σ
(IS) - 1)1(N1 - ISD!
- (W(S. =SS (i).o)]
(22)
N
1=1
SCN CEÊN o (f)=t
(1S) 1 (INI))2
= EΣ
- -
(23)
SCN CEBN iES
Group the permutations in G that contain each permutation π E Ggg
(IS 1 1S
=EΣ E SCN iES
(0) - (S)) {))
(24)
([
(ET(GN)
Substitute the characteristic function with w:
(IS) 1) - (S))
(S1)!
EΣ> SCN mESS AES
[w(TT) w(T
(25)
Note that SCN
(S}) U
(i)) therefore:
SCNV()
(ISU 1IIINIIIIIII IS ISU (i)
Σ Σ
(w(x) - w(
(26)
|N U (
(CN((() xEG
IS S 1)!
-E SENV(D REGSU()
E
|N +1 [wT(T) - w(
(27)
Note that we can recover the set of permutations in G @suft) by taking each permutation G and inserting each possible index k, i.e.
S|+1
S
U FEBs Am
U
Therefore, rewrite Eq. 27 as follows:
3
Σ
Σ
(|N) |S| 1)!
| + 1) [w(TT).)) wT))
(28)
SCNV( +E
This recovers the Sanchez-Bergantiño value [63] for player i in the game (N,
23

---

C.3 Proof of Theorem 2
Proof. For a given characteristic function W. define the following corresponding characteristic function w, which was introduced in App. C.2:
(IT(T))) |N|!
((x)
E (T(()),)
(29)
GEM()N.)
We want to show that the optimal coefficients for the problem in Eq. 11 using (u)) and characteristic function ü are equivalent the SB values under w.
IN - (
We first set up the Lagrangian for the optimization problem in Eq. 11, with multiplier À E R.
L(a,, 3,)
Σ m(IS))
Σ 04
Σ iES
0 (1) - 2 3 - (00)5((55(0000000
SCN NEEN S40,N
LIES
Σ
((N,0)+3(0,0t)
aj
|
VEN
(30)
First order conditions with respect X.
0L
Σ a;
Σ (
(31)
|il
iN
dEGN
Σa
a.
IN Σ
a(N.)) 3(8.0%)
(32)
VEN
From Eq. 32, substitute the characteristic function w with w. We change to æ for notational consistency.
Σ
Σ w(T) w(Ta)
(33)
N!
TEEN
First order conditions with respect az-
dc = 2 Σ Σ M(S)))
Σ Eaj+
aj + Σ
- o) {(,
- -x=0
Dox
SCN JESN 1ES8 SøN
jES
jES
(34)
- Σ Σ p((S))
Σaj aaj
Σ 0 (G) - Fl B, (5(S -100) [
(35)
2
SCN OEEN IES SAN
JES
JES
From Eq. 35, separating the components related to 3y. Note that EoEG (0 (1) E| Aj E.
2 ト M Σ m(SS))
Σ a, (o(S.o) ((65,)) ((0,
SCN GEGN iES SyNN
JES
Σ
M((S)) Σ B Σ
o (o) (j) E]
(36)
SCN IES
0
24

---

Group the permutations in G that contain each permutation π E Gs.
; X E SC N REB o IES SYN
m(SS))
Σ oj ( - -0(,,
(37)
ト ー - ((SS))
M || JES
Σ
3j
E
(S, -00,0())))))))
(38)
SCN 8 SESS SPANN
OEI(GN.M)
Substitute the characteristic function w with w.
} - M
NI |||
((S)
Σaz ay - [w(x) - --
(39)
2
SCN TE IES S
JES
Sum both sides of Eq. 39 over e N and dividing by I.N).
Σ
[N
((SS))
Σ jES
aj (w( ( -
(40)
N
S!
EN
SCN 4EG3 iES VEN
SYN
1 = Σ
(INI-19) - 1)!
(((S)) Σ
aj
((SS) [w(x) w(+
(41)
|S|!
SCN REGs JES SFN
jES
We separate Eq. 41 into two components, 1 and which we will examine separately.
First, we take
excluding the outer summation ΣIEN.
(INI 1)!
-- SCN IES
((SS)) Σ
aj
(42)
SS!
JES
SYN
Change the outer summation be subsets of size
|N|-1
Σ
(IN) 1)! p(S))) Σ ag
(43)
JES
IES
|S
Separate the terms and] j + i.
IN|
Σ
(IN) - -
((S))
as + Σ aj
(44)
SCN
JESI(U)
IES -
-
NN - (
NN
(N - [IN--2 (NN)
(IN) - 1)4m(s)
+
Σ
Σ #=2
(NNI
- a
(45)
-
JEN
terms involving feature
terms involving features jztt
25

---

Separating and rearranging the terms involving feature i. Note that (1--T)) = =(W-T3 + (1-22)
-111--1) - 1)!
-
Σ
(s))
[
-2)
Cii +
M
Σ
mss)
[ -
2 az
2
8-1
N
(IN| -2
+ Σ Σ
M(s)
(46)
8 1-2 2
(()
s=2
Combining terms ( aa and (!N1129 Jaj.
Σ m(s) (1N1- -2)
INI-1
[ - m(8)
= -IN] - 1}!
ox3 +
Σ Σ
oj
(47)
1611
- 2
JEN
2=2
Note that EJEN Oj right-most component of Eq. 47.
1
E
N w(T) - ((T) from Eq.
33. We can substitute this in the
N
IN
-I1V] 1)!
Σ
u(s)
| - 2)
1
a +
Σ
mss)
IN
-2)
s s-2
|AV
1
Σ
w(T) ((
e=2
(48)
We next sum Eq. 48 over all features i E N (the outer summation in Eq. 41).
(INI - 1)!
EΣΣ iEN SCN TEGY IES SFN
((S)))
Σ
SS
=
JES
-1
(NN
|Nl-1
(IN) -22
[
Σ (INI-1))
Σ u(s)
0 +
Σ
Mss)
Σ w(T)) - w(
-
2
EEN
2=2
TEEN
(49)
-N-1
IN1-1
[IN
11.7 - 2)
- ))
1
Σ
()N1))){u(s)
8 s-1
Σ IEN
ac NN!!
Σ mls)
Σ
((())
|I
(50)
Again, plug in Eq. 33 for EiEN ai.
= Σ (NN- - 1)(u(()) 8 1
(1 (IN1-2
1 Σ -
II
1=
EEGN
INI-1
( 22
+ IN
Σ
m(s)
Σ - w
(51)
- 2
|N|I
2=2
xEGx
-1
(l - 2)
IN
((NNI-2) 2)
2)
| Σ w(T)- w(T
= (IN1 - 1)!
Σ mss)
+ I
Σ
m(s)
-
8 - 2
INT
2
TEBN
(52)
IN1-1
( -11)
[IN
= =(LV - 1)) m(1)+ + Σ splss
Σ w(T)) - -
(53)
2=2
N
26

---

(IN1 - 1)! |M
Σ )μ(s)
[
8 6-11 -1
1
Σ w(()) ((T))
(54)
TEEN
Next, we take component
(| 1)!
2 -E
U(IS)) (w) ((T))
(55)
|S|!
KENSCL V TEB3 EES SYN
ΣΣΣ
([N - 1)!
1}! m((()) [w( ((T
(56)
SCN xEGg 8
Substituting the results of 1
2
into Eq. 41:
WN-1
2 = (IN1 - 1}!
1
Σ (pihs
[
1
1
Σ
w(x)) w(=b)
- 1
Ni
==1
([N 1)!
Σ
(|S) 1)! ((SS (W<(T) - w(=)) (57)
SCN TEBS
S사0,N
Set the RHS of Eq. 57 equal to the RHS of Eq. 39
ΣΣ
[N!! ils (((s))
Σ 0r [w( - (m
SCN T iEs S
JES
Eq.
|
( 1!
= [ - 1}! Σ sp(ss
NI Σ w(T)) w
TEGN
(|NI-1) - 1)!
Σ
[[S--1) 1)! (((S))
[W<(T) w(+))
(58)
SCN TEOg
S
27

---

Separate the terms in the LHS of Eq. 58 and substitute in Eq. 48.
|N 11
(N - 2
|M
(INI-
(IN - 2)
- 2 )
[ 1 Σ
(IN) - 1)!
Σ mss)
ai +
Σ mss)
w(T) - w
- 1
|N
s
MEGN
Eq. 48
|N|
Σ Σ
|i!! ((|S))
- -
SEN xEG
IES SYN
1
(IN
[
1
= -INV - 1)! Σ su(ss
Σ w(x) w(
MEGN
(LVI 1)!
Σ
([S) [||-1)! 1)! (((S)) [W0(TT) - w(+)) (59)
SCN 7E6y SyØ,N
Rearrange and combine terms.
IN - 2)
NN
Σ m(s)
1 )
IN
N
(1N7 -1 1 Mi
IN
| | 2)
- 2 )
Σ
su(s}
Σ a(s)
Σ ((T)) w(T
IN
IN TEG
x=2
IN!!
(IN1- 1)!
+
Σ
Σ
((S))
[W(TT] [(+x)]]
Σ
Σ
(1S) 11) (((S)) [W((TT w(T
S||
SSCN S
SCN #g6g
IES S
S+2.N
(60)
III
[NN 2) s s-1 1
-2)
Σ m(s)
ai
1
-I
Σ
sp(ss
(INI-1)
-
|i
Σ m(s)
| | -2
8 - 2 )
| Σ w(T) - (
|N
G
1
((S))) (w(
--+r)
Σ Σ
|N[([S]- - 13. (IS1)) [w(TT] - [w(())
|S
LSCN 8
SCN TEGS
IES
S
SAN
(61)
28

---

Solve for as..
1
Σ w(T)) - w(T
N
MI!
TEÊN
IN
- 1
| -
+
Σ
m(s)
Σ
|S| ((() (w( w(Tr
s=1
LSC.N TEG3 iES
SyNN
Σ Σ
|N[[S| 1)! ((S) [W((T) w(T
(62)
SEN
xc8g
$48,N
3
In 3 separate the subsets S C N where E S and i & S, and the subsets = N {t).
3
Σ SCN 8
Σ
[N[([)) - 1)! M(SS)) (W(T) (x+6)]
(63)
xE63
M(S)
[([S 1) [N[(|] 2) [w(TT) - [(T
Σ L
NN([S) - 1)! (W(T) w(TH)]
SCN
iES
(*
Subsets with iES
Subsets with ig.
M([N - 1)
μ(1)
E
NNINNI 2)! (W(TT) - w(TT))]
(w(T) - w(x))) (64)
[N|
TEG
S=NYO
S=(()
Substitute T....
m((S))
M(SS - 1)
Σ
[N[([S) - 1)! [W(T) w(TB)]
[w(TTs) - w0( (T
| (1S) 2)!
SCN x iES (7),
M(NI- -)
[w(T=:)- w(Tr) + (d1)
Σ
Σ
(w(TT) - w( (65)
[ - 2)!
|N
TEEN
Substituting the results of back into Eq. 62.
N
a, =
N
1
NII
Σ
w(T)
((T)))
+
Σ m(s)
NN - 2
1 )
x
TEEN
|
|S|
|S| -
Σ
Σ
INISS! ((SS)) (W((T) w(mr0))
|N||s!! M(SS) 1) [W(T w(T))
SCN IES 87
M(IN- - 1)
[w(TT>) wxx + (|NN - 1)
Σ
N ((1)
Σ
(w(T)) - ((T (66)
INN 2)!
TEBx
TE
29

---

Eq. 66 provides general result for a given weight function μ No - Rt. To complete the proof,
we ((s) = प (--
1 N
Σ
w(T) - w(T))
=
MI
TEGN
[w(T ((T))
+
Σ
[T(T) - w(T
N x(
TEGK
(IN) - ())(((S) 1)!
Σ
(w( - w( (67)
|!!!!!
SCN
HE6s
iES S
Σ
(W(TT w(T=i) +
Σ
(w(T) - w(T=:))
NI TEBN
N
(IN) (S))((SS) 1)!
Σ
Σ
(w( - w(x (68)
!N!!!!!
SCN
HERg
iES S
ΣΣΣ
(IN) - |SD!US - 1)!
(w(TT) w(T=))]
(69)
INSS!!
SCN REGS iES
Note that this result equivalent Eq. 25 in the proof of Theorem (App. C.2). We follow the remaining steps in App. C.2 to complete the proof.
L
C.4 Proof of Corollary 2.1.
We first restate Remark 3 in Sanchez. and Bergantiños [63] using our notation.
Remark. Given a game (N,w), Nowak and Radzik [53] defined the "averaged game" (N, 7) as
[ST Eoes, o) AS C N. Then ((N, D) 5B (JN(N,w)
As we established in Theorem 1, (. (N,ww) Therefore, it follows that Remark 3 in Sanchez and Bergantiños [63] holds for (N)))..
D Experiment Detail
D.1 Datasets
MIMICII [34]. The Medical Information Mart for Intensive Care (MIMIC-III) dataset contains health records for over 60,000 intensive care unit (ICU) stays at the Beth Israel Deaconess Medical Center from 2001-2012. Each patient's stay is represented as sequence of codes that include laboratory tests, medications, and drug infusions. We follow the preprocessing in Hur et al. [31], using the provided public repository To summarize, create dictionary using all codes, excluding codes have less than occurrences in the dataset. For codes that contain numerical component (e.g. drug dosage), we exclude the numerical component reduce the number of unique codes. We then pass the dictionary to the HuggingFace tokenizer and tokenize each patient's 's stay using the dictionary, taking only the codes from the first 12 hours of the patient's stay, and limiting the total sequence length to 150 tokens to reduce computational cost. We split the patients into a training set (80%) and test set (20%). After preprocessing, we train modified BERT model [19] using the
Phttps:// //tp://b//tttbb cot.s////co00044///e00t2t
30

---

Huggingface library [85] with 6 attention heads, 3 hidden layers of width 384, and a dropout rate of 0.5.
EICU [57]. The EICU Colaborative Research Datahase is multi-center observational study for over 200,000 ICU stays in the United States from 2014-2015. We use the same preprocessing and model training steps as in MIMICIII but with different dictionary.
IMDB [45]. The Large Movie Review Database (IMDB) contains 50,000 written movie reviews. We use a DistilBert Sequence Classifier [64] to predict positive or negative sentiment. The pretrained model is loaded using the Huggingface library [85]. We then apply XAI methods infer the importance each sentence towards the sentiment prediction. We use the NLTK tokenizer [8] to separate each review into sentences. For perturbation bassdd methods (e.g. KemnllSHA, LIME, OrdSHAP), we pass the individual sentences as "superpixels" for which attributions are calculated. For gradient-based methods (DeepL.FFT, IG) average attributions over each token and sentence.
D.2
Explainers
Local Interpretable Model-agnostic Explanations (LIME) [60]. LIME explains individual predic- tions by training surrogate linear model locally for each instance be explained. It perturbs the input data around the sample, weighted by the distance to the original sample, then fits a linear model. The coefficients of the linear model are used as the feature attributions. We use the implementation from the Captum library [37] in our experiments. To ensure fair comparison, we use the same baseline as OrdShap (i.e. the mask token) and disable feature selection, but otherwise use the default parameters.
KernelSHAP (KS) [42]. KerneISHAP is a model-agnostic method for approximating Shapley values. It builds off the LIME least-squares approach by selecting weighting that corresponds to Shapley value approximation. We use the official implementation of KemelSHAP from the SHAP library 4 We use the same baseline as OrdShap (i.e. the mask token) and disable feature selection.
Leave-One-Ce Oww--Oneerirrt--OOnae (LOCO) [40]. LOCO measures feature importance by comparing model's prediction with and without a particular feature. For each feature, LOCO removes or randomizes that feature while keeping all others unchanged, then measures the resulting change in prediction. We use the implementation from the Captum library [37] in our experiments.
DeepLift (DL) [66]. DeepLift is gradient-based approach that attributes changes in neural network outputs to specific input features. We use the implementation from the Captum library [37] in our experiments. For gradient- based methods, average the attributions over each token embedding.
Integrated Gradients (IG) [76]. Integrated Gradients assigns importance features by calculating gradients along straight- line path from a baseline to the input. We use the implementation from the Captum library [37] in our experiments. The baseline is set be the embedding for the mask token. For gradient-based methods, we average the attributions over each token embedding.
D.3 Metries
Insertion / Deletion AUC (IAUC/ / dAUC) [56] Insertion/Delltion AUC evaluates explanation quality by measuring changes in model output as features are progressively added or removed. iAUC calculates the area under the curve when pixels or features are added in order of importance, while dAUC measures the curve when they removed in order of importance. Higher iAUC and lower dAUC indicate better explanations, as they show that the most important features identified have the greatest impact on the model's 's predictions.
Inclusion/ Exclusion AUC (incAUC/ excAUC) [33] contrast to iAUC and dAUC, Jethani et al. [33] evaluate the constancy of the masked model outputs to the original model output. This can be measured using post-hoc accuracy, which is the accuracy of the masked prediction compared to the original prediction. This post-hoc accuracy metric has also been used in other works [46]. We would expect that masking the most important features from the original sample would lead to a larger drop in accuracy (excAUC). Conversely, including the most important features in masked sample would lead to a larger increase in accuracy (incAUC). We then calculate the area under the resulting curve for both metrics.
'https:/ ttatts://githuu www.coo///ssa.ccce shap/shap
31

---

Table 4: The associated AUC values for the curves shown in Fig. 4, with standard error parentheses. The best results are bolded.
Method
EICU-LOS
EICU-Mortality
MIMICII-LOS
SOMICIII-Moralin
IMDB
DI 0.777 (0.015)
0.724 (0.023)
0.759 (0.012)
0.779 (0.011)
0.869 (0.010)
IG
0.779 (0.014)
0.746 (0.021)
0.764 (0.012)
0.781 (0.011)
0.874 (0.009)
KS
0.780 (0.014)
0.752 (0.021)
0.757 (0.012)
0.782 (0.011)
0.870 (0.010)
LIME
0.774 (0.013)
0.792 (0.018)
0.758 (0.012)
0.783 (0.011)
0,862 (0.010)
0.783 (0.014)
0.775 (0.021)
0.756 (0.012)
0.783 (0.011)
0.870 (0.010)
Random
0.761 (0.012)
0.781 (0.018)
0.752 (0.011)
0.783 (0.011)
0.870 (0.009)
OrdShup-Pl
0.885 (0.005)
0.963 (0.007)
0.809 (0.010)
0.808 (0.010)
0.883 (0.008)
DespUit
--
KemelSHAP
LIME
---
LOCO
Random
Ordshap-Vi
EICU MCUUMMrtallity
MIMiCIll-Mrrtlity
EICU-LOS
MIMICIII-LOS
IMDB
Post-Hoo Accurity
0.9
0.9
0.8
0.6
0.7
0.8
0.5
1.0
0.0
0.5
1.0
0.0
0.5
0.5 Features included
3.0
0.0
0.5
Features Included
96 Features Included
16 Features Included
95 Features Includd
EICU-Mortality
MIMICIlI-Mortalii lity
EICU-LOS
MIMICHI-LOS
IMDB
Pos Acouracy
0.9
0.50
0.8
0.78
0.8
0.85
0.78
0.6
0.00
0.7
0.74
0.6
0.75
3.6
0.72
0.5
0.5
0.5
Figure 8: Evaluation of OrdShap-V VI using Post-Hoc accuracy when iteratively unmasking features from a baseline sample (Top, higher better) and masking features from the original sample (Bottom, lower is better). The AUC values for these curves are presented in Table 2.
E
Additional Results
E.1
AUC Metrics
Position Importance. AUC and standard error results for Fig. 4 are shown in Table 4.
Value Importance. The curves associated with Table 2 shown in Fig. 8 for reference. Additional iAUC and dAUC results are shown in Table 5, with standard error values shown in Table 6.
E.2
Additional Synthetic Dataset Results
In Fig. 9, show the results of OrdShap-VI and OrdShap-PI synthetic dataset with linear model.
E.3
Sensitivity Analysis
In this section, we investigate the sensitivity of OrdShap-PI sample size in the Least-Squares algorithm approximation. We recalculate the position importance metric in $6.1, varying the number of subset samples K and permutation samples L, then calculate the resulting standard errors. Results calculated on 100 explanation samples from each dataset, and shown in Table 7. We observe that the standard results relatively stable across different sampling configurations.
32

---

Table 5: Insertion AUC and Deletion AUC metrics for evaluating OrdShap- VI. bolded. Standard error values are provided Table
The best results are
Metric
Model
DL
IG
KS
LIME
LOCO
Random
OrdShap- VI
BICU-LOS
0.689
0.690
0.690
0.672
0.692
0.655
0.695
EICU-Mortality
0.678
0.680
0.679
0.669
0.680
0.664
0.682
Insertion
MIMICHI-LOR
0.692
0.695
0.694
0.687
0.692
0.661
0.696
AUC
MIMICIII- Morttlllyy
0.697
0.701
0.700
0.677
0.697
0.633
0.702
IMDB
0.719
0.730
0.794
0.790
0.785
0.736
0.797
EICU-LOS
0,611
0.611
0.610
0.643
0.608
0.661
0.597
HICU-Mortality
0.646
0.645
0.646
0.655
0.647
0.661
0.643
Deletion
MIMICIII-OS
0.624
0,613
0.617
0.644
0.620
0.666
0.608
AUC
MIMICIII MMriiall
0.523
0.518
0.520
0.581
0.527
0.626
0.513
IMDB
0.789
0.793
0. 7.3
0.735
0.722
0.774
0.717
Table 6: Standard error values for Insertion AUC and Deletion AUC metrics.
Metric
Model
DL
LIME
LOCO
Random
OrdShap-V
FICU-LOS
0.009
0.008
0.008
0.009
0.008
0.009
0.009
EICU-Mortality
0.011
0.011
0.011
0.012
0.011
0.013
0.011
Insertion AUC f
MIMICII-LOS
0.008
0.008
0.008
0.008
0.008
0.009
0.008
MIMICIII-Mone -Mortality
0.008
0.008
0.008
0.009
0.008
0.011
0.008
IMDB
0.012
0.012
0.009
0.009
0.010
0.011
0.011
EICU-LOS
0.011
0.011
0.011
0.009
0.011
0.009
0.012
EICU-Mortality
0.017
0.017
0.017
0.014
0.016
0.013
0.018
Deletion AUC 4
MIMICIII-OO
0.009
0.009
0.009
0.010
0.009
0.009
0.009
MIMICIII -orttlltyyy
0.016
0.016
0.016
0.013
0.016
0.011
0.016
IMDB
0.010
0.011
0.012
0.012
0.014
0.012
0.015
(A)
Token t
Value vtt:)
VI
Pl
Legend
(B)
OrdShap Attributions
(C)
00
10
A
0.18 +
/
O
Artribumtn Vallues
B
0.18
/
/
A
C
2
/
X
D
0
X
X
A,
-(A,BC)
1
/inear (1) = E (14)
KS
LIME LOCO
Figure 9: Attributions for a synthetic dataset and model (finear. (A) Token values; tokens are assigned Value Importance (VT) and/or Position Importance (PI) with respect sequence index (B) OrdShap- VI and OrdShap-PI attributions are able to separate the different tokens based on VI and PI effects. (C) contrast, attributions from existing methods cannot distinguish between the different tokens, since VI and PI effects are entangled.
Table 7: Sensitivity analysis of OrdShap-P PI sample size in the Least-Squares algorithm approximation. Standard errors are calculated for different combinations of subset samples K and permutation samples L for the position importance metric described in $6.1.
EICU
MIMICIII
Permutation Samples
Subset Samples
LOS
Mortality
LOS
Mortality
0.0046
0.0117
0.0100
0.0107
10
50
0.0046
0.0117
0.0100
0.0107
100
0.0046
0.0117
0.0100
0.0107
10
0.0046
0.0120
0.0099
0.0100
50
50
0.0046
0.0120
0.0099
0.0100
100
0.0046
0.0120
0.0099
0.0100
10
0.0046
0.0118
0.0103
0.0099
100
50
0.0046
0.0118
0.0103
0.0099
100
0.0046
0.0118
0.0103
0.0099
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims in the abstract and introduction properly supported in the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in $7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
34

---

Justification:
All proof details are provided in App. C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All experiment details are provided in App. D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification: All datasets are publically available, as described in App. D. All source code will be provided for the review process. Public release of the source code is contingent an internal review process and will be released if/when possible after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (uttps://nips. cc/ public/gue welggggeeceeeuuuaeeee
cessinnnlycyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results, See the NeurIPS code and data submission guidelines (https: //nips. cttpps..ccpbbbicc///c hwwc//ggisssseeeeeeeeeeeeeeeeese CodeeSummiss setommmssionn magiooooiccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experiment details are provided in App. D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes [Ye]]
Justification: Error bars for the quantitative experiment results are provided in App.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
36

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are detailed in App. D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics tttt:s:///eurip... httpp://harrps...cppbbbt////t/t cww.iiesscllllleccc lnes?
Answer: [Yes]
Justification:
This research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts are discussed in App. A.I.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used are publically available and properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
39

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https for what should or should not be described.
40


Reviews:
[{"review_id": "x7dfpoatMD", "reviewer": "Reviewer_sQ2K", "summary": "in this paper the authors introduce OrdShap, a novel feature attribution method designed for sequential deep learning models - these often require understanding of both feature values and their positions within input sequences and OrdShap proposes a novel approach to tackle this. Traditional attribution methods assume a fixed feature order, conflating the effects of feature values and their sequence positions. OrdShap addresses this problem by quantifying how model predictions change when feature positions are permuted, thus disentangling the effects of feature values and positions which ultimately shows superior performance. The method is grounded in game theory, specifically relating to Sanchez-Bergantinos values (which the authors do a good job of introducing and motivating) and provides a structured representation of feature importance based on position. Empirical evaluations on health, natural language, and synthetic datasets demonstrate that the proposed method, OrdShap, is effective in capturing both feature value and position attributions, offering deeper insights into model behavior. The paper also proposes efficient algorithms to approximate OrdShap, addressing computational challenges inherent in Shapley-based methods. The experimental evaluations show benefits in terms of both interpretability as well as superior performance to several benchmarks.", "questions": "Please see strengths/weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is very well written and the motivation is clear. The authors give a nice introduction and the technical preliminaries section is well written. One thing I really like about this work is that authors are very pedagogical about their approach and explanations. There is consistently a rigorous theoretical explanation and alongside it a intuitive explanation or a motivating example showcasing what it means. The sanchez and Bergantinos values are also well motivated and if it is the case that they have not been investigated in XAI literature (in which I am not well versed but could not find many works online combining the two), I think the idea is neat and clear. \n\nRegarding the introductory example of predicting LOS, could the authors give an example of the \"medical tokens\" and how does it look in practice, maybe in appendix in a small section? Although I really appreciate the figure 2 and it made the example much clearer, I think this would be the cherry on top that would make the example perfectly clear. For many of us who have been working with LLMs the word token and its meaning is now a bit contrived, I think this would possibly clear any remaining confusion that the reader might have. \n\nI would once again like to emphasize my satisfaction with the authors' ability to straightforwardedly explain the concepts, as I found the Table 1 and the example given with Bag, Gloves and Hats very intuitive and nice to motivate the need for using OrdShap. \n\nAlthough the method is clear, the question of computational expense naturally arises but the authors did a good job of both presenting it as the technical limitations of the paper and did a good job in producing sampling and least-squares approximations. I am not too sure if the least-squares approximation can give any downsides and what are its weaknesses, and I would appreciate if the authors could comment a bit more on this but in general I was satisfied with both approaches that authors proposed (5.1 and 5.2). Regarding experiments, I was convinced with the benefit of using OrdShap and was happy that authors gave execution time results as well as presented their details so detailedly. However, looking at Table 7 in the supplementary material I was a bit surprised with how much slower the OrdShap methods are and I think that the authors should have highlighted this in the main paper as it is extremely slower. Although it is mentioned in the limitations and conclusion section, the authors a bit misleadingly mention that they have mitigated this limitation through sampling and approximation. Also, I think that the significance results should be reported in the main paper and not the supplementary material - especially as (unless I am mistaken), most of the results for OrdShap-VI in the e.g., Inclusion AUC are not statistically significantly different to the performance of second and third best performing method (e.g., IG and KS). \n\nOverall I think it is a well written paper that contributes to XAI with clear motivation and applications but there are few weaknesses that I think the authors should address and comment upon. I think that the authors open a nice alleyway for algorithms that can disentangle value and position importance of each feature in a sequence and introducing the SB values is definitely something that can be important/useful for XAI, but after examining the appendix and the statistical (in)significance of some results that come at an expense of much longer computation and execution times, I was reluctant to give a larger score.", "was_revised": true, "final_justification": "The authors gave clear examples and added more experiments which answered my questions.", "rebuttal": "Thank you for your thoughtful feedback and for appreciating the clarity of our presentation and the novelty of introducing Sanchez-Bergantinos values to XAI. We are glad that you found our examples helpful. We have provided responses to your concerns below, which we hope will help provide additional clarification. Please feel free to ask any other follow-up questions. \n\n---\n\n**Q1: Examples of Medical Tokens**\n\nWe appreciate the suggestion of including additional explanation and examples of medical tokens. In our experiments and examples we use electronic health record (EHR) data, which consist of clinical events that occur during a patient’s interactions with hospitals or clinics. Clinical events can include medical diagnosis (such as ICD9 codes), prescribed medications, medical procedures, or laboratory tests. Each sample in our EHR datasets contains the history for an individual patient, which consists of lists of these clinical events over time. Recent works (e.g. [1,2]) have explored tokenizing these clinical events for use with Transformers in various prediction tasks. While there are a variety of approaches, most convert each unique clinical event to a separate token, which we refer to as medical tokens in the manuscript. Therefore, the processed EHR datasets in our paper are tokenized samples, where each sample represents a separate patient, and each token represents a clinical event.\n\nBelow we provide some examples of medical tokens. The MIMICIII and EICU datasets include medications, laboratory tests, and infusions.\n\nMedications:\n* Sodium Chloride 0.9% IV\n* Insulin Lispro (Human) 100 unit/mL\n* Dextrose 50% IV Solution\n* Hydrocortisone Sodium Succinate PF 100 mg\n* Magnesium Sulfate 50% Injection Solution\n\nLaboratory Tests:\n* total bilirubin\n* platelets x 1000\n* WBC x 1000 [White blood cell count]\n* MPV [Mean Platelet Volume]\n* Glucose\n\nInfusions:\n* Normal Saline 20K\n* Epinephrine\n* Sodium Bicarbonate mL/hr\n* Propofol\n* Isoproterenol\n\n\nWe will add this additional clarification to the revised manuscript.\n\n\n---\n\n**Q2: Downsides of the Least Squares Algorithm**\n\nThank you for raising this point. The main limitation of the Least Squares (LS) Algorithm compared to the Sampling algorithm is that it only approximates the summary metrics OrdShap-PI and OrdShap-VI, and not the full OrdShap values $\\gamma_{i,\\ell}$ (Eq. 8). Users who need to calculate the full OrdShap values, such as for visualization (as in Figure 3B) or for analyzing nonlinear position effects, would need to use the Sampling algorithm. Since many users primarily need the summary metrics for interpretation, this tradeoff is often worth it.\n\nA second, minor limitation is that the LS algorithm always solves a weighted least squares problem with $K \\times L$ samples and computes attributions for all features simultaneously. For very small/fast models, or when only a few features need attribution, the Sampling algorithm could theoretically be more efficient. However, in most applications the LS algorithm is generally faster than the Sampling algorithm.\n\nWe will add this discussion to the revised manuscript to improve clarity.\n\n---\n\n**Q3: Time Complexity Concerns**\n\nWe appreciate the feedback regarding time complexity. We placed the empirical time complexity results (Table 7) in the supplement due to space constraints. We are happy to move these results to the main text in the camera-ready revision, which allows for an additional page of content.\n\nOur claims of efficiency improvement are with respect to the naive implementation of Eq. 8, which is generally infeasible for any non-toy dataset. Our proposed Sampling and LS algorithms make OrdShap values computationally feasible in practice. We also highlight that this work is the first to investigate this notion of position-dependent attributions, the other methods presented in Table 7 are not able to capture this type of attribution.\nUltimately, capturing position-dependent effects requires evaluating the model under different permutations, which is computationally expensive. This computational cost can be justified in applications where the positional effects are important to understand. \n\nWe will make this time complexity limitation more clear in the revised manuscript to avoid misunderstanding.\n\n\n\n---\n\n**Q4: Statistical Significance of Results**\n\nSimilar to the previous response, we would be happy to move the standard error results to the main text, given the extra space for the camera-ready revision. We acknowledge that some of the OrdShap-VI AUC scores are not statistically significant. We highlight that OrdShap-VI still performs competitively compared with the top-performing traditional attribution methods. While these other methods still perform relatively well in generating attributions for *value importance*, they are unable to disentangle and provide the effects of *position importance*, which we are able to do with OrdShap-PI."}, {"review_id": "7BPZu14k2b", "reviewer": "Reviewer_fm2z", "summary": "This paper develops OrdShap, a method for attributing predictions of sequential models to both the value and the position of each input feature. Existing attribution approaches typically assume a fixed feature ordering, which makes it difficult to separate the attribution of a feature's value from its position in the sequence. OrdShap addresses this by quantifying how predictions change when the positions of features are permuted.\nThe method incorporates permutations with the familiar ablations for Shapley values, which the authors connect to Sanchez-Bergantiños values from game theory. It produces two scores per feature: one for its value (OrdShap-VI) and one for its position (OrdShap-PI). The paper also proposes efficient approximation algorithms for computing these scores, using random draws of permutations and feature subsets, and a least squares formulation to calculate attributions for all features at once. Experiments on healthcare, language, and synthetic datasets show that OrdShap can distinguish between value and position effects in model predictions.", "questions": "To expand on my comment from \"weaknesses\":\n\n1) Many models use inputs such as positional encodings to denote a feature's place in the sequence. Why is a simple baseline that adds an input with the position of the feature and runs shap on this extended set of {features, positional inputs} not applicable to the problem? If it is applicable, should the paper be comparing to this?\n2) While order of features is an important aspect, isn't it that in most applications, such as the medical example given in the paper, the timing of the feature is important and not just its order? That is, we would have a tuple (t, x) with the time of a medical event/token, and its value, and what matters is not only if a medication was given before/after some token like a lab test, but how much after/before. The paper mentions irregular intervals, says the method can be extended, and points to the appendix. But from the appendix I did not really understand how this extension to such attributions is performed and what properties it has.\n\nSince I am unfamiliar with the literature on methods like TIME and PoSHAP, I would be glad to learn how these works and the current one under review address these questions.\n\nSome smaller comments:\n1) Figure 1 is not very easy to understand for a reader that starts reading the paper. It is unclear why the raised predicted probability is a desired outcome and what is it an alternative to. A figure with a clearer story may be helpful for many readers that go through the intro.\n2) Section 2 says \"While these attributions methods can be applied to sequential models, most assume feature independence\". What does \"independence\" mean here? Is it statistical independence, or some kind of other independence? I guess that KernelSHAP has some independence assumptions, but I was not aware that most shapley-based baselines make statistical independence assumptions.\n3) According to theorem 1, OrdShap-VI has the desirable properties that are common in Shap based methods, does OrdShap-PI have any such guarantees as well?", "limitations": "The authors have appropriately discussed the limitations of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and clear. It studies an important problem, and while I am not an expert on attribution methods, it seems like the problem has not been properly addressed before.\n\nWeaknesses:\n\nI think that some motivation for the necessity of the specific permutation-based method would've been beneficial. I will clarify what I mean by that under the \"questions\" part of the review.\nAnother small comment is that the evaluation of the order attribution is not entirely clear to me. While the evaluation with Inclusion and Exclusion AUC is pretty intuitive and common for the value attribution, I think the evaluation with permutations at section 6.1 should be better motivated and explained. There is no scalar metric compared between ordshap and and the baselines, and it is also unclear how one should compare it with methods that do not attribute the order at all.", "was_revised": true, "final_justification": "Please see my comment to the rebuttal for the final justification.", "rebuttal": "Thank you for your thoughtful review and for recognizing the importance of the problem we address and the clarity of our presentation. We appreciate your constructive questions about the motivation for our permutation-based approach and the evaluation methodology. We have provided detailed responses below that we hope will clarify these points. Please feel free to ask any other follow-up questions. \n\n---\n\n**W1: Clarification of the order attribution evaluation metrics**\n\nThe goal of Figure 4 was to evaluate how well OrdShap-PI can capture positional importance. Intuitively, the magnitude of OrdShap-PI answers the question of model sensitivity to permuting the given feature; i.e., if the feature is moved to the beginning or end of the sequence, how strongly does the model output change? The positive/negative directionality of OrdShap-PI indicates whether moving the feature to the end increases the model output (positive OrdShap-PI) or vice versa.\n\nTo test this property, we took inspiration from the more standard Inclusion AUC metric for value importance. That is, we first sort OrdShap-PI values based on magnitude. Then, starting from the original sample, we selected an increasing percentage of features (the X-axis in Figure 4) and permute the features to either the beginning or end of the sequence, depending on their OrdShap-PI sign. Similar to Inclusion AUC, we would expect the curve to increase quickly for well-performing attributions.\n\nWe follow a similar procedure for competing methods. However, there is ambiguity regarding whether to permute features to the beginning or end of the sequence; we tried both methods and presented the best-performing approach (end of the sequence). We will include an additional plot in the revised manuscript that shows results for the alternate approach (permuting to the beginning of the sequence) to make this more clear.\n\nAs we observe in Figure 4, OrdShap-PI performs very well in capturing positional importance. Additionally, as expected, competing attribution methods do not capture positional importance and therefore perform relatively poorly. This experiment also shows that simply applying existing attribution methods is not sufficient if a user wants to understand positional importance.\n\nRegarding scalar metrics, we can convert Figure 4 to an AUC value, which we show in Table 3 in the supplement.\n\nWe thank the reviewer for identifying this point of ambiguity. We will add this additional discussion in the revised manuscript.\n\n---\n\n**Q1: Why not use positional encoding?**\n\nWe appreciate the interesting suggestion for applying (e.g.) SHAP to the positional encodings (PE), however this approach is not necessarily trivial. For example, Shapley-based methods generally require defining a baseline value which is used for masking purposes; this is not trivial in the case of PE, since you need to ensure that the masked sequences are valid (e.g. the masked positional encodings cannot be constant “null” values, cannot be repeated in the sequence, must be “valid” PE values, etc.). Additionally, one of the advantages of OrdShap is that it is model-agnostic; it can be applied to any sequential model. A PE-based method would be more limited in the models it can be applied to.\n\n---\n\n**Q2: Attributions due to feature order vs timing**\n\nThank you for bringing up this point. We agree that in many applications we are interested in the time that a feature occurs rather than its position. In the main text of our manuscript, we assume that the number of possible permuted positions = the number of features (d) to simplify the notation. However, this can be easily generalized to a time index or any groupings of features, as we describe in App. B.1. Intuitively, the position index we use in the main text can be mapped to any other index, including time. Under this new mapping, for a given feature, OrdShap-PI would give the position importance w.r.t. units of time (rather than position index). OrdShap-VI would give the value importance w.r.t. the mean time of the sequence. We will add more detail to the App. B.1 to clarify this.\n\nRegarding your provided example (paraphrased here), “what matters is not only if a medication was given before some other token, but how much after/before” – we want to clarify that this is a slightly different (though related) problem than what OrdShap solves. The provided example would require *pairwise* positional attributions; i.e. a positional attribution for one feature with respect to another feature. This is similar in concept to feature interaction attributions in XAI literature (e.g. [3]). In contrast, OrdShap generates attributions with respect to a feature’s position/time within the sequence. However, we agree that pairwise positional attributions would be an interesting follow-up work.\n\n---\n\n**Q3: Figure 1 is not very easy to understand. It is unclear why the raised predicted probability is a desired outcome**\n\nWe appreciate the feedback. The purpose of Figure 1 is to help motivate the importance of capturing the effects of feature order in sequential data. Existing attribution methods generally work by quantifying the sensitivity of model output to small changes in the feature *values* for a given sample (see Section 2 for examples), but otherwise assume that the feature order is fixed. However, in Figure 1 we show that prediction models can also be sensitive to changes in feature *order*. In particular, we permute feature order of a given sample while leaving feature values unchanged, and plot the resulting model outputs in the left line plot (in this binary prediction task, model output is the predicted probability of LOS $\\geq 3$). We observe that the samples with permuted features often result in different model outputs and predictions (red vs blue lines). This model sensitivity towards feature order cannot be captured by existing methods that assume fixed ordering. We will add this clarification in the revised manuscript.\n\n---\n\n**Q4: TIME and PoSHAP?**\n\nAs mentioned in Section 2, TIME and PoSHAP are attribution methods for sequential models. We can provide a more detailed summary below, which we will add to the revised manuscript.\n\n\nPoSHAP applies KernelSHAP to each sample in a given dataset while also saving the position of each feature in each sample. Then the authors propose to visualize the results as a PxV matrix, where P is the number of possible positions and V is the number of unique features in the dataset, by averaging the KernelSHAP attributions for each position and feature. In contrast with OrdShap, PoSHAP is a global visualization that does not characterize model sensitivity to changing feature value or position for a given sample.\n\nTIME selects a contiguous subsequence of features (referred to as a “time window”) that is important with respect to model prediction. This differs from Ordshap in that it is 1) global, i.e. averaged over the entire dataset rather than w.r.t. individual samples, and is 2) feature selection rather than attribution, i.e. it doesn’t generate an attribution score for each feature. TIME also tests whether the model is sensitive to changes in feature order within each time window; this is again a binary indicator over the entire window and dataset. In contrast, OrdShap generates disentangled, Shapley-based scores for quantifying model’s sensitivity towards a feature’s value and position w.r.t. individual samples.\n\n\n---\n\n**Q5: Feature Independence for Shapley?**\n\nIn this context, we refer to the feature independence assumption in Shapley baselines, sometimes referred to as “off-manifold” [1] or “interventional” [2] Shapley (in contrast with “on-manifold” or “conditional” Shapley). This is the same assumption as used in KernelSHAP, which allows the baseline feature values to be sampled without conditioning on the selected features in each coalition. While this assumption is used in several Shapley methods, we acknowledge that the term “most” may be overstated; we will revise this section and include more detail to clarify this.\n\n---\n\n**Q6: Shapley Axioms for OrdShap-PI?**\n\nIndeed, while we show that OrdShap-VI satisfies the SB value axioms (Theorem 1) under certain conditions, OrdShap-PI does not inherit these guarantees since its value has a fundamentally different meaning than OrdShap-VI or SB values. Most likely we would need to create a different axiomization for position importance, which could be an interesting direction for future work. We will add this clarification in the “Limitations” section of the paper.\n\n---\n\n[1] Frye et al, “Shapley Explainability on the Data Manifold”  \n[2] Kumar et al, “Problems with Shapley-value-based explanations as feature importance measures”  \n[3] Dhamdhere et al, “The Shapley Taylor Interaction Index”"}, {"review_id": "HYg4bDgquI", "reviewer": "Reviewer_dM39", "summary": "OrdShap, a feature attribution method designed specifically for sequential black-box models, is introduced in this work. It's motivated by observation that existing works like SHAP conflate the importance of a feature's value with the importance of its position within a sequence. A SHAP extension is proposed linking feature-and-position attribution to Sanchez-Bergantinos values, and permutes features in subsets of values. Attributions of value-importance and position-importance are summary metrics from OrdShape. Experiments are done on synthetic data, EHR data, and IMDB to demonstrate the approach.", "questions": "* How many permutations were used for experiments in Tab. 2?\n* Is there an empirical analysis on the sensitive of the approach to choices of K and L for approximation?\n* Is this primarily memory or time bound setup -- what is the maximum sequence length d that's practical?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* The connection to Sanchez-Bergantinos values to XAI is novel and provides theoretical grounding for the approach. \n* The paper overall is well-written, motivated and provides good theoretical and empirical proofpoints.\n* Even though acknowledged, the computational cost is absolutely immense: d! * 2^d. LSq-approx is better, but still requires sampling K subsets and L permutations, making it significantly more expensive than KernelSHAP. This also as a result limits the applicability to long sequences (e.g., thousands of doc tokens).\n* OrdShap-PI is ultimately just a single linear coefficient. It's likely ultimately an oversimplication and connects weakly to the non-parametric, game-theoretic concept from the theoretical motivation.", "was_revised": true, "final_justification": "Feedback from authors seems sufficient. Holding my original rating.", "rebuttal": "Thank you for appreciating the novelty and theory of our work. We have provided responses to your clarifying questions below and will update the manuscript accordingly. Please feel free to ask any other follow-up questions. \n\n---\n\n**W1: Computational Cost**\n\nWe acknowledge the reviewer’s concern about computational cost. This cost is inherent to the problem: capturing position-dependent effects requires evaluating the model under different permutations, which is computationally expensive. However, this cost can be justified in applications where the positional effects are important to understand. We also highlight that this work is the first to investigate this notion of positional attributions; there may be methods to further improve efficiency, which we leave for future work.\n\n---\n\n**W2: OrdShap-PI is simple**\n\nWe acknowledge that OrdShap-PI's linear formulation is simple; this is by design for easy interpretation of the OrdShap-PI value for users. Most, if not all, users of attribution methods should be familiar with the meaning of linear coefficients. OrdShap-PI therefore provides an interpretable answer to the question: does this feature matter more when it appears later in the sequence, and by how much more?\n\n---\n\n**Q1: How many permutations were used in the experiments in Table 2?**\n\nIn our experiments we used $L=100$ permutations. During our testing, increasing the number of permutations past 100 showed minimal effect on AUC results.\n\n---\n\n**Q2: Empirical sensitivity towards choices of K and L?**\n\nThank you for your suggestion to include a sensitivity analysis on the choice of parameters K (number of subsets) and L (number of permutations). We agree that these results would help guide users in selecting appropriate parameters. We are conducting an empirical sensitivity analysis to show the tradeoff between the number of permutation and subset samples and the variance in OrdShap approximation, which we will include in the camera-ready revision.\n\n---\n\n**Q3: Memory or Time bound setup?**\n\nThe main constraint is typically time rather than memory. Both algorithms require evaluating the black-box model on the permuted and masked samples, which will be the primary bottleneck in practice.\n\nMemory is generally not a practical concern for the following reasons: 1) The black-box model can be queried in batches and only requires a single forward pass per query, 2) the Sampling algorithm can be calculated using batches of samples, and 3) the LS algorithm can be calculated in batches using stochastic gradient descent.\n\nRegarding maximum sequence length: there is no technical limitation due to sequence length, however the number of possible permutations in a sequence increases factorially with $d$, which increases variance when approximating OrdShap with a fixed number of samples. Therefore, for sequences with more than a few hundred features, we recommend first grouping features into interpretable subsets and explaining each group individually, and/or applying a feature selection step to remove irrelevant features. We discuss these strategies in Section 5.2, and will add additional detail in the revised manuscript."}, {"review_id": "8RyynAotyU", "reviewer": "Reviewer_Sk6y", "summary": "This paper introduces a Shapley-value-based feature attribution method designed to explain the importance of a feature's position within sequential data. To address the high computational cost of Shapley value estimation, the authors propose a least-squares approximation technique. Experimental results using inclusion/exclusion evaluation demonstrate that the proposed method, OrdShap-VI, produces more faithful explanations compared to existing approaches.", "questions": "Please see the weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n* The paper addresses a critical limitation in existing feature attribution methods for deep learning models. Most current approaches overlook the role of a feature’s sequential position and focus solely on perturbing feature values. By explicitly modeling positional importance, the proposed method fills a notable gap in the interpretability literature for sequential data.\n\n* The methodology is grounded in a solid theoretical framework, leveraging Shapley values and least-squares approximation to ensure both theoretical soundness and computational feasibility.\n\nWeaknesses:\n\nThe primary concern lies in the evaluation strategy of the proposed method.\n\n* Although the method is designed for sequential data, the experiments are limited to a narrow set of domains. While the EHR dataset is sequential in nature, it belongs to a specific subdomain (medical records). The evaluation would be more convincing if the method were tested on standard time-series benchmarks from broader application areas, such as forecasting or regression tasks in finance, weather domains.\n\n* The NLP evaluation is conducted on the IMDB sentiment classification dataset, which is a relatively simple binary classification task. It would be valuable to assess the scalability and robustness of the method in more complex settings—e.g., multi-class classification with thousands of labels, or tasks involving long-context inputs such as those seen in modern large language models (e.g., inputs exceeding 100k tokens). It remains unclear whether the method maintains its sensitivity to positional importance when the input length becomes very large or when the task complexity increases.", "was_revised": true, "final_justification": "The authors' responses have addressed by concerns in the Weaknesses section. I am inclined to Accept.", "rebuttal": "Thank you for taking the time to review our work. We are encouraged that you appreciate the novelty of our work as well as the theoretical framework. We have provided responses to your concerns and will update the manuscript accordingly. Please feel free to ask any other follow-up questions. \n\n---\n\n**W1: Experiments are limited to a narrow set of domains**\n\nWe thank the reviewer for their feedback. OrdShap is a  general method applicable to any sequential model. We primarily tested on real-world medical record datasets because they represent domains where position effects are particularly consequential and interpretable. In fact, it was through working in this real-world medical domain that inspired us to develop OrdShap.  Note that we also included results on natural language and a synthetic dataset. Based on your suggestion, we agree that demonstrating OrdShap's applicability to other time-series benchmarks would strengthen the evaluation. We will include additional results on a benchmark forecasting task (such as web traffic or climate) in the camera-ready revision. \n\n---\n\n**W2: Experiments on scalability and robustness of the method**\n\nThank you for the feedback regarding scalability to (A) multi-class classification with thousands of labels and (B) long-context inputs. Regarding (A), OrdShap follows other Shapley-based methods by calculating separate attributions with respect to each class in a one-vs-all approach. This is because the attributions are directional – their sign indicates importance with respect to one particular class. Therefore, similar to other Shapley methods, OrdShap generally scales linearly with respect to the number of classes to be explained. In particular, if the model outputs for the permuted samples can be stored in memory, then the samples can be reused for each class which further reduces computational cost. In the case of 1000’s of classes, we recommend focusing on the top-k predicted classes or classes of interest. We will edit the manuscript to add more detail based on this discussion.\n\nRegarding (B), the scalability for long-context inputs, we acknowledge that the computational complexity of OrdShap makes it challenging to explain sequences with 1000’s of tokens. For these scenarios, practitioners would likely have to combine tokens into “batches” (e.g. superpixels, text segments, subsequences, etc) and explain batches rather than individual tokens, and/or apply feature selection methods. This cost is inherent to the problem: capturing position-dependent effects requires evaluating the model under different permutations, which is computationally expensive. However, this cost can be justified in applications where the positional effects are important to understand. We also highlight that this work is the first to investigate this notion of positional attributions. We acknowledge this limitation in Section 7, but will add more detail based on this feedback."}]

Author rebuttal:

Meta-review: This paper proposes OrdShap, a Shapley-value-based attribution method for sequential models that disentangles the influence of feature values from their positions on model predictions. The proposed approach builds on Sanchez-Bergantiños values from game theory, and uses efficient sampling and least-squares approximations to address the computational cost of exact computation. Experiments on synthetic, healthcare, and NLP datasets validate the approach.

Reviewers praised the paper’s clarity, pedagogical examples, and the novelty of connecting positional attribution to Sanchez-Bergantiños values. Initial concerns regarding computational cost, evaluation scope, and statistical significance of reported results were satisfactorily addressed in the rebuttal. Overall, the contribution is well-motivated, technically sound, and has potential to make a significant impact in explainable AI for sequential models. The authors are encouraged to revise their final version according to the rebuttal, including clarifying figures, attributing time, and the feature independence assumption.
Decision: Accept (poster)