SeqSHAP: Subsequence Level Shapley Value Explanations for Sequential Predictions
Authors: Guanyu Jiang
Fuzhen Zhuang
Bowen Song
Jiani Li
Ying Sun
Yongchun Zhu
Tianyi Zhang
Weiqiang Wang
deqing wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: XAI
Explainability
SHAP
Sequential Predictions

Abstract: With the increasing demands of interpretability in real-world applications, various methods for explainable artificial intelligence (XAI) have been proposed. However, most of them overlook the interpretability in sequential scenarios, which have a wide range of applications, e.g., online transactions and sequential recommendations. In this paper, we propose a Shapley value based explainer named SeqSHAP to explain the model predictions in sequential scenarios. Compared to existing methods, SeqSHAP provides more intuitive explanations at a subsequence level, which explicitly models the effect of contextual information among the related elements in a sequence. We propose to calculate subsequence-level feature attributions instead of element-wise attributions to utilize the information embedded in sequence structure, and provide a distribution-based segmentation method to obtain reasonable subsequences. Extensive experiments on two online transaction datasets from a real-world e-commerce platform show that the proposed method could provide valid and reliable explanations for sequential predictions.

Under review as a conference paper at ICLR 2023
SEQSHAP: SUBSEQUENCE LEVEL SHAPLEY VALUE EXPLANATIONS FOR SEQUENTIAL PREDICTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
With the increasing demands of interpretability in real-world applications, various methods for explainable artificial intelligence (XAI) have been proposed. How- ever, most of them overlook the interpretability in sequential scenarios, which have wide range applications, e.g., online transactions and sequential recom- mendations. In this paper, we propose Shapley value based explainer named SeqSHAP to explain the model predictions in sequential scenarios. Compared to existing methods, SeqSHAP provides more intuitive explanations at subse- quence level, which explicitly models the effect of contextual information among the related elements in a sequence. We propose calculate subsequence-level feature attributions instead of element-wise attributions to utilize the information embedded in sequence structure, and provide distribution-based segmentation method to obtain reasonable subsequences. Extensive experiments on two on- line transaction datasets from a real-world e-commerce platform show that the proposed method could provide valid and reliable explanations for sequential pre- dictions.
INTRODUCTION
Sequential prediction tasks have wide range of applications in real-world, e.g., Online Trans- action (Wang et al., 2017: Zhang et al., 2018; Weber et al., 2018; Tam et al., 2019; Zhu er al., 2020: Chen Lai, 2021) and Sequential Recommendation (Quadrana et al., 2017: Tang & Wang, 2018; Sun et al., 2019; Shen et al., 2021: Cui et al.. 2022), since sequences contain continuous sig- nals which are important for model predictions. With the development of deep learning technique, requence-based models have achieved desirable performance in recent years (Hidasi et al., 2015; Quadrana et al., 2017; Wang et al., 2017: Tang & Wang. 2018: Zhang et al., 2018; Sun al., 2019; Zhu et al., 2020; Qiao & Wang, 2022). However, the complicated sequential data and increased model complexity make hard for humans to understand the prediction of models. Indeed, for se- curity and trust considerations, is essential to develop effective explainable artificial intelligence (XAI) methods for sequence-basee based models in scenarios like fraud detection and medical care, so that end-users could understand how model predictions are produced with these complicated sequential data and models.
In recent years, considerable efforts have been made on the model explanation algorithms (Ribeiro et al., 2016; Shrikumar et al., 2017; Lundberg & Lee, 2017: Selvaraju et al., 2017: Wachter et al., 2017; Alvarez-Melis & Jaakkola. 2018; Mothilal et al., 2020; Slack et al., 2021; Ghalebikesabi et al., 2021; Ali et al., 2022). Among these works, feature attribution methods (Ribeiro et al., 2016: Shrikumar al., 2016; 2017: Lundberg Lee, 2017) are popular family post-hoe XAI methods. They calculate an attribution score for each feature to capture those important features for model predictions. However, most existing methods mainly pay attention to explain tabular data or images. And when dealing with the data and models in sequential scenarios, the complex input sequences make the element-wise explanations produced by these methods less explainable. The high-dimensional features and abundant interactions bring difficulty to existing element-wiss XAI methods to provide explanations. Separately assigning attribution scores to individual feature cells in the sequence is not informative enough for users understand the predictions. In addition, the great amount of features in a sequence could bring an extensive execution cost for existing methods, since the time complexity of them are mostly related to the number of features be explained.

---

Under review as a conference paper at ICLR 2023
In this paper, we propose SeqSHAP, a Shapley value based method explain model predictions in sequential scenarios. SeqSHAP provides explanations at unique subsequence level, which is more intuitive in sequential scenarios for humans compared to the clement- wise explanations. Mcan- while, we propose a distribution-base subsequences which utilizes the distribution information of sequential features. With obtained sub- sequences, we group the feature elements under each subsequence as independent units. Then Shap- ley value estimations for feature units are calculated, capture the important features that strongly influence the model prediction.
segmentation method to split the sequence into reasonable
Extensive experiments on two large-scale online transaction datasets collected from real-worlds are carried out. We analyze the local explanations produced by SeqSHAP and prove that our method provides intuitive explanations with meaningful subsequences, compared to existing feature attribu- tion methods in sequential scenarios. Our contribution could be summarized as follows:
We propose an effective XAI method to explain sequential predictions at a subsequence level, which is a unique and intuitive view in sequential scenarios.
We propose a distribution-basse segmentation method characterizing the distribution in- formation of sequential features to capture the context information and obtain reasonable subsequences.
real-world transaction datasets provided to evaluate the validity of our segmentation method and subsequence-level explanations produced by Se- qSHAP.
Extensive experiments
BACKGROUND
In this section, we firstly introduce the task of explaining model predictions with sequential in- puts. Then we introduce the background of SHAP (Lundberg Lee, 2017), a popular interpretable framework based on Shapley values in game theory.
2.1 EXPLAINING PREDICTIONS WITH SEQUENTIAL DATA
Machine learning (ML) models for sequential prediction tasks have been widely applied in real- sequentne
world applications (Hidasi al., 2015; Tang & Wang, 2018; Sun al., 2019; Zhu et al., 2020), since the historical behaviour records in a sequence contain valuable information for the prediction task. However, while different models with desirable performance are proposed, predictions getting particularly difficult to explain due to the increasing model complexity, which blocks the application of new techniques in some scenarios requiring high degree of interpretability. As result, the demand of XAI methods in sequential domains is growing rapidly, as existing methods mostly focus tabular data and are not suitable for data with sequence structure.
Task Description In this paper, our task is building an interpreter to explain model predictions in sequential scenarios. Specifically, given a classifier f and sequence X which could be formed as:
{el, e2, ...y er where e {=f 2 16,....,hhh
where is the length of sequence and M the number of features, e, E RM represents the 1-th element of sequence which has M feature fields to describe it. The interpreter g is expected to generate an explanation for the model prediction á = f(x) E [0, 1]. For the family of additive feature attribution methods,
element-wise explanation (1,5 < < T,1 < j < M)) to the corresponding feature cell z² in the sequence X, which represents the influence of features the model prediction.
Txx assigns importance score
2.2 SHAPLEY VALUE BASED EXPLANATIONS
SHapley Additive exPlanation, termed as SHAP (Lundberg & Lee, 2017), is a popular framework to explain model predictions based the Shapley value in game theory. Through summarizing previous methods (Lipovetsky & Conklin, 2001; Štrumbelj & Kononenko, 2014; Bach et al., 2015; Datta et al., 2016; Ribeiro al., 2016; Shrikumar et al., 2017), SHAP builds an additive explanation model g
2

---

Under review as a conference paper at ICLR 2023
M dizi,
g(z) - 0D +
where M the number of features, 2 E RM is simplified features in a binary feature space, and an attribution score is assigned to each participating feature by solving Shapley values in designed cooperative game. As long as there has been a lot research and methods on SHAP's properties and applications in recent years (Sundararajan & Najmi, 2020; Frye al., 2020; Zhang et al., 2020; Slack et al., 2020; Kumar et al., 2021; Jethani et al., 2021; Covert & Lee, 2021; ; Bento et al., 2021;
Watson, 2022), here mainly introduce KemelSHAP (Lundberg & Lee, 2017) which is most relevant to our work.
KernelSHAP KernelSHAP model-agnostic explainer for local predictions which adopts the same objective function as the classic feature attribution method LIME (Ribeiro et al., 2016) shown Eq. (2), while adjusting the choice of several settings to satisfy three desirable properties. Given classifier ʃ and input sample x, the objective function could be solved using weighted linear regression with the loss function in Eq. (3), where h2 is a mapping function that maps simplified features to the original input feature space, and Te is weighting kernl.. The solution ø {db i E 2 M)} provides an estimation of Shapley values for the input features.
ş = arg min L(S,, T2) 2(g)
TU T E((h((z) - -
SEZ
KernelSHAP treats each feature independently and calculate the attribution score using the weighted sum of feature's 's marginal contributions. It works well for tabular data since there is less context information, and the calculation marginal contribution could partly model the interactions among features. However, for the case of sequential data, various contextual information is embedded in the sequence. And when explaining sequences, KerneISHAP has to adopt a Monte Carlo sampling strategy which sacrifices the precision of Shapley values to reduce the computational complexity, since the number of features in a sequence is too large to build a power set and calculate marginal contributions for all features. Accordingly, abundant contextual information could be ignored, and the element-wist wise explanation by KernelSHAP is less reliable.
3 SEQSHAP
In this section we introduce our method SeqSHAP which could provide more intuitive explanations for sequential predictions. Firstly, we discuss the motivation and advantage of explaining sequential predictions at the subsequence level. Then we propose segmentation method to obtain reason- able subsequences from the input sequence. Finally, the process of generating subsequence level explanations with SeqSHAP is given.
3.1 EXPLAINING SEQUENTIAL PREDICTIONS AT SUBSEQUENCE LEVEL
A sequence is a stack of events that happened in a range of time, interactions of features among neighboring events often contains some hidden patterns (e.g., continuously changing, alternating, and recurring fields), which can be captured by MI. models and has a significant impact the model predictions. We note that applying attribution methods to set of individual cells of the sequence (i.e., the cell-level explanation), which returns an importance matrix with the same shape as the
91.1 81,M
92.1
92.32
input sequence: Geell
can not explicitly model the effects of this interaction.
9T.1
ST.
And in practical applications, it is also difficult for end-users understand the prediction with such an importance matrix, since humans tend to make predictions based on finding abnormal patterns rather than single feature cells in sequences.
We are inspired by the concept of session in recommendation system, which means several opera- tions by a user over short period of time. Now that session could represent the user characteristics
3

---

Under review as a conference paper at ICLR 2023
during this time period, we attempt to group related neighbouring events in the sequence as sessions, and adopt XAI methods on them to explain the sequential data. By splitting the sequence into several subsequences and calculating importance scores for each feature under each subsequence (i.e., the
91,1 91.M
92.1
92.M
subsequence-level explanation), explanations are provided as Guhhee
where
SK.1
9K.M
K T is the number of subsequences split from the imput sequence, Gh.s the importance score of the i-th feature field under k-th subsequence. In this way, feature interactions between adja- cent events are taken as a unit be explained using attribution methods. And the explanation with smaller shape provides a clearer guide for end-users to focus on the important areas in the sequence.
3.2 DISTRIBUTION-BASEDD SEGMENTATION METHOD
As mentioned above, hope those patterns of features can be explained as grouped units, through splitting the sequence into several subsequences. So the problem becomes how to split sequence properly while ensuring the events that make up pattern can be grouped into the same subsequence. Simply split the sequence randomly or split with a fixed window could easily separate the related events and break the patterns, and the subsequences obtained are meaningless.
Here we suppose those hidden patterns and contextual information can viewed as a specific dis- tribution of the features, and propose a distribution-based segmentation method get reasonable subsequences from the sequence. We attempt to maximize the distribution discrepancy among ad- jacent subsequences, in order make adjacent subsequences include different context information. Firstly, the events happened within a specific time range are grouped as units s, (1 S i S k) and these units make up the initial set Sanst waiting to be segmented:
Sinit {81, 82, .8k)
= [ei..... Cng J: Vi k), (s(en=-1+1) S ō1 S [sg 82.
{Eny itl..... ,t,,,.,,,,
where k is the number of grouped units, w is the size of time window, ts(e) is the scaled timestamp feature of event e, 51 and are defined to limit the size of subsequence. Then, insert split points into Smnt gradually, the point that maximizes a metric function will be chosen as the split point of the current round. The segmentation process is shown in algorithm 1:
Algorithm Distribution- based segmentation
Input: Initial set Smitt subsequence amount K, Metric function
1: S + {S } = (1 $2 ..... sx)). Split points P +-
2: while |S| < K do
3: dmaax 0, p < 0, Sp < Φ
4: for - to [Smiil do
5: if i P then
6:
P' + Sorl(P +i)
Add point to P temporarily
7:
S' < ((s,, (pp......pu))))(ppuup........... ( spppj,.......................... ... 1 883]
8:
d D(S')
Calculate the metric
9:
d > dmaz then
p p i, dmax t Sp S'
11:
end if
end if
13:
end for
P Sp
> Update split points and segmented subsequences
15:
end while
Output: Segmented subsequences
The metric function D we design is shown Eq. (5), where the input Sp is the set of subsequences obtained after the latest insert index p. faiet is distance function measuring the distribution dis- crepancy between two subsequences in S, (e.g.. MMD (Gretton et al., 2012), KL-divergence), |s[] and [s") are used to limit the size of subsequences. And m is the size of the measuring window
4

---

Under review as a conference paper at ICLR 2023
which determines how many neighbouring subsequences should be included calculate the dis- tance with the current subsequence. Our purpose is to distinguish the subsequences under different
distribution, to capture the related events into unit. The process of segmentation stops when the number of subsequences segmented reaches given parameter K << |Simit
1 min(i+m.15.ll)
fdist (5,, 88)
D(S,) = Σ ==1
(5)
jmmin(i-m. 1)
3.3 PROVIDING EXPLANATIONS WITH SEQSHAP
With sequence divided into K subsequences, feature matrix of sequence X RTxM could be formed as X" E RKXM Each row in X' corresponds to a subsequence contains several events and N [1,, ,,,,...... [x] is the number of events in subsequences. The background values B 22, JM] are sampled with average feature values in the dataset, to fill the absent features as uninformative feature values for the computation of Shapley values.
When explaining a sequence X' directly using KemeeISHAP, the explanation model g is shown in Eq. (6). It takes K M feature units in X' to build the coalition game, and the large feature space could bring an obvious loss the precision using Monte Carlo sampling strategy to approximate the Shapley values, as mentioned in Subsection 2.2. To reduce the computational cost and the loss of precision, our method SeqSHAP calculates the subsequence-level Shapley values with two stages. For the first stage, we build feature-level explanation model g like TimeSHAP (Bento al., 2021) as shown in Eq. (7), where each feature field of the sequence is taken as unit and the feature-level Shapley values of E RM are calculated with KernelSHA..
K
M
(z) R (x = ob
g
+
(6)
M
[(xx((z)) Re (z of Σ 3/
The simplified features 2f E {0, 13 M in Eq. (7) could treated as coalition of z in Eq. (6), i.e, z/ = O is equivalent to Vi E [1, K], 2 = O, since each column of X' is taken as a unit in o6 since + equivalent to zf while all the features are absent. Therefore, the feature-level explanation of is actually an estimation of the sum of features' Shapley values in KernelSHAP:
g!. And there of = (+xx ~ ((f))
K
Σ S Isiss,,,
(8)
The second stage of SeqSHAP is shown in Algorithm 2, with feature-level explanations of, traverse M feature fields of X' to provide subsequence-level explanations. For the case of j-th field, the explanation model و. built with the candidate feature set S = (=1,, "e..,,=2...-- -2..:--.. ".., ) where 2f, represents the i-th subsequence of j-th feature field. The simplified feature .00 E Rk corresponds to the presence of subsequences S₁ and the mapping function hy is defined to map _seg to the original feature space: Zj
T/,1
he (z
I
12,1
h (22.5
261
x2,,
La [T7]]
seg is
hy (2,( 5000)
hhw (2
Fij
if = 0"
seg
by (2
x'x dKK _
K.
Thus when it is j-th feature's 's turn, other features in the mapping result are original input features, and the simplified features 25 determines the subsequences under j-th feature whether to be re- tained with original input values replaced by uninformative feature values. Notably, each element
5

---

Under review as a conference paper at ICLR 2023
Jh in the matrix represents subsequence of j-th feature with the shape of n4, and the replace- ment of element is achieved by filling the same number uninformative values as n,. Afterwards, KemelSHAP is applied to solve the explanation model:
99 ( (2)
00. + Σ Pag 4,5 90. 0f + Σ o/. $F1
(10)
the definition of $ is based on Eq. (6) and Eq. (7), since other feature fields are taken as the background and will not change during sampling coalitions, the effect of them should be added into the bias part of the explanation model. Hence the calculated explanations satisfy the property of Eq. (11), which maintains the consistency of explanation among features. Finally, sub-sequence level explanation & is obtained by repeating the process M times, and each element 65 E RK the Shapley value for K subsequences under j-th feature.
Σ
seg Pi
o/.
(11)
Algorithm Subsequence-level explanation
Input: sequence X' E RK* x classifier f. feature-level attributions of
1: for f-1 1 to M do
2: S + j Fii.
Candidate feature set
3:
2)
[ 2 22.........
11
(0,
Simplified features
4:
h (2²) --quatio(9)
b mapping function Explanation model
5:
g(x0) + Equation(10
6:
2 t KernelSH 23,21,
7: end for
Output: Φ 1 0 6²)
Through calculating Shapley values with two stages, SeqSHAP reduce the size of feature space significantly compared to applying KemneISHAP directly to the sequence X'. Indeed, SeqSHAP acquires fewer perturbed samples calculate the Shapley values and could provide subsequence- level explanations for sequence data more precisely with lower computation cost.
EXPERIMENTS
To evaluate method, experiments two online transaction datasets from e-commerce plat- form in real world are carried out. The task of XAI methods to provide local explanations for sequence-based fraud detection models, which used to help end-users understand the model predictions, as described in Subsection 2.
4.1 EXPERIMENTAL SETUP
Our datasets are collected from large-scale e-commerce platform, consisting of approximately 1.IM (Dataset A) and 1.6M (Dataset B) samples separately. Each sample X in these datasets is tabular and corresponds ordered by the time, as formed in 2.1. An operation record, which is called an event here. includes M features to describe the details of the event. The details of the datasets are shown in Table 1.
sequence of user's historical operation records
the platform
For considerations of privacy, the feature names are encoded into several types according to the content they describe (e.g., location, time, account, description of transactions). For dataset A, given classifier fA predicts whether the last event of the sequence is fraudulent transaction, based on the historical events e[l T 1]. And for dataset the classifier fB predicts whether the user is a fraudster based on the operation records e[1 7'' happened recently. In our experiments, choose RNN- based models as the classifier for the prediction tasks, which is built with an embedding layer, two LSTM layers and several on two datasets separately. Additionally. the detailed settings parameters our segmentation method could be found in Appendix A.
layers. Both models FA and fø fit optimally

---

Under review as a conference paper at ICLR 2023
Table I: Dataset details
Dataset A
Dataset B
Max Events 7
150
150
Feature Fields M
33
30
Categorical Features
18
20
Numerical Features
15
10
Samples
1.1M
1.6M
Classifier
fA
fB
4.2 COMPARATIVE EXPERIMENTS
Metric of Feature Attributions. In our quantitative experiments, we take feature removal exper- iments to evaluate the local explanations produced by different feature attribution methods. Since the ground truth of an explanation is hard to get, remove the top a% elements in the ordered explanations and observe the difference model predictions, A large change of the predicted score means that the explanation does capture those important features for model predictions. We define the metric Dr, as Eq. (12), where x is the origin input and x' is the perturbed input obtained by replacing the top a% elements in the explanation with uninformative feature values.
N
f(x') f(x) f(x)
Dr
(12)
Experiments on Segmentation Methods. We compare the performance of different segmentation methods show our distribution-based method could capture relative events and generate reason- able subsequences. The subsequence- attributions are calculated as local explanations with the subsequences obtained from different segmentation methods. And the metric Dr are used to com- pare the perfomance explanations. The result shown in Table 2, where segmentation method Uniform means split the input sequence with fixed window of size [T/K], and Random means split the sequence into K subsequences randomly. Ours(Kl) and Ours(MMD) are our distribution- based method using KL-divergence and MMD distance the distance function fdists separately. We choose the drop rate a from {1%. 1.5%, 2%, 2.5% and calculate the mean of changes of model pre- dictions after removing top elements in the explanations. The result shows that the subsequences split using our distribution-based segmentation method are more reasonable for the subsequence- level explanations, while the contextual information helpful for the model predictions are better grouped in subsequences and removing the top elements could bring significant changes to model predictions.
Table Dra of different segmentation methods
Dataset
Dataset
Method
Dr1%
Dri.5%
Dr2%
Dra.3%
Dr
Dr1,5% 15%
Dr
Dra.5%
Random
0.1304
0.2128
0.2612
0.2921
0.2943
0.3677
0.4015
0.5823
Uniform
0.2215
0.2707
0.3384
0.5168
0.3471
0.4162
0.5699
0.6881
Our)(KL)
0.2870
0.3375
0,4135
0.5938
0.3667
0.4134
0.5589
0.6918
Ours(MMD)
0.3006
0.3828
0.4941
0.6487
0.4382
0.4826
0.6204
0.7284
Experiments on Local Explanations. To prove that the subsequence-level explanations from Se- qSHAP outperforms the element-wise explanations from existing feature attribution methods, we choose two popular feature attribution methods, KernelSHAP (Lundberg & Lee, 2017) and LIME (Ribeiro et al., 2016) as our baselines. We compare the explanations produced by SeqSHAP and the baselines with the metric Dr. mentioned above. Since both baseline methods provide element- wise explanations, we drop the top [T M a%] elements the explanations from baselines, and [ * M a%] elements in SeqSHAP when calculating Dr for faimesss. The amount of per- turbed samples for KernelSHAP and LIME are set to 64K to make a trade-off between the efficiency and precision. As shown in Table 3, the performance of KernelSHAP and LIME are similar since

---

Under review as a conference paper at ICLR 2023
they apply the same objective function (Eq. (2)) with different kernel weights for perturbed samples (Eq. (3)). Our method outperforms the baselines in most cases on two datasets, which means the important feature subsequences accurately captured and assigned with higher attribution scores by SeqSHAP, and the removal of them makes a large difference to the model predictions.
Table 3: Dra of different attribution methods
Dataset
Dataset
Method
Dr1%
Dr1.5%
Dr2%
Dr2.5%
Dr1%
Dr1.5%
Dr2%
Dr2.5%
LIME
0.2728
0.3249
0.4551
0.6028
0.4248
0,4865
0.5471
0.6022
KernelSHAP
0.3015
0.3595
0,4498
0.6261
0.4161
0.4709
0.5954
0.6422
Ours(MMD)
0.3006
0.3828
0.4941
0.6487
0,4382
0,4826
0.6204
0.7284
4.3 CASE STUDY
In this section, we analyze several local explanation cases with sequence samples to show the ef- fect of subsequence-
explanations from SeqSHAP. Firstly, we choose a positive sample from Dataset A whose latest event er in sequence is fraudulent, which predicted correctly with large confidence by the RNN-based classifier fA- We generate local explanations for the sample applying KernelSHAP and SeqSHAP separately.
shown Figure I, we visualize the feature attributions geeenel E R and dz09 E RKxM, where T is the length of sequence and K is the number of subsequences obtained with our segmentation method. For simplicity we drop the feature fields where all cells under the feature have a smaller absolute attribution score than threshold 8., and mask the cells whose absolute attribution score Φi,j is smaller than another threshold 6, (the grey cells in Figure 1). The darker the cell in heatmaps, the larger attribution score it gets which means more important for the model prediction. It is obviously that the element wise explanation @ in Figure (a) is not intuitive for
understand with many events and feature fields, even if have dropped those less important elements. And for the case of subsequence-leved explanation oreg Figure 1(b), each cell represents the importance of subsequence including several related events under feature field. It is more intuitive with less subsequences (K < T') for users to understand which parts of the sequence promotes the classifier to make such a prediction.
tace budure
2012
buds feature
0.00
aconaro batuoc
Fads fratus
2010
acomaic
0.08
acanar Nudiin
scusaro fuatin
0.008
locebor Nomrre
0.04
ocation budire
2,008
copooda budure
copodle boduos
2.004
0,002
amcunt fruture
ancurt frcture
0.00
02
amourt boturs
tato futurs
fealure
-0.02
She bolur
-8002
tie hature
fretue
ane Vuluo
(a) Element-wise explanation
(b) Subsequence-level explanation
Figure 1: Local explanations with KernelSHAP and SeqSHAP
Our method SeqSHAP provides explanations at subsequence-level and the feature attribution results goey E RKXM assign an importance score to each subsequence for each feature field. Through analysing the distribution of attributions along different axes, higher level explanations could be obtained. Figure 2 visualize the higher level explanations in two different views, capturing the im- portance subsequences and features separately. Figure 2(a) shows the importance of subsequences where the Shapley values of features under the i-th subsequence are plotted along the y-axis, which could help locate the abnormal subsequenes. Figure 2(b) provides a feature-level importance expla- nation, which could help identify the influential features in sequence.
As mentioned in Subsection 3.1, SeqSHAP explains a sequence at subsequence-level to capture those contextual information and hidden patterns included in the related neighhouring events of the
8

---

Under review as a conference paper at ICLR 2023
trada festue
5.09
trade fature
sosrarto frature
fealure?
0.04
location fucture
localion feature
arount faaur
armunt fealure
amount festure
trade feature
iin fuature
frature
10
-000
-0.1
0.00
0.02
0.04
0.08
0.8
(a) Subsequence importance
(b) Feature importance
Figure 2: Two higher level explanations
sequence. We analyze the elements in explanation 6.t4 with large attribution scores and summarize the corresponding input features of 6e00 As shown in Figure 3, the top elements with largest Shapley values are labeled with the summary of corresponding features in the sequence sample. Multiple abnormal patterns discovered in the 10-th subsequence, including several transactions with large amount, variable locations and uncommon operations. Other subsequences with patterns like transactions happened midnight and repetitive failed transactions also assigned with high scores. Indeed, our segmentation method could provide reasonable subsequences including explicit feature patterns, and SeqSHAP is applied find the important subsequences for model predictions.
S s10:amount feature 1
Continuous large amount transactions
0.08
S 11::scenario feature 3
Dangerous transaction scenes
$1 O:location feature 2
Changing transaction locations
0.06
t10dr feature 3
Repeating login-transaction operations
$21:time feature 2
Transactions at 1am
-0.04
$12:trade feature 5
Continuous transfer to account
S 11:ttrade feature 10
Failed transactions
0.02
Figure 3: Semantic patterns included in the important subsequences with higher Shapley values.
CONCLUSION
For security and trust considerations real world, the increasing need for explainable AI promotes the study on post-hoc feature attribution methods. However existing XAI methods mostly overlook the interpretability in sequential scenarios which has a wide range of applications. The widely used element-wise explanations which assign importance scores to all feature cells in sequence is not intuitive for end-users to understand and could cause huge reduction of the precision of explanations.
In this work, we propose SeqSHAP to explain sequential predictions at the subsequence-level, unique view for feature attribution methods. We provide a distribution-basee segmentation method to obtain reasonable subsequences to capture the hidden patterns and contextual information among neighbouring events. Through evaluating the explanations on online transaction datatsets collected from real-world, SeqSHAP is proven to be able to generate reliable Shapley value explanations for sequential data. With the user studies looking into the explanations and input features. the subsequence-level explanations confirmed to be aligned with human concepts and could help users find out the abnormal patterns in the sequence that significantly influence the model predic- tions.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. Xai for transformers: better explanations through conservative propagation. arXiv preprint arXiv:2202.07304, 2022.
David Alvarez-Melis and Tommi S Jaakkola. On the robustness of interpretability methods. arXiv preprint arXiv:1806.0804 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Robus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non- non-linear classifier decisions by layer- layer-wise relevance propagation. PloS one, 10(7)=e01301 40140, 2015.
João Bento, Pedro Saleiro, André F Cruz, Mário AT Figueiredo, and Pedro Bizarro. Timeshap: Explaining recurrent models through sequence perturbations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2565-255277 2021.
Joy long-Zong Chen and Kong-Long Lai. Deep convolution neural network model for credit-card fraud detection and alert. Journal of Artificial Intelligence, 3(02):101-112, 2021.
Ian Covert and Su-In Lee. Improving kernelshap: Practical shapley value estimation using linear regression. In International Conference on Artificial Intelligence and Statistics, pp. 3457-3465.
PMLR, 2021.
Chuan Cui, Qi Shen, Shixuan Zhu, Yitong Pang, Yiming Zhang. Hanning Gao, and Zhihua Wei. In- tention adaptive graph neural network for category- -awaee ression-basee recommendation. In- ternational Conference on Database Systems for Advanced Applications, pp. 150-165. Springer, 2022.
Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influ- ence: Theory and experiments with learning systems. In 2016 IEEE Symposium Security and Privacy (SP), pp. 598-617. IEEE, 2016.
Christopher Frye, Damien de Mijolla, Tom Begley, Laurence Cowton, Megan Stanley, and Ilya Feige. Shapley explainability on the data manifold. arXiv preprint arXiv:2006.011 01272, 2020.
Sahra Ghalebikesabi, Lucile Ter-Minassian, Karla DiazOrdaz, and Chris C Holmes. On locality of local explanation models. Advances in Neural Information Processing Systems, 34:1 33:15599-18807, 2021.
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances in Neural Information Processing Systems, 25, 2012.
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. Session-based rec- ommendations with recurrent neural networks. arXiv preprint arXiv:1511.0665 2015.
Neil Jethani, Mukund Sudarshan, Ian Connick Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. In International Conference Learning Representations, 2021.
Indra Kumar, Carlos Scheidegger, Suresh Venkatasubramanian, and Sorelle Friedler. Shapley resid- uals: Quantifying the limits of the shapley value for explanations. Advances in Neural Information Processing Systems, 34, 2021.
Stan Lipovetsky and Michael Conklin. Analysis of regression in game theory approach. Applied Stochastic Models Business and Industry, 17(4):319333)) 2001.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 2017.
Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 607-617, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jingjing Qiao and Li Wang. Modeling user berrvioo and original interest via adaptive multi- attention network for session-basee recommendation. Knowledge-Ban Basee Systems, 244:108567, 2022.
Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. Personalizing session-based recommendations with hierarchical recurrent neural networks. In proceedings of the 11th ACM Conference on Recommender Systems, pp. 130-137, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Qi Shen, Shixuan Zhu, Yitong Pang. Yiming Zhang, and Zhihua Wei. Temporal aware interest graph neural network for session-based recommendation. arXiv preprint arXiv:2112. 2021.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713, 2016.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning, pp. 3145-3153. PMLR, 2017.
Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. Fooling lime and shap: Adversarial attacks post hoc explanation methods. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp. 180-186, 2020.
Dylan Slack, Anna Hilgard, Sameer Singh, and Himabindu Lakkaraju. Reliable post hoc expla- nations: Modeling uncertainty explainability. Advances in Neural Information Processing Systems, 34:9391-9404, 2021.
Erik Štrumbelj and Igor Kononenko. Explaining prediction models and individual predictions with feature contributions. Knowledge and Information Systems, 41(3):647-665, 2014.
Fei Sun, Jun Liu, Jian WU, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequen- tial recommendation with bidirectional encoder representations from transformer. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, pp. 1441-1450, 2019.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Pro- ceedings of the 37th International Conference on Machine Learning, pp. 9269-9278. PMLR, 2020.
Da Sun Handason Tam, Wing Cheong Lau, Bin Hu, Qiu Fang Ying, Dah Ming Chiu, and Hong Liu. Identifying illicit accounts in large scale e-payment networks-a graph representation learning approach. arXiv preprint arXiv: 1906.9900000 05546, 2019.
Jiaxi Tang and Ke Wang. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of the 11 ACM International Conference on Web Search and Data Mining, pp. 565-573, 2018.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Harv. JL. & Tech,, 31:841, 2017.
Shuhao Wang, Cancheng Liu, Xiang Gao, Hongtao Qu, and Wei Xu. Session- based fraud detection in online e-commerce transactions using recurrent neural networks. Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 241-252 Springer, 2017.

---

Under review as a conference paper at ICLR 2023
David Watson. Rational shapley values. In 2022 ACM Conference Fairness, Accountability, and Transparency, pp. 1083-1094, 2022.
Mark Weber, Jie Chen, Toyotaro Suzumura, Aldo Pareja, Tengfei Ma, Hiroki Kanezashi, Tim Kaler, Charles Leiserson, and Tao B Schardl. Scalable graph learning for anti-money laundering: A first look. arXiv preprint arXiv: 1 182000077, 2018.
Hao Zhang, Yichen Xie, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions dnns. arXiv preprint arXiv:2010.050455 2020.
Ruinan Zhang, Fanglan Zheng, and Wei Min. Sequential behavioral data processing using deep learning and the markov transition field in online fraud detection. arXiv preprint 180.05/80 2018.
Yongchun Zhu, Dongbo Xi. Bowen Song. Fuzhen Zhuang, Shuai Chen, Xi Gu, and Qing He. Mod- cling users' behavior sequences with hicrarchical explainable network for cross-domain fraud detection. In Proceedings of The Web Conference 2020, pp. 928-938, 2020.
12

---

Under review as a conference paper at ICLR 2023
A PARAMETERS SETTING OF SEGMENTATION METHOD
For the first step we divide input sequence into units Sinit (Eq. (6)), the size time window is defined
100 0.01), where (s(en) is the scaled time interval between the earli- est event and latest event. 81 3 and 82 imum mean discrepancy(MMMD Gretton et al. (2012) is chosen the distance function faist to split the sequence(Eq. (5)), the embedding representation of events obtained from the given model used to calculate the distribution distance. The number of sessions K is defined as min(maz((1)) | 30), for the precision and efficiency of computing Shapley values.
maxi (s(ee)
are set to limit the size of initial units. Max-
13


Reviews:
[{"review_id": "rLXnDxVvMHT", "reviewer": "Reviewer_nQqx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes SeqSHAP to study the interpretability of sequential data based on Shapley value. Instead of making element-wise explanations, SeqSHAP segments a sequence into sub-sequences, then uses Shapley value to characterize important features for a model decision. User studies were conducted to show the explainability ability of this method. ", "summary_of_the_review": "This work prompts explanations for sequential data, which can be complmentary to existing XAI methods. A sub-sequence level explanation method is proposed and validated in this work. However, the technical novelty aspect is somewhat limited to me, and require to be addressed during the author feedback period. ", "strength_and_weaknesses": "Strength: \nIn this work, the idea that splits a sequence into performing sub-sequences and subsequence-level explanation are reasonable. The proposed distribution-based segmentation method is also interesting and easy to understand. The user studies show the effectiveness of SeqSHAP.\n\nWeakness: \nThis method is built upon Shapley value with certain modifications specific to sequential data, which compromises the technical novelty of this work. And the selection of parameters is not clear to me, which should influence the final performance. For example, how to choose an optimal K suitable for sequences with different lengths. Besides, the comparison methods in Table 2 are vanilla so that they not quite convincing. Maybe more stronger methods need to be considered. In addition, the baseline methods, i.e., LIME and KernelSHAP, used in Table 3 are proposed in 2016 and 2017 respectively. More recent methods should be considered as baselines, for example, “On locality of local explanation models, NeurIPS 2021”. Please address the weakness parts in the rebuttal.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pRMZCZOLI1L", "reviewer": "Reviewer_Sgob", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present SeqSHAP: a method to provide SHAP explanations for sequences exploiting subsequences. The methodology starts by segmenting the sequence in input. The authors proposed a method for segmenting the sequences based on the distributions of the sequence elements. Once they obtain the subsequences, they group the feature elements under each subsequence as independent. On this result, they apply KernelSHAP.\nThey provide the pseudo-code of the segmentation procedure, which is based on different steps, considering both the time and a metric function based on MMD or KL-divergence, and they validate their methodology by conducting experiments on 2 datasets from e-commerce platforms. They compare the goodness of the explanations obtained by applying insertion/deletion methodologies. \n", "summary_of_the_review": "The paper provides an interesting idea, straightforward. There are a few typos, but overall, the paper is well written and the authors provide the pseudo code for each part of the methodology. The experiments are conducted\ton 2 multivariate datasets from e-commerce, and show good performance w.r.t. the simple application of the KernelSHAP method, even if they reach results which are not so good overall (the best performance is achieved with the segmentation method at 2.5%, but the values are around 0.64 and if I recall correctly this metric highlight a good feature importance with values closer to 1). Also the idea of showing the semantic patterns is also interesting and may be a starting point to improve this methodology. \nHowever, I also have a concern about the validity of the method proposed: in the methodology the authors clearly stated that they took inspiration from recommender systems in the implementation of the segmentation procedure, which is based on a split on the time and then they apply a metric function (MMD or KL divergence). Then the authors test the method on e-commerce datasets, which are close to recommender systems. Given this setting, it seems to me that this approach may be feasible for this kind of data, but may fail for other kinds of sequences, such as energy consumption or trajectories. Have the authors tried other kinds of data? \n", "strength_and_weaknesses": "+ The idea presented is interesting: the sequences are often long and may also have several features to take into account, hence an explanation on them may be difficult to understand. \n+ The authors provide the pseudo-code for the subsequence extraction, as well as for the application of the KernelSHAP, in a clear manner.\n+ The idea of considering subsequences may overcome these problems, providing an explanation that is easier to understand for the end user. \n- However, the authors stress the fact that \"for the end user the importance matrix is too difficult to understand, since humans tend to make predictions based on finding abnormal patterns rather than single features in sequence’\" but the citation is missing: did the authors make some experiments to justify this belief? Did the authors read it in some other article?\n- Regarding the methodology part, there are a few typos and at some point the authors use the letter K to indicate different things.\n+ The experiments are well-written and correlated by several plots, which allows for an easier understanding of the metrics and properties tested. \n- However, the datasets used are only 2 and from the same kind of data, e.g. e-commerce. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ke8c4A4ZGv", "reviewer": "Reviewer_EXsb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the interpretability of deep learning in sequential scenarios. To this end, the authors propose to explain sequential predictions at the subsequence-level, and a distribution-based segmentation method.", "summary_of_the_review": "Some problems are not clarified well. The related work is not discussed well. See above Strength And Weaknesses for details.", "strength_and_weaknesses": "Strong points:\n1. The motivation of subsequence-level explanation is clear.\n\nWeak points:\n1. The proposed distribution-based segmentation is given in Algorithm 1 without any explanation, which is hard to understand. Some notions and technical details are unclear. See details. \n2. Sequential prediction has abundant applications, such as online transaction, sequential recommendation. However, the authors only validate their method in a binary classification fraud detection task with simple LSTM networks. It does not apply the proposed method to explain other reprehensive tasks and models in sequential prediction, such as BERT4Rec. \n3. Some problems are not clarified well. For example, the authors point out the high-dimensional features and abundant interactions in sequences. But, they do not give specific data or evidence to explain the difficulty. In addition, the number of features in experimental datasets also cannot reflect the challenge.\n4. The related work is not discussed well. Many methods are proposed to explain DNN in sequence data, such as [A1] and [A2]. What is the difference between the proposed method and these methods.\n5. The authors point out the weakness of element-wise explanations. However, many works also consider the effect of multiple input variables [A1, A3]. What is the difference between multiple variables explanation and subsequence-level explanation?\n\n[A1] Zhang, Hao, et al. \"Interpreting multivariate shapley interactions in dnns.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.\n[A2] Bento, João, et al. \"TimeSHAP: Explaining recurrent models through sequence perturbations.\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.\n[A3] Murdoch, W. J.; Liu, P. J.; and Yu, B. 2018. Beyond word importance: Contextual decomposition to extract interactions from LSTMs. arXiv:1801.05453 .\n\n\nDetails：\n* In Eq. (4), what is the relationship between the time window $w$ and the limit of subsequence size $\\delta_{1}$ and $\\delta_{2}$？\n* In Algorithm 1, what is the meaning of the function Sort()?\n* How to select an appropriate subsequence amount K?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e-hZbQkpnz7", "reviewer": "Reviewer_z5Gv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new way to compute Shapley values when presented with time series data. In particular for fraud detection in time series settings, understanding the reason behind the fraud detection algorithms is essential in many real-world settings. In this paper, the authors first proposed to split the time series into subsequences and then explain the prediction of the sequential models in terms of the features as well as the subsequence itself. In particular they propose a computationally efficient way to compute shapley values on each of each of the subsequences which thus allows a practitioner to understand which segment of the time series was most important. In addition, due to their setup they are also able to look into the importance of the features themselves as well.\n\nThey validate the idea on two unknown datasets and show superior performance on both datasets compared to naive approaches.\n", "summary_of_the_review": "I have clearly outlined my concerns about the paper above and will be looking forward to hearing from the authors.\nI am also more than happy to increase my score if the above have been addressed.", "strength_and_weaknesses": "I will start with the strengths of this paper:\n\n- Interesting setting to consider Shapley values in the sequential data setting.\n- An interesting way to find the subsequences using MMD\n- New way to think about how to interpret time series data i.e. in terms of subsequences which is more useful to a practitioner than each time step separately.\n- Show improved performance compared to naive alternatives.\n\nNext, the weakness of the paper and what needs to be clarified\n- The clarity of the paper is not very good, firstly, when trying to find the subsequences, how many time does one need to compute the MMD? Intuitively speaking the worst case, the choices of splitting points could be enormous i.e. searching through every configuration of K subsequences. How do you deal with this and could you please add the number of MMD evaluations required (on average?). I might have missed this, please direct me to the place where you mention this.\n- The paper in section 3.3 states that the algorithm has two stages. if I understand correctly, stage one aims to look at the Shapley values irrespective of the subsequences and then the second stage takes into account the subsequences. (looking at eq 7 here). Is that correct?\n- Next in eq10 the authors claim to \"apply KernelSHAP\". could the authors please elaborate on this and write down the actual optimization objective to solve here? I am confused about how to compute these Shapley values. The way I understand it atm is that in standard Shapley values we have a binary mask where we remove subsets of features and thus are able to determine the shapley values of each feature. Here, we are removing sets of features i.e. subsequences in batches, hence obtaining the Shapley values for the subsequence?. I would appreciate it if the authors would write out the exact objective and also justify how their methods still follow the 3 axioms for Shapley in their setting. The main reason is, I see many \\approx signs and I have no idea why they are approx and not equal.\n- Experiments:\n  -The authors only apply the model on proprietary datasets and have no synthetic toy experiments to show how their method works. Hence I am doubtful about how well the model actually performs. I highly suggest the authors run some synthetic data experiments to show how the methods work in a controlled environment where we know that feature 1-2 are important and 3-6 are not important at time step 1->k and vice versa for k+1-> T. That way the authors would also be able to verify how well the subsequence works. As it stands the authors seem to have no idea how well or what the subsequence even produces. All they know is that it is better than random and uniform. Hence I would highly recommend running some synthetic experiments to really illustrate how the method works through ablation.\n- The next thing that has to be added is the sensitivity to K the authors should run a study varying over K to understand how sensitive the algorithm is to K. I assume at some point random/uniform will be just as good as your MMD/KL version. Please add these simple experiments on the toy experiments. In your dataset A/B what does the MMD produce? i.e. what is the distribution of the subsequence lengths? I assume there is no pattern as else the uniform would perform well as well?\n- There are no standard deviations on the table of results and hence to my eyes useless and unreliable especially because there is no code. How many times were the experiments run? and what are the 2std? are they overlapping?\n- How were the lengthscales of the MMD chosen? I see so many heuristics in the appendix that are simply not justified at all\n\n\nThese are my initial thoughts but I am more than happy to change my scores if the above have been addressed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject