Temporal Dependencies in Feature Importance for Time Series Prediction
Authors: Kin Kwan Leung
Clayton Rooke
Jonathan Smith
Saba Zuberi
Maksims Volkovs
Venue: ICLR 2023 poster (2023)
Keywords: time series
recurrent
explainability

Abstract: Time series data introduces two key challenges for explainability methods: firstly, observations of the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance to model predictions over time. In this paper, we propose Windowed Feature Importance in Time (WinIT), a feature removal based explainability approach to address these issues. Unlike existing feature removal explanation methods, WinIT explicitly accounts for the temporal dependence between different observations of the same feature in the construction of its importance score. Furthermore, WinIT captures the varying importance of a feature over time, by summarizing its importance over a window of past time steps. We conduct an extensive empirical study on synthetic and real-world data, compare against a wide range of leading explainability methods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics.

Published as a conference paper at ICLR 2023
TEMPORAL DEPENDENCIES IN FEATURE IMPORTANCE FOR TIME SERIES PREDICTION
Kin Kwan Leung
Clayton Rooke Univ. Waterloo
Jonathan Smith Meta
Saba Zuberi Layer 6 Al
Maksims Volkovs Layer Al
Layer AI
ABSTRACT
Time series data introduces two key challenges for explainability methods: firstly, observations the same feature over subsequent time steps are not independent, and secondly, the same feature can have varying importance model predic- tions over time. In this paper, we propose Windowed Feature Importance in Time (WinlT), a feature removal based explainability approach address these issues. Unlike existing feature removal explanation methods, WinlT explicitly accounts for the temporal dependence between different observations of the same feature the construction of its importance score. Furthermore, Winl' captures the varying importance of feature over time, by summarizing its importance over a win- dow of past time steps. We conduct extensive empirical study synthetic and real-world data, compare against a wide range of leading explainability meth- ods, and explore the impact of various evaluation strategies. Our results show that WinIT achieves significant gains over existing methods, with more consistent performance across different evaluation metrics.
INTRODUCTION
Reliably explaining predictions of machine learning models is important given their wide-spread use. Explanations provide transparency and aid reliable decision making, especially in domains such as finance and healthcare, where explainability is often an ethical and legal requirement (Amann et al.) 2020 Prenio & Yong/ 2021). Multivariate time series data ubiquitous in these sensitive domains, however explaining time series models has been relatively under explored. In this work we focus on saliency methods, a common approach to explainability that provides explanations by highlighting the importance of input features to model predictions (Bachrens et al. 2010 Mohseni et al. 2020).
has been shown that standard saliency methods underperform on deep learning models used in the time series domain (Ismail et al. 2020). In time series data, observations of the same feature at different points in time are typically related and their order matters. Methods that aim to highlight important observations but treat them as independent face significant limitations. Furthermore, it is important to note that the same observation of a feature can vary in importance to predictions at different times, which refer as the temporal dependency in feature importance. For example, there can be delay between important feature shifts and change in the model's predictions. These temporal dynamics can be difficult for current explainability methods to capture.
To address these challenges, we propose the Windowed Feature Importance in Time (WinlT), feature removal based method which determines the importance of given observation to the pre- dictions over time window. Feature removal- based methods generate importance by measuring the change in outcomes when particular feature is removed. However, removing a feature partic- ular time does not account for the dependence between current and future observations of the same feature. WinIT addresses this by assigning the importance of a feature at a specific time based on a difference of two scores, that each take the temporal dependence of the subsequent observations into account. To capture the varying importance of the same feature observation to predictions at differ- ent times, WinlT aggregates the importance its final score. This allows our approach to identify important features that have delayed impact on the predictive distribution, thus better capturing temporal dynamics in feature importance.
predictions over window of time steps generate
is well known that the evaluation explainability methods is challenging because no ground truth identification of important features exists for real world data (Doshi- Velez & Kim 2017). A

---

Published as a conference paper at ICLR 2023
common approach is to evaluate the impact of removing features that are highlighted important by the explainability method (Lundberg et al. 2020) by masking them with a prior value. For time series data, an additional challenge for evaluation is that since observations of the same feature are related, removing the information from an important observation is non-trivial. This fact is under-explored in prior work. We present detailed investigation of masking strategies remove important observations and demonstrate their large impact on explainability performance evaluation. We present several masking strategies with complementary properties that we propose should be part of comprehensive evaluation scheme.
In summary, our main contributions are:
We propose new time series feature importance method that accounts for the temporal depen- dence between different observations of the same feature and computes the impact of feature observation to predictions over window of time steps.
We conduct a detailed investigation of evaluation approaches for time series model explainability and present several complementary masking strategies that we propose should be part of robust evaluation scheme.
We expand synthetic datasets from Tonekaboni et al. (2020) to explicitly evaluate the ability of explainability methods to capture shifted temporal dependencies.
We conduct extensive experiments and demonstrate that our approach leads to significant im- provement in explanation performance on real-world data, and is more stable under different eval- uation settings.
2 RELATED WORK
A wide range of saliency methods have been proposed in the literature. These include gradient based methods such as Integrated Gradients (Sundararajan et al. 2017)). GradientSHAP (Erion et al.) 2020). LIFT (Shrikumar et al., 2017), and DeepSHAP (Lundberg & Lee 2017). which lever- age gradients of model predictions with respect to input features to generate importance scores. Feature removal based methods such as feature occlusion (Zeiler & Fergus 2014) and feature ab- lation (Suresh et al., (0177) are model-agnostic methods which remove feature from the input data and measure the changes in model predictions. Model based saliency methods, such Choi et al. (216)); Song et al. (2188; Xu et al. (2018); Kaji et al. (2019) for attention-based models, use the model architecture, applied to time series models, these explainability methods do not directly consider the temporal nature of the problem and have been shown to underperform (Ismail et al. 2020).
this case the attention layers, to generate importance scores. However, when
In contrast there has been little work on saliency methods for time series. TSR (Ismail et al. 2020) separates the importance calculation along the time and feature input dimensions to improve per- formance. FIT (Tonekaboni et al. 2020) measures each observation's importance to the prediction change at the same time step using a KL--ivergence based score. Dynamask (Crabbé & van der Schaar 1 is perturbation method that leams a mask of the input feature matrix highlight important observations. We propose a new feature removal based saliency method that explicitly accounts for the temporal nature of the data in novel way.
To understand how our work relates to other feature removal based methods we can utilize the unified framework presented in Covert al. (202T) that categorizes feature removal based methods into three dimensions: feature removal method, model behaviour explained and summarization method. Approaches such feature occlusion (Zeiler & Fergus 2014). feature ablation (Suresh al.) 2017) and RISE (Petsiuk et al.) 2019) replace the removed features with a specific baseline value. Like us, approaches such FIT (Tonekaboni et al. 2020) and FIDO-CA (Chang al. 2019) use a generator to replace the removed features. In terms of model behaviour explained, like most methods we aim to explain the model prediction before and after feature removal, while methods such as INVASE Yoon et al. 2019) measure the change in the loss instead. The summarization method of explainability approach describes how the importance score generated. Feature occlusion uses the LI norm of the difference in model predictions, which does not account for the temporal nature of the problem, while FIT uses the KL-divergence between the prediction with and without a feature at the same time step. Our summarization method differs from these approaches by accounting for the importance of a feature observation over multiple time steps, improving our ability to capture temporal patterns in feature importance.
2

---

Published as a conference paper at ICLR 2023
I(S. a.
i(S)
i(S)
i(S)
X
1.b
X B.b
Fastures
plybl
p
plyb
6
5
a
P (b)
Time
(a)
Figure 1: (a) The importance of feature by the difference of the importance of S over multiple time steps a... b and a a 1..1... 6, denoted by a(S)) and i(S)a+1 respectively. (b) (S)) is obtained by comparing P. the model prediction given the full input feature matrix to time b, X1.b, and Q, the prediction when the feature S from a...
single time step a on the prediction at time b given
are unknown, indicated in blue and denoted by X a:b- These features are removed using generative model. g is measure used compare the distributions P and Q
3 OUR APPROACH: CAPTURING TEMPORAL DEPENDENCIES IN FEATURE IMPORTANCE
In this section we introduce our approach WinlT. We assume that multivariate time series has been observed up to time step T and a black-box model, fe. has generated predictive distribution at each time step. The change in predictions over time represents updated state of the model as new data becomes available. WinIT assigns importance by summarizing how a given observation has impacted model predictions over time.
We denote X R Px as a sample of a varti-variate time series with features and time steps. We also let x, X.,
f RDD be the of all feature observations a particular time and X1 [x1; X2; ...; xt] Dxt. Let yt {1...K² K} be the label each time step for classification task with classes. The model, fo. estimates the conditional distribution p(y: X1:e) each time step f. Let C (l...., DJ be subset of features of interest, S its complement, and x be the observations of that subset at time t.
T
A given observation x can impact predictions all subsequent time steps. Our main goal is to generate an importance score for each observation that summarizes its impact on the following prediction time steps. We denote I(S, a, b) the importance the feature set S at time the prediction at time = b, where b < a N is bounded above by window size N. To compute the total importance for S a, we aggregate T(S, a, b) across the window N.
on
3.1 IMPORTANCE SCORE FORMULATION
To define T(S, a, b) we first determine the importance i(S) ) to the prediction at time = of S over multiple time steps a...b b. which define in Eq. 2 using feature removal based approach. The importance of S at a single time step t = a on the prediction at t is then given by:
I(S, b)
{(S) i(S) (i(S)
att
a<b a<b<atN b=a
(1)
where N is the window size. Since there is dependence between subsequent observations of the same feature, eg. x and x in order to isolate the importance of feature set S at on the prediction we can not simply remove a single observation, and instead we must account for importance of subsequent observations of S. This is done by formulating the importance in Eq. as a difference between the importance over the time intervals (a, b) and [a + 1, b). This represented pictorially in Figure
In our formulation, the importance i(S) over multiple time steps in Eq. can be obtained using variety of feature removal based importance methods (Covert al. 202T). Let X denote the input time series when all features from time steps 1...b are known and X := X1.01.X X denote the input features from 1...6 when the feature set S unknown at [a, b). We focus approaches that evaluate the impact of feature removal the model prediction and consider the
3

---

Published as a conference paper at ICLR 2023
following measures to compare the two distributions P p(yb (1:b) and Q p(vo x
(pn(5) = P 1l1i (5) - (KI(())));; ins(s) (P
IM) (
2
where M = _ Here PD, KL and JS denote prediction difference, KL divergence and Jensen- 2
Shannon divergence respectively. This is represented pictorially in Figure Ib. where y denotes the measure used to compare the distributions P and Q.
Prediction difference is similar to methods such as Feature Occlusion (Suresh et al. 2017). while KL based scores used in Tonekaboni et al.] (2020); Yoon et al.
(a)
((S,ue.
((S%,
((
(019)). however these formulations are different to that in Eq. 2 Based on empiri- cal evaluation, we adopt Prediction Differ-
(6)
ence for (()) in our approach and present details of the empirical comparison in Sec- tion 5
1(5, (1S.
((5).
i( h1-1
To gain further intuition into the formu- lation of our importance score, we con- sider different scenarios depicted Figure 2 In our toy example, there is a signal in the feature set S at time step t a1 that causes the prediction to change at future time step f. = b. For f = An < a1, both i(5), and i(5) would be large as the signal is contained in both intervals (ag. b] and |ao- 1. For t ag > dj, both i(S), and i(S) a2+1 would be small as the signal is not contained either [az, b] or
(c)
1(S,a,,))
=
((S) ~0
(S)
toy example in which there signal in feature S at time = that causes a prediction change at time tance scores for S at / og. d1, 02 are illustrated in (a), (b) and (c) respectively. P indicates the prediction distribution with all inputs, while Q is the prediction with the the grey feature observations removed.
Figure 2:
b, where ag a < 02 b. The impor-
leg + 1. b). Thus in both cases, the importance I(S,ag ao. b) and [(S, 02 b) would be small. In con- trast, T(S, a1,b), b) would be large as i(S)) large because the signal (a1, b) while i(S) as +1 is small because the signal is not in [a) 1. Thus our formulation captures as important the moment of the feature change that would cause the predictions to change.
3.2 FEATURE REMOVAL WITH GENERATIVE MODEL.
To obtain the importance score for WinIT we need to compute p(ys g where X
a:b X XI ozb. This is the predictive distribution where all feature values are known up to time step except for feature set S from to Approaches to feature removal include setting features to a default value (Petsiuk et al. 2019). sampling from a fixed distribution (Suresh et al.] (017). or re- moving features by replacing them with samples from conditional generative model (Chang al.)
2019 Tonekaboni et al. 2020). We use generative approach to replace the removed observations with plausible counterfactual alternatives.
The marginal probability we would like to compute can be written as:
p(m 5 = p(v)) X 11a-1, 5%
E &x x 1) .)lp(v//X X 11a-1, addr {
(3)
where the conditional distribution is evaluated at specific values of X a:b and averaged over the distribution of all values of X To estimate the distribution of X aih we use a non-deterministic feature generator Gş that uses past observations, from time 1...a 1, to generate distribution of
features S at time a.. b for a < b a a N
p(X S X1a-1) GS((X.a-1117
Here the second argument in G5 is the number of time steps the generator is going to generate, where b = a + n and < n < N. We sample L times from the distribution in Eq. and average to obtain the desired output in Eq. Details are presented in Algorithm I in Appendix A.4
4

---

Published as a conference paper at ICLR 2023
3.3 AGGREGATION
Our method provides an importance score for feature subset S at time L - a to the prediction at 1= b. This allows the same feature observation to have varying importance for different prediction time steps, and provides more granular view of temporal dependencies. However, it may also be of interest to provide single importance score for each time step that summarizes the overall impact of S. In this case we aggregate the importance scores across the forward prediction time window:
I(S, a)
Σ a(S,a,b, N+1 b=a
Ñ min(N N - 1,,7 1 -a)
(5)
In our framework we use I(S, a) to compute the importance for feature set S at time point and use to compare WinIT against leading baselines.
4 EXPLAINABILITY EVALUATION IN TIME SERIES
In most real world applications, the true importance of features to a model's prediction are not known. A common evaluation strategy for feature importance based explainability methods is to remove important features and measure the difference the model's performance (Hooker et al.) 2019 Lundberg al., 2020, Tonekaboni et al.,, 2020). Important features assumed be more informative to the model predictions, therefore the more important a removed observation is, the bigger the performance drop. For time-series, the important feature is masked by replacing it with its value at the previous time step or its mean value, for example. However, since subsequent ob- servations of the same feature are correlated, we cannot remove the information of an important observation by only masking that single value. TTonckaboni et al. (2200) removes the information in an important observation, x by masking all the following observations of that feature in the time series, ie. x + x = x vt' 2 t. We refer to this ask the END masking strategy. However, when evaluating explainability methods by removing important features using END masking, the actual number of observations masked may be very different depending on where they are in the time series- and thus provide unfair advantage methods that flag earlier time points as important. On the other hand, forcing the number of masked observations to be equal may also introduce bias as information spread over more time steps will be masked less. To this end, we adapt three masking strategies, defined below, to investigate their impact on results: (1) END: When an observation is declared important, we mask all the subsequent observations of that feature until the end; (2) STD: When observation is declared important, mask all the subsequent observations, until the fea- ture value has changed more than a standard deviations with respect to the first masked observation. (3) STD-BAL: We designate fixed number of masked observations, 7mm Here, iterate over the most important feature-time (S, t) instances (in order of importance) and apply STD until the number of masked observations reaches 7m..
The STD strategy stops masking once the observed value has changed significantly, reflecting the dynamics of the underlying time series. It therefore more conservative than the END approach, as will mask fewer observations. Furthermore, unlike the END approach, it does not favour explain- ability approaches that identify important observations earlier in the time series. STD-BAL follows a similar approach to STD, but fixes the number of observations masked. By comparing STD and STD-BAL. we can see the impact on evaluation from fixing the number of important observations to remove versus the number of observations that are masked. As we will show, the choice of masking strategy can lead to very different conclusions about the performance of competing explainability approaches. Given their complementary focus, we propose different masking strategies should be part of robust evaluation scheme for time series explainability.
EXPERIMENTS
For all experiments we use 1-layer GRU models for fo with 200 hidden units. We use Adam opti- mizer with learning rate 10-3 (10 for MIMIC-III) and weight decay 10-3 3 to train the model. In practice, S can be a single feature or a set selected by prior domain knowledge. For example, in the healthcare setting, S can be subset containing certain lab measurements, indicators of different
5

---

Published as a conference paper at ICLR 2023
Table 1: Performance on the simulated datasets. For WinlT we use window size of 10. All evaluations are conducted over 5-fold cross-validation and averaged.
SPIKE
DELAVED-SPIKE
STATE
METHOD
AUPRC
MEAN RANK
AUPRC
MEAN RANK
AUPRC
MEAN RANK
DEEP LIFT
0.887+0.100
=.207=
0.883+0.148
1.312
0.022+0.0
284.544+1337
GRADSHAP
0.426+0..04
2.739
0,465+0.112
2.67440.445
0.021
286.51
IG
0.378+g.065
3.0g0.21
0.421+0.102
3.081
0.022+0.0
285. 285.12+1.188
FO
0..332 10.332
5.221:22794
0.00511.001
15.61614.515
0.027
198. 1.964
AFO
0.944
1.011=0.01
0.005+0.001
26.051 +13.466
26.05
0.027
201.64
FIT
0.
1.019
0.002+0.001
167.
167.615+34.068
0.23+0.013
116.08+12.44
DYNAMASK
0.503
1.0
0.50210.002
1.125 10.171
0.278
L0.000
19.91
WINIT
1.0
1.0
0.996
1.004
0.26+0.01
84.69+2.058
chronic diseases. For simplicity, all experiments we set = {i} to contain a single feature. We also use 1-layer GRU model for the generator with 50 hidden units, and train it by fitting a Gaus- sian distribution with diagonal covariance to reconstruct each feature N time steps forward. We use Adam optimizer with learning rate 10-4, weight delay 10-3 and 300 epochs with early stopping. To estimate the predicted distribution with masked features we average L = 3 Monte Carlo sam- ples from the generator, taking more samples does not improve performance significantly. We measure stability by splitting the training set into folds and report results averaged across the folds with corresponding standard deviation error bars. To demonstrate that WinlT performs equally well on other common architectures we apply it to a 3-Layer GRU, LSTM, and ConvNet on MIMIC-II and results are shown in Appendix A.6 All the results with STD masking are with experiments were performed with 40 Intel Xeon CPU@2.20GHE cores and Nvidia Titan V GPU.
1. All
We compare our approach against leading baselines: Gradient-based methods. This includes meth- ods that use the gradient of the prediction with respect to the feature-time input. We compare against Deep-LIF' (Shrikumar al. 2017). DeepSHAP (Lundberg & Lee 2017) and Integrated Gradients (IG) (Sundararajan et al.J (017). For these comparisons we make use of the implementation pro- vided the Captum library (Kokhlikyan et al. 2020). Feature occlusion (FO) (Suresh et al. 2017) Feature importance is assigned based on the difference in prediction when each feature is replaced with a random sample from the uniform distribution. We also include Augmented Feature Occlu- sion (AFO) which was introduced in Tonekaboni et al. (2020) to avoid generating -utto--diittibution samples. Feature Importance in Time (FIT) (Tonekaboni et al.) 2020) assigns importance to an observation based where the rest of the features are unobserved at the last time step. Dynamask (Crabbé van der Schaar 1 is perturbation-bate -based method for time series data that learns mask for the input sequence, that is almost binary.
the KL-divergence between the predictive distribution and counterfactual
5.1 SIMULATED DATASET
Dataset Spike is benchmark experiment presented in Tonekaboni et al. (2020) which uses a mul- tivariate dataset composed of 3 random NARMA time series with random spikes. The label is 0 until a spike occurs in the first feature, at which point it changes to for the rest of the sample. We also add a new delayed version of this dataset, Delayed Spike, where the point at which the label changes to is two time steps after the spike occurs. In this case, the observation that causes the label to change mark dataset State (Tonekaboni et al., 2020), which uses a non-stationary Hidden Markov Model to generate observations over time and presents more challenging task.
1 is the first spike two time steps before. We also include the synthetic bench-
Evaluation Since here we know the exact reason for the positive label, we can use this reason the ground truth importance to evaluate the explanations. The explanation label the specific observations that cause the prediction label to change is 1, otherwise it is O. We evaluate our feature importance using AUPRC. In addition, we introduce the mean rank metric for each instance we compute the rank of the feature importance score of the positive ground truth observations within that instance. We take the average of this rank across all test instances. See Appendix(A.5ffor further discussion of the metrics. Mean rank is especially effective in assessing the performance of the spike and spike-delay datasets, as there is only one positive ground truth observation per instance.

---

Published as a conference paper at ICLR 2023
Table Performance on the MIMIC-III mortality task using END masking.
PERFORMANCE (TOP 5%)
PERFORMANCE (K=50)
AUC DROP
PRED. CHANGE
AUC DROP
PRED. CHANGE
DEEP LIFT
0.005
0.031
0.000
0.02940.000
GRADSHAP IG
0.002200004
0.03
0.001
00007
0.001
0.034
0.02
0.027+0.002
FO
0.047 0.047=00004 0.004
0.045
0.067
40.004
AFO
0.019
0.05
0.066
0.06
FIT
0.066=0.007
0.05644.00 003
0.073=20.007
0.07 00003
DYNAMASK
0.068
0.068
0.082_0.007
0.076
WINIT 0.098 20.005
0.08 0.08g0.004
0.099 =00008
0.079 #0.004
Labela
Results. The results on simulated data are pre- sented in Table [ We see that WinlT outperforms other methods on the spike and delayed spike datasets in AUPRC and is on par with Dynamask in mean rank. Unlike WinIT, the performance of other fea- ture removal based meth- ods, FO, AFO and FIT, degrades significantly for the delayed spike experiment compared to spike. This indicates that the feature removal based importance formulation in WinIT is more robust to de- layed importance signals. Similarly we note that Dynamask and the gradient based approaches are able to adapt to the delayed signal. The strong performance robustness to non-stationary signals. Saliency maps for the delayed spike experiment are shown in Figure 3 WinlT captures the important feature in this setting, while gradient- based method, IG, also highlights subsequent spikes. A more complete set of saliency maps are shown in Appendix A.2
Data
Predictions
Ground
Integrated
Gradients
AFO
Winl
Figure 3: Saliency map for delayed-spike dataset.
the state dataset also shows WinlT's
Note, unlike other methods Dynamask encourages the importance result it generates a substantial number of repeated values and this impacts its evaluation on sim- ulated data. The results in Table [ use the trapezoid rule compute AUPRC, following CCabbé & van der Schaar (202T). Without interpolation (Davis & Goadrích] (2066), Dynamask's AUPRC much lower (0.038 E 0.0 for the state experiment), while other methods AUPRC values remain similar. Similar problem arises for mean rank. See Appendix A.5.2 for further analysis.
be almost binary. As
5.2 MIMIC-III MORTALITY
Dataset MIMIC-III is multivariate clinical time series dataset with range of vital and lab mea- surements taken over time for around 40,000 patients at the Beth Israel Deaconess Medical Center in Boston, MA JJohsson et al. 20166). It is widely used in healthcare and medical AI-related research. There are multiple tasks associated, including mortality, length-of-stay prediction, and phenotyping Waan et al. 2020). We follow the pre-processing procedure described in Tonekaboni et al. (2020) and use 8 vitals and 20 lab measurements hourly over a 48-hour period to predict patient mortality.
Evaluation We report performance using both global and local evaluation, as is done in Tonekaboni et al.] (2020). The global method corresponds to measuring the impact of removing the observations with the highest (top 5%) importance scores across the test set. While the local method corresponds to removing the observations with the highest (top K = 50) importance scores within each instance. For both experiments we present the metrics AUC drop and average prediction change when im- portant features are masked. Average prediction change, defined as the average absolute prediction change, allows us to evaluate the explainability method without relying on the ground truth labels in the dataset. Our base model has AUC of 0.81 = 0.003 in the test set. For the masking method, use the END masking method from Tonekaboni et al. (2020) for comparison purposes (See Section
We explore the impact of other masking strategies Section 5.3
7

---

Published as a conference paper at ICLR 2023
AUC Drop
Pred. Change
Masked Count
STD
STD
ais
STD-BAL
STD-BNL
nveauu
T
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
0.08
230
500
750
1000
1250
Dynamask
GradSHAP
WinT
Figure 4: AUC Drop, Mean Prediction Change and the number of masked observations for STD- BAL, STD and END masking method for different explainability methods on K 50.
Results. Our main results are presented in Table WinIT uses a window size of 10. We can see that WinIT significantly outperforms all the other methods for both the global top 5% and local K 50 experiments. WinIT improves over baselines both in terms of identifying observations that impact AUC most and the change in prediction. WinIT has a 44% increase in AUC Drop and 15% increase in prediction change over the leading baseline Dynamask on the top 5% experiment. Compared to other feature removal based methods, FO, AFO and FIT, the gain is even larger.
5.3 ADDITIONAL ANALYSIS
Impact of masking strategy. As discussed in Section when evaluating explainability methods by removing features and measuring the impact on performance, the strategy used to mask observa- tions can introduce bias into the evaluation. In Figure we show results for the K 50 MIMIC-III mortality task using the masking strategies outlined in Section 4 We see that masking has a large impact on the performance. WinIT outperforms other methods across all masking strategies, how- ever the relative performance of gradient versus feature removal based methods, FO, AFO and FIT, methods. Although for given explainability method
changes between the END and STD/STD-E the AUC drop and prediction change is highest with the END strategy, which tends to mask the most observations, this approach penalizes approaches (such as gradient- based methods) that have bias towards more recent observations. This can be seen in Figure 5 which shows the counts of masked observations at each time step for different explainability methods We see that the gradient- method IG masks more recent time steps for all masking strategies, likely due to the vanishing gra- dients phenomenon (Pascanu et al. 2012). In contrast the masked observations for methods such as AFO, WinlT and Dynamask more evenly distributed across time steps.
STD masking reduces the evaluation bias of END masking against explanation methods that flag features later in the time series. While STD-BAL makes sure the total number of observation masked are roughly the same across all methods. Given their impact on evaluation, these masking strategies should be used part of broader set of evaluation methods in order to make the evaluation of time series explainability methods more robust.
AFO
IG
Dynamask
Winl
Medied counts
0
20
40
20
40
20
90
20
40
0
20 90
Timesteps
END
STD
STD BAL
Figure 5: Count of masked observations for Integrated Gradients, FIT, Dynamask and WinIT at each time step in the MIMIC-III K = 50 performance drop experiment. Note that the bar charts are overlapping, instead of stacked.
Note the distribution of masked observations for Gradient SHAP and Deepl IIF is similar to IG; and the distributions of FO is similar AFO.
8

---

Published as a conference paper at ICLR 2023
40
0.12
END STD -ST--BL
0.1
0.8
AUPRC
Mean Rank
AUC Drop
0.08
0.6
20
0.06
0.04
0.2
0.02
6 10
10
15
20
Window Size (a)
Window Size
(b)
Figure 6: (a) WinlT performance on the Delayed Spike dataset as the window size from to 10. (b) WinIT AUC Drop performance on the MIMIC III mortality task (K=50) as the window size from to 20 for different masking schemes, Error bars show the standard deviation across the folds.
Effect of window size N. WinIT assigns importance observations by aggregating the impact on predictions over a window of subsequent time steps. As the window size increases, WinlT captures longer range interactions between important signals and changes in predictions, which can lead to better performance since some signals can be heavily delayed. In Figure 6a we analyze this effect for different window sizes on the delayed spike dataset, where the delay exactly two time steps. As expected we see that as N increases to 3, the performance, substantially improves. Note that N corresponds to looking backward two time steps (plus the current one). We also see that the performance quickly stabilizes after N 3.
similar pattern can be observed for the MIMICIII mortality task shown Figure (6b] We see that the AUC Drop increases with window size and then stabilizes, with large range of choices of the hyper-parameter N leading to similar results. We also note that there is linear relationship between time and window size, shown in Appendix A.4] allowing for reasonable sweep to find a good choice of window size. This pattern holds for different masking strategies, demonstrating WinlT's robustness evaluation strategy.
Effects of Different i(S). We investigate the impact performance of using different defini- tions of the importance to the prediction at t = of S over multiple time steps [a, i(S). de- In Table 3 we show the AUC
Table 3: AUC Drop on MIMIC-III with STD masking using different i(S) in WinIT.
Top 5%
K=50
fined in Eq. 2 drop on the MIMIC-III mortality task using STD masking. We see that WinlT-PD gives some gain over the other approaches when dropping the top 50 observations for each time series. We opt use the prediction difference in our formulation. It is worth noting that all three approaches outperform the other baselines.
WINIT-KL.
0.076 10.006
0.072
WINIT-JS
0.075 10.007
10.0010.007
WINIT-PD
0.078 0.07840.000 00.008
0.076 +00.06
CONCLUSION
In this paper we propose WinIT, feature removal based explainability method that produces a fea- ture importance score for each observation multivariate time series. Unlike prior approaches, WinIT calculates the importance as the difference of the importance over multiple time steps and aggregates the importance of each observation over window of predictions. Empirically, WinIT outperforms leading baselines on both synthetic and real-world datasets with more consistent perfor- mance under different evaluation strategies. For future work, we would like to tackle two possible limitations in WinIT. First, we would like to explore approaches to automatically determine the window size parameter introduced in our method. Secondly we would like to extend to time series explainability approaches that do not take the form of saliency maps, with single importance for each observation, but which instead focus on identifying important higher-level patterns or trends series of observations, believe this form of output would further aid interpretability.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we describe the implementation details of our approach and conduct evaluation experiments on publicly available datasets. In addition, we follow the pro- cedures outlined in previous studies process the data. The code for our work is publicly available at|https / qithub. com/ hwtp:aalyerraaaa labi/ WinIT which includes the detailed settings for experiments.
ACKNOWLEDGMENTS
We would like thank the anonymous reviewers for their feedback, which helped improve paper. We would also like to thank Gabriel Loaiza-Ganem for useful discussions. This work was conducted while Clayton Rooke and Jonathan Smith were at Layer 6 AI.
REFERENCES
Ahmed Alqaraawi, Martin Schuessler, Philipp WeiB, Enrico Costanza, and Nadia Berthouze. Eval- uating saliency map explanations for convolutional neural networks: A user study. CoRR, abs/2002.00772, 2020.
J Amann, A Blasimme, E Vayena, D Frey, and V.I. Madai. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making, 20(310), 2020.
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 1 11(6)), 2010.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference Learning Representation, 2019.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In Neural Information Processing Systems, 2016.
Ian C. Covert, Scott Lundberg, and Su-In Lee. Explaining by removing: A unified framework for model explanation. Journal of Machine Learning Research, 22((09), 2021.
Jonathan Crabbé and Mihaela der Schaar. Explaining time series predictions with dynamic masks, In International Conference on Machine Learning, 2021.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning. ICML '06. pp. 233-240, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1143844 143874. URL htllhttps://dddo /doi www.wo.g/......... 111111.111111111 1143874
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv: Machine Learning. 2017.
Gabriel G. Erion, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, and Su-In Lee. Learning explainable models using attribution priors. CoRR, abs//906. 10670, 2020.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. benchmark for interpretabil- ity methods in deep neural networks. In Neural Information Processing Systems, 2019.
Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. Benchmark- ing deep learning interpretability in time series predictions. In Neural Information Processing Systems, 2020.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. MIMIC-III, freely accessible critical care database. Scientific Data, 3, 2016.
10

---

Published as a conference paper at ICLR 2023
DA Kaji, JR Zech, JS Kim, SK Cho, NS Dangayach, AB Costa, and EK Oermann. An attention based deep learning model of clinical events in the intensive care unit. PLoS One, 2019.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for PyTorch. CoRR, abs/2009.07896, 2020.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Neural Information Processing Systems, 2017.
Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2(1), 2020.
Sina Mohseni, Niloofar Zarei, and Eric D. Ragan. A survey of evaluation methods and measures for interpretable machine learning. CoRR, abs/1811. 11118811111899 2020.
Razvan Pascanu, Tomás Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/1211 2012.
Vitali Petsiuk, Abir Das. and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In International Conference on Learning Representation, 2019.
Jermy Prenio and Jeffry Yong. Humans keeping Al in check emerging regulatory expectations in the financial sector. FSI Insights on Policy Implementation, 35, 2021.
Qiaomu Shen, Yanhong Wu, Yuzhe Jiang. Wei Zeng, Alexis K H LAU, Anna Vianova, and Huamin Qu. Visual interpretation of recurrent neural network on multi-dimensional time-series forecast. In IEEE Pacific Visualization Symposium, 2020.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, 2017.
Huan Song, Deepta Rajan, Jayaraman J. Thiagarajan, and Andreas Spanias. Attend and diagnose: Clinical time series analysis using attention models. In AAAI Conference on Artificial Intelligence, 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017.
Harini Suresh, Nathan Hunt, Alistair E. W. Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. CoRR, /55/1705.08499 2017.
Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David K Duvenaud, and Anna Goldenberg. What went wrong and when? Instance-win feature importance for time-series black-box models. In Neural Information Processing Systems, 2020.
Shirly Wang, Matthew BA McDermott, Geeticka Chauhan, Marzyeh Ghassemi, Michael C Hughes, and Tristan Naumann. MIMIC-extract A data extraction, preprocessing. and representation pipeline for MIMIC-III. In ACM Conference Health, Inference, and Learning, 2020.
Yanbo Xu, Siddharth Biswal, Shriprasad R. Deshpande, Kevin O. Maher, and Jimeng Sun. RAIM: recurrent attentive and intensive model of multimodal patient monitoring data. In ACA SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. Invase: Instance-wise variable selection using neural networks. In ICLR, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision, 2014.

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 SUMMARY OF NOTATION
A summary of the notation used throughout the paper is provided in Table 4
Table 4: Summary of Notation
NOTATION
DESCRIPTION
T
Number of time steps in the time series.
D
Number features in the time series.
Jo
A trained model that outputs the predictive distribution given the input time series.
X
The time series of D features and T time steps.
The vector of all feature observations at time step 1, 1 S & T.
X1.
The set of all observations in the time series X up to time
The number classes for a given classification task.
The label at time step for a classification task with A classes.
5
The subset of features of interest, $ _ {1...., D)- The complement set to the features of interest (1..... D) \
5t
The observation feature set at time-step
a.
X
The observations of feature set S from = a...b.
2.6
X
X with observations in feature set S removed from = a a...b.
i(S)
b of S over time steps The importance of the feature set S at time 1 a on the prediction at time A non-deterministic generator for feature S using past observations.
The importance the prediction time
a,)
Gs
N
The maximum window size.
N
min(N N 1,T a).
I(S,a)
The feature importance score aggregated over the window b a...N. N.
A.2 SALIENCY MAPS
Saliency maps one way of visualizing feature importance and have been found to be helpful for interpreting predictions for image classification in user studics Alqaraawi et al. (2020). In the mul- tivariate time series setting Shen et al. (2020) found visualizations can also improve interpretation.
Saliency maps for the simulated spike and delayed spike datasets are shown Figure 7 and respectively. These maps illustrate that WinlT captures the most important features in both the spike and delayed spike setting. The gradient- based methods, capture the important observation, but also highlight subsequent spikes that do not impact the change in label.
We also shows the saliency maps for the MIMIC-III mortality experiments in Figure 9 We see that gradient- based methods tend to highlight later observations in the time series important. In contrast FO and AFO tend to flag earlier points in the time series. FIT tends to mark many features within the same time step important, while the salient observations are more distributed in time for WinIT and Dynamask.
A.3 AGGREGATION METHODS FOR WINIT
In section 3.3
we introduce a method of aggregation to compute the importance of feature set S by taking the mean of the importance scores across time window, according to Here we consider an an alternate choice aggregation, given below:
at timestep t =
Eq. 5
Imaz (S,a)
max I(S,a,0 bE(a,a+ NN
(6)
where N max(N - 1,7 a). Table 5 shows the results for STD masking on the MIMIC-III
mortality task with different aggregation choices. We see that while the performance for different aggregation methods are similar, taking the mean gives the best results, in terms of both AUC Drop and prediction change. One explanation for this is that if an observation is important for multiple future predictions, it is more important than the case when an observation is important for a single future prediction.
12

---

Published as a conference paper at ICLR 2023
Labels
Data
Predictions
Ground Truth
Deep LIFT
Gradient
SHAP
Integrated
Gradients
FO
AFO
FIT
WinIT
Figure 7: Saliency maps for simulation spike
Labels
Data
Predictions
Ground Truth
Deep LIFT
Gradient
SHAP
Integrated
Gradients
AFO
FIT
Win
Figure Saliency maps for delayed simulation spike
A.4 ALGORITHM AND RUN TIME ANALYSIS
We provide outline of the WinlT algorithm in Algorithm
We also explore the run time for different window sizes, N, and present our results in Figure I0 We see that the run time has linear relationship with the window size. Since the importance calculation for each feature is independent in our approach, we are able to parallelize this computation which improves the run time.
A.5 EVALUATION SIMULATED DATA
A.5.1 METRICS FOR SPIKE AND DELAYED SPIKE DATASETS
In this section we provide insight into the metrics used to evaluate the simulated spike and delayed spike datasets. These datasets have only one positively labelled observation for each time series and the ground truth explanation is known. Here we consider AUPRC and mean rank, which we use for evaluation and AUROC, another common metrie, which show is less suitable for this case.
13

---

Published as a conference paper at ICLR 2023
Gradient SHAP
Deep LIFT
Integrated Gradients
FO
AFO
Dynamask
WinTT
Figure 9: Saliency maps for MIMIC-III
Table 5: AUC Drop and Prediction Changes for different aggregation methods in WinIT on the MIMIC-III mortality task
PERFORMANCE (TOP $ 3%)
PERFORMANCE (K=50)
AUC DROP
PRED. CHANGE
AUC DROP
PRED. CHANGE
MEAN
0.078 10.005
0.07 004
0.076 008
0.07
MAX
0.066+0.008
0.06140.00
0.063
0.059_0.003
For simplicity, assume all the feature importance values distinct. Suppose there obser- vations time series. Let < r < n be the rank of the feature importance among all observations in the time series.
AUROC can be interpreted as the probability that the prediction of the Sesitieel-labellle obser- vation is greater than that of negatively labelled observation. As there is positively-labelled observation rank r in our case, this is the case for n r out of total of n -1 comparisons. Thus the AUROC is 7-1-
14

---

Published as a conference paper at ICLR 2023
Algorithm WinIT
Input: fo: trained Black-box predictor model; S: a subset of features of interest; N: feature importance window size: Gg: trained generative model; X1:7 E RPT time series where T is the max time and D is the number of features; L: number of samples Output: Importance score matrix I IR"
1: Train Gs using NI. for all 0 S N do
2: for all f: =
fo (1:e) 4: Initialize ((S) = 0 for all 0, b.
3: p(y:
5: for all n O... min(Ninniiii. 1,7 1) do
6: P(X
2 Gs(Gi 1 11 1)
7:
for all I := 1. do (0)
8:
Sample Sttttn ~ p( Xs Stttta (
9:
p(iitn) So 111-1 X 1: 3,1:44) end for
10:
p(yt+n X 1: X (t ) 1 Σ/= p(
11:
i(S)t** [py(t 1 p(V++n 1 X, xaal))))
I(,,t,) n) i(S) i(5)
14:
end for
end for
16: I(S,t)
1 1iT 1(S,t.n)
-1,7--11111)+3 m=
17: Return I
30
Run time (sec)
20
10
10
15
20
Window Size
Figure 10: WinTT run time on the simulated spike delay task for different window sizes.
For AUPRC, as there is only positively labelled observation, the recall can only or 0. When the recall is 1, the best precision is 1/r the positively-labelle observation is of rank When the recall O, if the rank r of 1, the precision is also Thus, for r > 1, there are only points in the precision-recall curve, (0, and (1, )). This means AUPRC is for 1. For AUPRC=1 by definition.
n-1 and AUPRC= 2.- Since AUROC depends which AUPRC only depends r, AUROC is less sensitive to changes in performance than AUPRC. For example, our simulated spike dataset contains 3 features and 80 time points, therefore the number of observations is n 240. If the Seaitivel--abbllles feature is the 4th most important feature, i.e. r 4, the AUROC is 0.9874 while the AUPRC is 0.125. the rank is 10, AUROC 0.958 and AUPRC is 0.05. This shows that AUPRC is much more sensitive to changes in rank that AUROC.
To summarize, when rank is AUROC=
To compare performance across explainability methods on the spike and delayed spike datasets, we argue that mean rank and AUPRC provide the clearest comparison.
15

---

Published as a conference paper at ICLR 2023
Table 6: AUPRCiinterpolation, average precision, mean rank and mean average rank for all methods in State dataset.
AUPRC-INNERPOLLATINN
AVERAGE PRECISION
MEAN RANK
MEAN AVG RANK
DEEP LIFT
0.022
0.022
284.54 11.237
289. 1
GRADSHAP
0.021
0.021
286.51 +1.462
29..4
IG
0.02240.0
0.02210.0
285. 285.1221.188
289. 289.0441. 1.03
FO
0.027
0.027
198. 498.444+1.966
198.4441..00
AFO
0.027
0.028
201. 101.6441.707
201.64
FIT
0.234
0.194
116.08
116.37
DYNAMASK
0.27810.000
0.278 10.003
0.03810.0
79.909
179.7710.849
WINIT
0.2620.01
0.26
84.69212.058
84.692.2.058
A.5.2 EVALUATING DYNAMASK ON SIMULATED DATASETS
As described in Section 5 Dynamask (Crabbé & van Schaar 2021 is expected to produce almost binary masks, while the importance scores for other methods are expected to be continuous values. This makes the evaluation of Dynmask on simulated data very sensitive to the way in which ties are resolved.
For completeness we present the impact of these choices on the AUPRC and mean rank metrics for the State dataset in Table 6 In particular present mean rank where ties resolved by taking the minimum rank and the average rank, where we refer to the latter as mean average rank. We also present the results of AUPRC computed with and without interpolation. We refer AUPRC without interpolation as Average Precision. In the main text we use AUPRC with interpolation following the presentation in Crabbé & van der Schaar (202I), and note that elsewhere in the literature (Davis & Goadrich (006) it is argued that average precision is preferred. We see that Dynamask performance drops significantly when interpolation is not used in computing AUPRC and average minimum rank is used to resolve ties. Figure 11 shows the precision-recal curve and the two different types of interpolations.
Precision Recall Curve for Dynamask
1.0
PRC (AP-0.038)
PAC using traperoidal interpolation (AUmc-0.278)
0.8
Proctzion (hoatue tabo:
0.6
0.9
0.0
0.2
10
(Positive 1)
Figure 11: Precision Recall Curve for Dynamask. The black dots correspond to the precision and recall for different thresholds.
A.6 ALTERNATIVE MODEL ARCHITECTURES FOR MIMIC-III MORTALITY
In this section show that WinIT continues to perform well different model architectures. We show the performance on the MIMIC-III Mortality dataset on three additional architectures: 3- Layer GRU, LSTM and ConvNet. We show the results using different masking scheme and show
16

---

Published as a conference paper at ICLR 2023
the performance for the K = 50 task. The performance of explainability methods follows similar trend for different model architectures.
AUC Drop
Pred. Change
Masked Count
STD-BAL
STD BAL
STD-BAL
0.00
0.02
0.04
0.06
0.08
0.00
0.02
0.04
0.06
0.08
200
400
600
800
Deep LIFT
K3
APO
Dynamask
GradSHAP
WinfT
Figure 12: MIMIC-III Mortality performance with a ConvNet model architecture. (Base Model AUC=0.746 20.0122
AUC Drop
Pred. Change
Masked Count
STD
ais
STD-BAL
STD-BAL
TVP-LIS
LP0
0.00
0.02
0.04
0.06
0.08
0.10
0.00
0.02
0.04
0.06
250
500
750
1000
1250
Deep
Dynamask WinfT
GradSHAP
FO
RT
Figure 13: MIMIC-III Mortality performance with an LSTM model architecture. =09=00)7700.003)
(Base Model
AUC Drop
Pred, Change
Masked Count
ENO
ONB
STD
als
STD-BAL
STD BALL
0.000
0.025
0.050
0.075
0.100
0.00
0.02
0.04
0.06
250
500
750
1000
1250
Deep LIFT
IG
AFO
Dynamask WinlT
GradSHAP
Figure 14: MIMIC-III Mortality performance with a 3-layer GRU model architecture. (Base Model LUC=0.7931o.003
A.7 TRAINING CURVES FOR GENERATORS
The training curves for some selected generators are shown in Figure [5 to avoid overfitting.
We apply early stopping
17

---

Published as a conference paper at ICLR 2023
Training curves for generator for feature
Training curves for generator for feature 9
Training Loss
Training Loss
Validation Loss
Validation
0.6
Ratiion Suma
my t
MS2
0.2
0.0
50
100
150
200
250
300
150
200
250
300
Training curves for generator for feature 14
Training curves for generator feature 26
Training Loss
Training Loss
Validation
1.0
Velidation
DUODO wooohhee Aupa
SU000 Buddos Auro
MSt
0.4
0.2
0.0
50
100
150
200
250
300
100
150
200
250
300
Figure 15: Example training curves for the generators
18


Reviews:
[{"review_id": "nqzBBm8eMn", "reviewer": "Reviewer_CWz4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a Windowed Feature Importance in Time\n(WinIT) method for explaining time series predictions. They have also investigated the evaluation approaches with different complementary masking strategies. Experiments have been conducted to show the proposed approach performs well in synthetics and real-world data sets.\n", "summary_of_the_review": "It is a technically solid paper. The clarity of the proposed measure can be further improved. Without the explanations, it is difficult to judge the empirical impact of the methods for dealing with the pattern signals.\n", "strength_and_weaknesses": "S1: The paper proposes a method for solving the challenges in explaining time series predictions. \n\nS2: The paper is well structured and easy to follow.\n\nS3: Experiments are comprehensive.\n\n\nW1: The proposed method looks incremental. The measure of feature importance at a time step follows the measure of Tonekaboni et al. (2020) by extending it to prediction difference and Jensen Shannon divergence.\n\nW2: The algorithm needs more justifications. The reason for the average over a window of size N is unclear to me. The window size N introduces another parameter to tune. Also, the definition of $i(S)^b_a$ needs more justifications. “We formulate the importance in Eq. 1 as a difference due to the time series nature of the data” is not clear enough to explain the intuition. Figure 1b has not been explained in the text. What does exactly $i(S)^b_a$ mean? Is a large (or small) value of $i(S)^b_a$ preferred? Without these explanations, Figure 2 is not easy to understand.  Furthermore, the predictions of time series with the information being masked should be presented. It is confusing that the predictions in all three cases (a) (b) and (c) are the same (the right hand sides before and after the minus symbol '-'). I wonder why $i(S)^b_a$s are different in the three cases when the predictions before and after masking are the same. \n\nW3: The example of Figure 1 is too simple to show teh usefulness of the proposed method.  In Figure 1,  Y is determined by a single signal. Many time series predictions are determined by patterns (multiple signals). There needs an example to show that the proposed importance measure can handle pattern signals to convince readers that EinIt is useful for real world applications.\n\nW4: In Figure 4, the AUC drops and Prediction changes are quite inconsistent. Can authors provide some explanations?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "beIZjKI6bWI", "reviewer": "Reviewer_vrY7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new explainability framework for multi-variate time series data. This new explainability framework is a feature removal model. The authors motivate their approach by the fact that the observation of same feature over subsequent time steps is not independent and that feature can have varying level of importance to the model prediction over time. Using extensive experimental study, the authors show that the proposed method is superior to the existing explainability models for time series data.", "summary_of_the_review": "This paper addresses the practically important problem of explainability of black box classifiers for time series data. The authors introduce a novel framework that calculates the feature importance as the difference of the importance over multiple time steps and aggregates\nthe importance of each observation over a window of predictions. The authors also a generative approach for the feature removal, which is interesting. The authors also provide extensive experimental studies to support that the new framework works better than the existing explainability approaches for time series data. Overall, this is a good paper. ", "strength_and_weaknesses": "Strengths:\n1-\tAddresses a very important problem\n2-\tNovelty\n3-\tExtensive experimental study\n\nWeakness:\n1-\tNotation could be improved\n2-\tAt places, it is difficult to follow the text.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AM0l8JfGCEH", "reviewer": "Reviewer_4Fvg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new method, WinIT, for interpreting predictions of multivariate time series. It provides an importance score for each feature in a multivariate time series according to explainability, considering the dependencies of features between time steps and variation in feature importance over time. The method considers differences between consecutive time points and measures importance over multiple time steps to quantify temporal variation in importance. The novelty is in combining these two aspects in the importance score. Analyses on synthetic and real data sets are used to demonstrate advantages and consistency compared to currently available alternatives and assess model evaluation.", "summary_of_the_review": "Overall, this is am interesting and well documented contribution on a timely topic. Reproducibility of the work is a key limitation. ", "strength_and_weaknesses": "Strengths:\n+ Addresses a relevant and timely issue of improving interpretability of machine learning techniques \n+ Comprehensive set of evaluation approaches\n+ The method shows clearly improved performance over alternatives\n\nWeaknesses:\n- Fixed window size is a potentially weak spot in the algorithm design as it will require heuristic window size selection schemes\n- LImitations and weaknesses of the method could be addressed in more detail as these are relevant for any subsequent application\n- Publicly available implementation does not appear to be available, limiting the practical utility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster