Explaining Time Series by Counterfactuals
Authors: Sana Tonekaboni
Shalmali Joshi
David Duvenaud
Anna Goldenberg
Venue:  (2020)
Keywords: explainability
counterfactual modeling
time series

Abstract: We propose a method to automatically compute the importance of features at every observation in time series, by simulating counterfactual trajectories given previous observations. We define the importance of each observation as the change in the model output caused by replacing the observation with a generated one. Our method can be applied to arbitrarily complex time series models. We compare the generated feature importance to existing methods like sensitivity analyses, feature occlusion, and other explanation baselines to show that our approach generates more precise explanations and is less sensitive to noise in the input signals.

Under review as a conference paper at ICLR 2020
EXPLAINING TIME SERIES BY COUNTERFACTUALS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a method to automatically compute the importance of features at every observation time series, by simulating counterfactual trajectories given previous observations. We define the importance of each observation as the change in the model output caused by replacing the observation with a generated one. Our method can be applied to arbitrarily complex time series models. We compare the generated feature importance to existing methods like sensitivity analyses, feature occlusion, and other explanation baselines to show that approach generates more precise explanations and is less sensitive to noise the input signals.
INTRODUCTION
Multi-variate time series data are ubiquitous in application domains such as healthcare, finance, and others. In such high stakes applications, explaining the model outcome is crucial to build trust among end-users. Finding the features that drive the output of time series models is challenging task due complex non-linear temporal dependencies and cross-correlations in the data. The explainability problem is significantly exacerbated when more complex models are used. Most of the current work in time series settings focus on evaluating globally relevant features (Yang al., 2005; Yoon et al., 2005; Hmamouche et al., 2017). However, often global feature importance represents relevant features for the entire population, that may not characterize local explanations for individual samples. Therefore we focus our work individualized feature importance in time series settings. In addition, besides identifying relevant features, we also identify the relevant time instances for those specific features, i.e., we identify the most relevant observations. To the best of our knowledge this is the first sample-specific feature importance explanation benchmark at observation level for time series models.
In this work, we propose a counterfactual based method to learn the importance of every observation in multivariate time series model. We assign importance by evaluating the expected change in model prediction had an observation been different. We generate plausible counterfactual observations based signal history, asses temporal changes in the underlying dynamics. The choice of the counterfactual distribution affects the quality of the explanation. By generating counterfactuals based on signal history, we ensure samples are realistic under individual dynamics, giving explanations that are more reliable compared to other ad-hoc counterfactual methods.
2 METHOD: FEED-FORWARD COUNTERFACTUALS FOR TIME SERIES EXPLANATION
In this section we describe our method, Feed Forward Counterfactual (FFC), for generating explanation for time series models. A feature is considered important if it affects the model output the most. time series, the dynamies of the features also change over time, which may impact model outcome. As such it is critical to also identify the precise time points of such changes.
Maad-boor oatcoms
Hdbll
Tpp1
lutwees
catcome
time
Impust signaal
comerratio
In order to find important observations for high- dimensional time series models, we propose to use counterfactual based method. Specifically,
Figure 1: Illustration of Proposed Method

---

Under review as a conference paper at ICLR 2020
Z
Predictor Network
Homi
Pan
10
Xip
Genenalor
(a) Graphical model of the conditional generator. ht represents the hidden state of an RNN cell time t, and 2, is the latent represen- tation history
101
Ape
X,
Counterfactual X, is generated using signal history. We look into the difference of the original output Ye and V(s.)) where observation ((.e)) is replaced with counterfactual.
(b) FFC procedure:
Figure 2: Components describing the proposed method.
we assign importance an observation Ti, (feature i at time t) based on its effect on the model output at time T(> t). We replace observation 24, with a counterfactual εi,e, to evaluate this effect. Figure demonstrates how importance of a feature is characterized by replacing an observation with a counterfactual.
2.1 NOTATION
Multi-variate time series data is available in the form of X E gdxT (where d is the number of features with observations over time) for n E [N] samples. We are interested in black-bo -bx estimators that receive observations up time t, X2 E Rd. and generate output YI every time point t E [T]. F denotes the target black-box model we wish to explain through the proposed approach samples has been dropped for notational clarity. We index features with subscript i. x-1,t indicates features at time / excluding feature i. The notation used for exposition work is briefly summarized in Table
called FFC. For exposition, throughout the paper, the index
Notation
Description
[] for integer K
Set of indices [K] 11, K}.
i,
Index for feature in [d]. time step t and sample in [VV] respectively
Set [d]]
Observations and Outcomes
Observation at time t.
Xi E Rd
Vector [Ir1,1, #2,1, Td,t.
Xox E Rdxt
Matrix Xo, X1,
V: & F(Xa:)
Observed outcome of the black-box model F, at time
Generative Model and Estimator
G: 12m-1xm
Conditional Generative Model sampling for feature Latent encoding of history up to
E R"
Table Notation used in the paper.
2.2 EXPOSITION
We assign importance score to each observation Zi,t for t [T'] and E [d], following the definition: Definition 1. Feature Importance: The importance of the observation i at time t, denoted by Imp(i,t) t) is defined as absolute value and L.. is the counterfactual sample.
(Xxo) F(xxa- X- Li],)., where | denotes the
That is, the importance of an observation for feature i time t is defined the change in model output when the observation replaced by generated counterfactual. The counterfactual observation come from any distribution, however the quality of the counterfactual random variable directly affects the quality of the explanation. We generate the counterfactual sample conditioned on signal history up time by sampling from the distribution p( Xoa-1). -1). Using conditional generator
2

---

Under review as a conference paper at ICLR 2020
guarantees that counterfactuals sampled not only within domain but also specifically likely under the individual sample ( as opposed to having generator that models data across population. Conditioning on the history also allows us to learn the dynamics of the signal and therefore generate plausible counterfactual given the past observations.
p(x, (0.--1) represents the distribution at time 6, if there were no change in the dynamics of the signals. The counterfactual êi. is sampled from the marginal distribution p(ba;;, (oor-1), obtained from ((x1(X Let F(X0:t11, x x,,, Bx,e) be the output of the model at time T, when Ii, is replaced by the generated counterfactual Zi.,. We estimate feature importance Imp(i, t) as xxtte..)oxx-)))[((XXXX - F(Xoo-1 x-xxx- &.))],, summarized in figure 2(b).
2.3 PROPERTIES
Our proposed method has the following compelling properties in explaining the estimator F:
Time Importance (TI) For every time series, highlighting relevant time events for different features important for actionability of the explanations. For instance in clinical setting, just knowing feature like heart rate is relevant, is not sufficient to intervene it is also important to know when a deterioration had happened. With FFC, the most eventful time instances can be obtained
arg max [Impli, ( t) Vi E [d]
(1)
We can thus rank time instances in order of importance. max4e(4] (I (Imp(i,tt)) +1) 2 maxie[4 (Imp(i, t2)}.
That is, time t
12. if
Feature Importance (FI) At any time instance t, our method assigns importance to every feature of 2t2--
The magnitude of our importance function reflects relative importance. Comparing the importance values across features gives the flexibility to report a subset of important features at each time point and also reflects the correlation between various features of the time series.
2.4 GENERATOR MODEL FOR CONDITIONAL DISTRIBUTION p(x, (0:-1)
We approximate the conditional distribution of p(x; ( using a recurrent latent variable genera- tor model G, introduced in Chung al. (2015). The architecture use is provided in Figure 2(a). The conditional generator G models p (xx (2-1) where 2--1 E R the latent representation of history of the time series up time The latent representation is continuous random variable, modeling the distribution parameters. We only use past information in the time series to reflect temporal dependencies. Using the recurrent structure allows to model non-stationary generative model that can also handle varying length of observations. Implementation details of the generator in the Appendix.
Our counterfactuals are not derived by looking at future values which could be done for reliable imputation. Counterfactuals should represent the past dynamics. Note that our derived feature importance is limited by the quality of imputation that may have been utilized by the black-box risk predictor. For experimental evaluation on the effect of generator specifications on counterfactuals and the quality of explanations, see Section 4.1.
2.5 FEATURE IMPORTANCE ASSIGNMENT ALGORITHM
The proposed procedure is summarized Algorithm 1. We assume that we are given a trained block box model F. and the data (without labels) it had been trained on. Using the training data, we first train the generator that generates the conditional distribution, (denoted by 9). In our implementation model
multivariate Gaussian with full covariance to model all existing correlation between features. The counterfactual a is then sampled from G and passed to the black-box model to evaluate the effect the black-box outcome.
3 RELATED WORK
common method explaining model performance, time-series deep learning, is via visualization of activations of latent layers (Strobelt et al., 2018; Siddiqui et al., 2019; Ming al., 2017) or via
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 FFC
Input: F: Trained Black-box predictor model, Xo.T, where the max time and Number of Carlo samples samlees
1: Train G
2: for all E (T] and i e [d] do
3: yr = =((oa
4: p(x, (Xott) ~ ((ox=11)
5: for all i E [$] do
6: Sample (10) ~ p(az, (o.-11)
UT = F(x0,, x 1,1s ((1))
7:
8:
impr
(S) = yr DT
9: Importance, Sepportance_Morria((( t)
10:
Return Lwpprttaneeeaaaeeeeee Matrix
sensitivity analysis (Bach et al., 2015; Yang et al., 2018). Understanding latent representations, sensitivity and its relationship to overall model behavior is useful for model debugging. However, these but are too refined to be useful to the end users like clinicians.
Attention models (Vaswani al., 2017; Vinayavekhin et al., 2018; Xu et al., 2018) are the most commonly known explanation mechanisms for sequential data. However, because of the complex mappings to latent space in recurrent models, attention weights cannot be directly attributed to individual observations of the time series (Guo et al., 2018). To resolve this issue to some extent, Choi al. (2016) propose an attention model for mortality prediction of ICU patients based clinical visits. However attention weights may not be consistent as explanations (Jain and Wallace, 2019).
In vision, prior works tackle explainability from the counterfactual perspective, finding regions the image that affect model prediction the most. Fong and Vedaldi (2017) assumes higher importance for inputs that when replaced by an uninformative reference value, maximally change the classifier output. A criticism to such methods is that they may generate out-of -istrririittoonn counterfactuals, leading to unreliable explanations. Chang et al. (2019) address this issue for images using conditional generative models for inpainting regions with realistic counterfactuals.
Evaluating sample based feature importance remains largely unstudied for time series models. While more widely studied for image classification, (Bach et al., 2015; Fong and Vedaldi, 2017) these methods cannot be directly extended to time series models due to complex time-series dynamics. Most efforts in this domain focus population level feature importance (Tyralis and Papacharalampous 2017). Suresh et al. (2017) is one of the few methods addressing sample based feature importance and method similar to Fong and Vedaldi (2017), called "feature occlusion". They replace each time series observation by a sample from uniform noise to evaluate its effect on model outcome to attribute feature importance. We argue that carefully choosing the counterfactual selection policy is necessary for derive reliable importances. Specifically, replacing observations with noisy out-of-domain samples can lead arbitrary changes model output that are reflective of systematic behavior in the domain. Even if an observation is sampled from the domain distribution, it does not characterize temporal dynamics and dependencies well enough, potentially highlighting features that only reflect global model behavior, as opposed to sample specific feature importance. We therefore model the data distribution order to generate reliable counterfactuals. We demonstrate the implications of the choice of the generator (and hence the counterfactuals) on the quality of explanation.
EVALUATION
We evaluate our explainability method for finding important features in time series on 2 simulated datasets and real datasets. Our goal is two-fold a) comparison to existing feature importance baselines in time series and b) evaluating the choice of generators on the quality of counterfactuals and explanations.
We compare existing feature importance baselines described below:
4

---

Under review as a conference paper at ICLR 2020
1. Feature Occlusion (FO) (Suresh et al., 2017): Method introduced in Suresh al. (2017). This method is an ad-hoc approach for generating counterfactuals. When replacing 2i, with a random sample from the uniform distribution, the change in model output defines the importance for 2
2. Augmented feature occlusion (AFO): We augment the method introduced in Suresh et al. (2017) by sampling counterfactuals from the bootstrapped distribution each feature. This avoids generating out-of distribbbitino samples.
3. Sensitivity Analysis (SA): This method evaluates the sensitivity of the output to every observation, by taking the derivative of yr with respect to Ti,t, at every time point.
4. LIME (Ribeiro et al., 2016): One of the most commonly used explainabilty methods that assigns local importance to features. Although LIME does not assign temporal importance, for this baseline, we use LIME at every time point to generate feature importances.
4.1 SIMULATED DATA
(b)
Figure 3: Two samples showing performance on simulated data. Top row are the original sampled signals (one per subfigure). FFC and AFO assign importance at the time of spike, Additional samples are in Appendix A.I.
Evaluating the quality of explanations is challenging due to the lack of a gold standard/groume truth for the explanations. Additionally, explanations are reflective of model behavior, therefore such evaluations are tightly linked to the reliability of the model itself. Therefore we created the simulated environment in order test our method.
In this experiment, we simulate a time series data such that only one feature determines the outcome. Specifically, the outcome (label) changes to as soon as spike observed in the relevant feature. We keep the task fairly simple for two main reasons: 1) to ensure that the black-box classifier can indeed learn the right relation between the important feature and the outcome, which allows us to focus evaluating the quality of the explanations without worrying about the quality of the classifier. 2) to have a gold standard for the explanations since the exact important event predictive of the outcome are known. We expect the explanations assign importance only the relevant feature, at the exact time of spike, even in the presence of spikes in other non-relevant features.
To simulate these data, we generate d 3 (independent) sequences a standard non-linear auto-regressive moving average (NARMA) time series of the form: xtt + 1) = 0.5z(t) + (L)x(() 1-1 atl 1) + 1.5u(L (2 ()) + 0.5 for E [80]. where the order is 2 and (O,mal(O, 0.01). We add linear trends to the features and introduce random spikes over time for every feature. Note that since spikes not correlated over time, no of the generators (used in FFC, AFO, FO) will learn to predict it. The important feature this setting is feature 1. The complete procedure is described in Appendix 1.2.1. We train RNN-based black-box model on this data, resulting in AUC- 0.99 on the test set.
Figure 7 demonstrates explanations of each of the compared approaches simulated data for 2 test samples. As shown Figure 7(a). Sensitivity analysis does not pick up on the importance of the spike. Feature occlusion gives false positive importance spikes that happen in non-important
5

---

Under review as a conference paper at ICLR 2020
signals as well as the important one. Augmented feature occlusion resolves this problem since it samples the counterfactuals from the data distribution, however, it generates noisier results as samples from the bootstrap distribution. The proposed method (FFC) only assigns importance to the first feature at the time of spike. Hence, FFC generates fewer false relevance scores.
Note that all baseline methods provide importance for evry sample every time point. The true explanation should highlight feature 1 at time points of spike. Using this ground truth, we evaluate the AUROC and AUPRC of the generated explanations denoted by (exp). Table 2 summarizes the results for simulated data.
Simulated Data I
Simulated Data II
Log-probabilities
Method
AUROC (exp)
AUPRC (cxp)
AUROC (cxp)
AUPRC (exp)
(counterfactuals)
FFC
0.9995
0.8859
0.9548 (0.0057)
0.2599 (0.0057)
-5106586.33
AFO 0.9901
0.3768
0.724 (0.012)
0.0374 (0.0028)
-5134432.65
FO
0.7557
0.003
0.734 (0.0091)
0.0376 (0.0031)
-5149629.32
SA 0.4329
0.0052
0.7122 (0.011)
0.0428 (0.0013)
N/A
LIME
0.3331
0.0011
0.4214 (0.0803)
0.0181 (0.0008)
N/A
Table 2: Simulated Data 1 & II Explanation performance compared to ground-truth. truth. For Simulated Data II, we also show in the third column that the log- prg-rbbbabliites of our counterfactuals are higher under the true distribution.
4.2 SIMULATED DATA II
The first simulation does not necessarily evaluate feature importance under complex state dynamics as is common in applications. this simulation, we create a dataset with complex dynamics with known ground truth explanations. The dataset consists of multivariate time series signals with 3 features. Hidden Markov Model with 2 latent states, with linear transition matrix and multivariate normal emission probabilities used to simulate observations. The the outcome y is a random variable, which, in state 1, is only affected by feature and in state 2, only affected by feature 2. Also, we add non-stationarity the time series by modeling the state transition probabilities as a function of time.
The ground truth explanation for output at time T is the observation Ii, where is the feature that drives the output the current state and indicates when feature became important. time series setting, a full explanation for outcome at t = T should include the most important feature variable as well as the time point of importance (here state change).
Figure 4 demonstrates assigned importance for time series sample. The shaded regions indicate the top 5 important observations (x:,,) for each method, the color indicating the corresponding feature i. AFO, FO and FFC are able to learn the state dynamics and are able to find the important feature of each state. However, the top importance values AFO and FO do not correspond to the important time points. Only in FFC, the top important observations are indicative of state changes. Table shows the performance compared to mround-truth explanations for this data.
4.2. EFFECT OF GENERATOR SPECIFICATION
As mentioned earlier, the quality of explanations rely on the quality of the counterfactuals. The coun- terfactuals should reflect the underlying latent dynamics for an individual sample. Counterfactuals under the marginal (as used by AFO) need not be likely for specific sample. The conditional distri- bution we use, on the other hand, models the underlying characteristic of an individual sample, while the marginal is an average over the population. Counterfactuals unlikely under an individual patient's dynamics can result in inaccurate importance assignments since they can potentially overestimate the change in model outcome significantly. We demonstrate this by evaluating the log probability of the counterfactual under the true generator distribution p" (xx (x Results are summarized in Table 2, Column 3. Since we simulate data using an HMM, we can analytically derive the distribution p" (xt,e X)==1). Details of the derivation included in Appendix A.2.1.

---

Under review as a conference paper at ICLR 2020
feature
feature
Teature 2
Risk
Time seriee aggnela and Modal's prodicted nat
20
200
FFC
AFO
PO
Figure 4: Simulated Data II: The top plot shows the time series signals and the output risk of the black-box model. States are shaded in green and yellow. Plots below show the reported importance of all features over time for baseline methods. Shaded regions in these plots represent the top 5 important observations reported by each method.
4.3 MIMIC MORTALITY
Explaining models based on feature importance is critical for clinical settings. Most of clinical data come in form of time series therefore, in is important find critical time points in patients' trajectories along with important features. We evaluate our method on a benchmark mortality prediction task based on the Intensive Care Unit (ICU) time series data from MIMIC. The MIMIC-III dataset consists of de-identified EHRs for ~ 40, 000 ICU patients at the Beth Isreal Deaconess Medical Center, Boston, MA. The dataset has time series measurements such as vitals and lab results over patients ICU stay (Johnson et al., 2016). We use an RNN-based mortality predictor model as black-box for evaluation. The model trained on 14, 712 adult ICU admissions and reaches classification AUC of 0. (.399900007)) using 8 vital and 20 lab measurements, as well as patient static data. More details on model and data used are Appendix A.3.
Following the procedure in Algorithm 1, we train conditional generators for non-static time series features. We compare results across all four existing methods by visualizing importance scores over time. Figure 5 shows example trajectory of a patient and the predicted outcome. We plot the importance score over time for top 3 signals, selected by each method. Shaded regions in bottom four rows indicate the most important observations, color representing the feature. As shown in Figure 5, counterfactual based methods mostly agree and pick the same signals as important features. We further evaluate this by looking into accordance scores among methods, indicating the percentage of agreement. This analysis is provided in the Appendix A.3, and the heat map in Figure 10 demonstrates the average score across test samples. However, the methods don't agree on the exact time importance. As we observe in Figure 5 and other patient trajectories, FFC assigns importance to observations at the precise times signal change. This is exactly as expected from the method. The FFC counterfactual is conditioned on patient history and thus the counterfactual represents an observation assuming a patient had not change state.
4.3.1 EVALUATION USING CLINICAL ANNOTATIONS
Since evaluation of explanations can be subjective, we also use intervention information present in patient records to evaluate clinical applicability across baselines. Clinicians intervene with medication procedure when there is a notable, adverse change in patient trajectory. Therefore, picking up the most relevant features before an intervention onset is indicative of clinical validity of the method. While cannot directly associate intervention with specific cause (observation), we look at the overall distribution signals considered important by each of the methods, prior to intervention onset. Figure 6 shows these histograms for number of interventions. We consistent
7

---

Under review as a conference paper at ICLR 2020
BICARBONATE MACNESIUM PHOSPHATE
SODIUM
Risk score
phenylephrine crystalloid, bolus
HeartRate RespRata
vaso
Time seros Lignal and Modal's prodicted risk
PFC
10
Figure 5: The top row shows the output risk of the prediction model (black dashed line), and normalized time series signals. Shaded regions in the top row represent specific clinical interventions, The four bottom rows correspond to importance scores generated by each of the methods. The error bars indicate the standard deviation of the importance value. Hatched regions in bottom four rows indicate the top most important observations.
assignment of importance across all methods. This means they associate the same influential signals to the same medical intervention.
oprepinophrine
Isuprel FFC
crystallold bolus
FFC
FFC
AFD
AFD
BOAI EMAGEES UN JOCEN
TH2S HATTE
SOLIUM
ANON
SCRRONAAT SOUEM HOSNESLN
IN SIVER 5.0
HOMMONN
Sensitiity
LACTA
Sensitivtty
SNON OA7
BLAROORATE
300E CHLOMR MARESUU
2.5
Figure 6: Top four features assigned be important for each intervention compared methods.
4.4 GREENHOUSE GAS (GHG) OBSERVING NETWORK DATA SET
This experiment evaluates the utility of our method for attributing importance GHG tracers across different locations in California. The GHG data consists of 15 time series signals from 2921 grid cells. target time series is a synthetic signal of GHG concentrations, We use RNN model estimate GHG concentrations using all tracers. Evaluating which tracers are most useful in reconstructing this synthetic signal can be posed as a feature importance problem for weather modeling over time.
In order to quantitatively evaluate the proposed method on real data, we evaluate how well the method performs at selecting globally relevant methods as a proxy. We aggregate the importance of all features over time (and training samples) and retrain the black-box by i) removing top 10 relevant features as indicated by each method ii) using top relevant features only. The performance summary is provided in Table suggesting that among methods that derive instance wise feature importance over time, FFC also generates reasonable global relevance features. Results for both MIMIC-III and GHG datasets summarized in Table 3.
8

---

Under review as a conference paper at ICLR 2020
MIMIC
GHG
AUROC -model w/o top 10
AUROC -model
MSE -model w/o top 10
MSE -model w/ top 3 only
Method
w/ top only
(lower is better)
(higher is better)
(higher is better)
(lower better)
FFC
0.7807 (0,0025)
0.7213 (0.0014)
4546.460
4537.188
AFO
0.7861 (0.0055)
0.7208 (0.002)
4544.163
4532.027
FO
0.0.7861 (0.0026)
0.72 (0.002T)
4538.940
4537.044
SA
0.7788 (0.004)
0.6939 (0.0011)
4532.440
4552.386
LIME
0.7798 (0.0031)
0.6795 (0.0004)
4536.328
4551.874
Table Real Data Global Importance.
Data Randomization Test
Model Randomization Test
Method
AUROC
A AUPRC
A AUROC
Á AUPRC
FFC
-0.2888
-0.2129
-0.2351
-0.2202
AFO
-0.2060
-0.0184
-0.1662
-0.0174
FO
-0.2070
-0.0176
-0.1565
-0.0172
SA -0.2252
-0.0258
-0.3501
-0.0253
Table 4: Sanity Check Test Results for Simulated Data II. For all measures higher difference is better.
4.5 SANITY CHECK TEST FOR EXPLANATIONS
We additionally evaluate the quality of the proposed FFC method using the randomization tests proposed as 'Sanity Checks' in Adebayo al. (2018). Two randomization tests are designed to test for sensitivity of the explanations to i) the black-bor model parameters using model parameter randomization test, and ii) sensitivity to data labels using a using a data randomization test. We conduct this evaluation for Simulation Data II.
This test evaluates how different explanations are when the black-box model is trained on permuted labels (breaking the correlation between features and output label). If explanations truly rely on the output labels, as suggested in our definition, then the explanation quality should differ significantly when model trains on permuted labels. We evaluate the drop in the AUROC and AUPRC of the generated explanations compared to the ground truth.
Data Parameter Randomization Test:
2. Model Parameter Randomization Test: This test evaluates how different explanation quality is when the parameters of the model are arbitrarily shuffled. Significant differences in generated explanations suggests the proposed method is sensitive to black-box model parameters. In Adebayo et al. (2018), these tests are conducted for saliency map methods for images by evaluating the distance between saliency maps for different randomizations. The results are included for Simulated Data II, measured with AUROC and AUPRC as ground-truth explanations are available.
The results of both tests included in Table 4. They indicate the drops in explanation performance for both randomization tests. The performance of the model used for model randomization test drops to 0.52 AUROC as opposed to 0.95 for the original trained model on this simulated task (Simulation Data II). For data randomization, performance of the model drops to 0.62 from 0.95 in terms of AUROC. AUROCs and AUPRCs for FFC drop the most, suggesting the FFC explanation method is sensitive to perturbations in output labels (as tested by the data randomization test) and to randomization in model parameters. Significant deterioration compared explanation performance in Table (for Simulation Data II) indicates that the proposed method passes the sanity checks.
DISCUSSION
We propose new definition for obtaining sample-based feature importance for high-dimensional time series data. We evaluate the importance of each feature every time point, to locate highly important observations. We define important observations as those that cause the biggest change in model output had they been different from the actual observation. This counterfactual observation is generated by modeling the conditional distribution the underlying data dynamics. We propose a generative model to sample such counterfactuals. We evaluate and compare the proposed definition

---

Under review as a conference paper at ICLR 2020
and algorithm several existing approaches. We show that method is better at localizing important observations over time. This is one of the first methods that provides individual feature importance time. Future extension to this work will include analysis real datasets annotated with feature importance explanations. The method will also be extended to evaluate change risk based most relevant subsets of observations.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pages 9505-9515, 2018.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e013014 30140, 2015.
Christopher M Bishop. Pattern recognition and machine learning- springer, 2006.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations,
2019.
Edward Choi, Mohammad Taha Bahadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems, pages 3504-3512, 2016.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pages 2980-2988, 2015.
Ruth C. Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful pertur- bation. In 2017 IEEE International Conference Computer Vision (ICCV), pages 2017.
Tian Guo, Tao Lin, and Yao Lu. An interpretable Istm neural network for autoregressive exogenous model, 2018 URL https: /ttppp:://ppeeee http////ppeeeeeeeeee/eeeeeeeeeee .nt wwwimmidddddiiiiii
Youssef Hmamouche, Alain Casali, and Lotfi Lakhal. A causality based feature selection approach for multivariate time series forecasting. 2017.
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 3543-3556, 2019.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible eritical care database. Scientific data, 3:160035, 2016.
Yao Ming, Shaozu Cao, Ruixiang Zhang, Zhen Li, Yuanzhe Chen, Yangqiu Song, and Huamin Qu. Understanding hidden memories of recurrent neural networks. In 2017 IEEE Conference Visual Analytics Science and Technology (VAST), pages 13-24. IEEE, 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144. ACM, 2016.
Shoaib Ahmed Siddiqui, Dominique Mercier, Mohsin Munir, Andreas Dengel, and Sheraz Ahmed. Tsviz: Demystification of deep learning models for time-series analysis. IEEE Access, 2019.
Hendrik Strobelt, Sebastian Gehrmann, Hanspeter Pfister, and Alexander M Rush. Lstmvis: A tool for visual analysis hidden state dynamics in recurrent neural networks. IEEE transactions on visualization and computer graphics, 24(1) 24(1):667-676 2018.
10

---

Under review as a conference paper at ICLR 2020
Harini Suresh, Nathan Humt, Alistair E. W. Johnson, Leo Anthony Celi, Peter Szolovits, and Marzyeh Ghassemi. Clinical intervention prediction and understanding using deep networks. arXiv preprint arXiv:1705.08498, 2017.
Hristos Tyralis and Georgia Papacharalampous. Variable selection in time series forecasting using random forests. Algorithms, (0(4):114, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
Phongtharin Vinayavekhin, Subhajit Chaudhury, Asim Munawar, Don Joven Agravante, Giovanni De Magistris, Daiki Kimura, and Ryuki Tachibana. Focusing on what relevant: Time-series learning and understanding using attention. In 2018 24th International Conference Pattern Recognition (ICPR), pages 2624-2629. IEEE, 2018.
Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2565 2573. ACM, 2018.
Kiyoung Yang, Hyunjin Yoon, and Cyrus Shahabi. Clever: feature subset selection technique for multivariate time series. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 516-522. Springer, 2005.
Yinchong Yang. Volker Tresp, Marius Wunderle, and Peter A Fasching. Explaining therapy pre- dictions with layer-wise relevance propagation in neural networks. 2018 IEEE International Conference on Healthcare Informatics (ICHI), pages 152-162. IEEE, 2018.
Hyunjin Yoon, Kiyoung Yang, and Cyrus Shahabi. Feature subset selection and feature ranking for multivariate time series. IEEE transactions on knowledge and data engineering, 17(9):1186-1199 2005.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 SIMULATED DATA I
To simulate these data, we generate d = 3 (independent) sequences as standard non-linear auto- regressive moving average (NARMA) time series. Note also that we add linear trends to features and the form: (t + 1) = (t5x((t) f) + 0.5x(() Σ x(t 1) + 1.5u(t - (1 - (-))))((t) + 0.5 + agt for & E [80], a (0.065 for feature 2 and 0.003 for feature and where the order
Normal (0, 0.03). We additionally add linear trends to features. We add spikes to each sample
(uniformly at random over time) and for every feature d following the procedure below:
Ud ~ (ernoull((O.)):
7/d Poisson() À
if 1 = otherwise
gd Sample ma): Fd,t Fdd 1L E ga
where x >> indicates the additive spike. The label y/ = 1 > t1, where t1 = min gd, i.c. the label changes to when spike is encountered in the first feature and 0 otherwise.
We sample our time series using the python TimeSynth' package. Number samples generated: 10000 (80%. split).
A.1.1 ADDITIONAL SAMPLES
feature
1oaturo
festure 2
.00
140
(a)
(b)
Figure 7: Two samples showing performance on simulated data I. Top row are the original sampled signals (one per subfigure).
A.2 SIMULATED DATA II
This simulated data is a two state HMM (2 states) with initial state æ [0.5, 0.5). Transition probability T being:
[0. 0.9
T
0.1
0.9
The emission probability in each state is multivariate Gaussian: Wu1, En) and N ((1 En) where #1 [1.2, 1.5, 0.8] and μlg = [-1.2, -0.8, -15). Marginal variance for all features each state is 0.8 with only features and 2 being correlated (E12 = Σz1 0.01) in state 1 and only 0 and 2 om state 2 (Eg2 Σzo 0.01). In state 1, the label y only depends on feature 1 and in state 2, label depends only on feature 2.
hhttps://gi thw.. wwwphhwww....eeeeeeeeeeeeeeeeeee http://htth....mmm/mmmmmmm//////
12

---

Under review as a conference paper at ICLR 2020
The output Yr at every step is assigned using the logit in 3. Depending the hidden state time t, only one of the features contribute to the output and is deemed influential to the output.
Ite FI *1
St =
Pt
8t
(3)
Itc 2, V1 Bernoulli(pe)
A.2.1 DERIVATION OF p" (o=-1)
The true conditional distribution can be derived using the forward algorithm (Bishop, 2006) as follows:
p (oa-1)=
Σ
p(cxulss)((:(xxxt)
(4)
where,
p(s Xo:-1)
Σ
(5)
8t-1E(0.1)
where p(s Xo:t-) 1) is estimated using the forward algorithm.
JOINT CONDITIONAL GENERATIVE MODEL
Our generator 9j trained using an RNN (GRU). We model the latent state z, with a multivariate Gaussian with diagonal covariance and observations with multivariate Gaussian with full covariance. Parameter setting of the generator for each of the experiments are shown in tables below.
Setting RNN cell
value
GRU
Loss Optimizer
MSE
Adam
Table 5: General generator Setting
Software used: Python 3.7.3, Pytorch 1.0.1.post2
GPU Info: Quadro 400
CPU Info: Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz
The counterfactual for observation i at time & can now be sampled by marginalizing other features at time f. i.e, Fi,t _x_,p(x p( xo
A.3 MIMIC-III MORTALITY EXPERIMENT
Feature selection and data processing: For this experiment, we select adult ICU admission data from the MIMIC dataset. We use static patients' static, vital measurements and lab result for the analysis. The task is to predict 48 hour mortality based 48 hours of clinical data, therefor remove samples with less than 48 hours data. Table presents a full list of clinical measurements used in this experiment.
The predictor model takes in new measurements every hour, and updates the mortality risk. We quantize the time series data to hour blocks by averaging existing measurements within each hour block. use 2 approches for imputing missing values: 1) Mean imputatiopn for vital signals using the sklear Simplelmputer ?, forward imputation for lab results, where we keep the value of the last
13

---

Under review as a conference paper at ICLR 2020
lab measurement until value is evaluated. We also removed patients who had all 48 quantized measurements missing for specific feature. Overall, 22,98 22,988 ICU admissions were extracted and training process was on 65%, 15%%,5%%2200 train, validation, test set respectively.
Data class
Name
Static measurements
Age, Gender, Ethnicity, first time admitted to the ICU? LACTATE, MAGNESIUM, PHOSPHATE, PLATELET, POTASSIUM, PTT, INR, PT, SODIUM, BUN, WBC HeartRate, DiasBP, SysBP, MeanBP, RespRate, SpO2, Glucose, Temp
Lab measurements
Vital measurements
Table 6: List of clinical features for the risk predictor model
Parameter Settings for mortality risk predictor model: The risk predictor model is a recurrent network with GRU cells. All features are scaled mean, unit variance and the target is probability score ranging [0, 1]. The model achieves 0. 0.7939(0.007) AUC on test set classification task. Detailed specification of the model are presented in Table 7.
Setting
value
epochs
80
Model
GRU
batch size
100
Encoding size (m)
150
Loss
MSE
Regressor Activation
Sigmoid True
Batch Normalization
Dropout
True
Gradient Algorithm
Adam (learning rate 0.001, 3,
0.9, B2 0.999, weight decay 0)
Table 7: Mortality risk predictor model features
Generator Loss
0.6
Tran
Eéstion InC
Thin nes Featue Vilor on Ioce feature?
1.0
m n
0.5
70
10
50
75
85
50
75
100
125
150
175
200
(a) Risk predictor MSE loss
(b) Generator Loss
Parameter Settings for conditional Generator: The recurrent network with specifications show in 8 learns a hidden latent vector h, representing the history. h, is then concatenated with æ- and fed into a non-linear 1-layer MLP to model the conditional distribution p(a;, 1) ((xtt))..
Additional importance plots provided in Figure 9.
14

---

Under review as a conference paper at ICLR 2020
Setting
value
epochs
150
RNN cell
GRU
batch normalization
True
batch size
100
RNN encoding size (m)
80
Regressor encoding size (m)
300
Loss
MSE
Gradient Algorithm
Adam (learning rate 0.0001, 81 = 0.9, 32 = 0.999, weight decay = 0)
Table 8: Training Settings for Feature Generators for MIMIC-III Data
BICARBONATE
SODIUM
Risk score
pherylephrine colleid bolus
MAGNESIUM PHOSPHATE
RespRate
crystalloid_bed bolus
18
16
85
16
70
BICARBONATE GLUCOSE
MAGNESIUM
WBC
Risk score
FF
Figure 9: Additional patient trajectories and feature importance assignment with all baseline methods
Accordance testing: For this test we look into how much different baselines agree on important feature assignment. As we observed from the experiments, counterfactual methods mostly agree on the most important features for individual samples. We define accordance as the percentage top signals both identified as important. A score of 80 means on average over the test data, 80 of the assignments similar. This depicted in Figure. 10.
between 2 methods
15

---

Under review as a conference paper at ICLR 2020
FFC
AFO
FO
1.00
FFC
0.00
88
AFO
5.50
9.75
2.70
FO
Figure 10: Heat map showing the accordance between pairs of methods for the most important 6 signals out of 31 clinical features
A.4 RUN-TIME ANALYSIS
In this section we compare the run-time across multiple baselines. Table 9 shows inference runtime for all the baseline methods on machine with Quadro 400 GPU and Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz CPU. The runtime for the counterfactual approaches (FFC, FO and AFO) depends only on the length of the time series. It is also the case for FFC since the conditional generator models the joint distribution of all features. This an advantage, over approaches like LIME, the runtime depends both on the length of the signal as well as the number of features.
Overall, FFC performs reasonably compared to ad-hoc counterfactual approaches, since inference on the RNN conditional generator is efficient. This is one of the reasons that the RNN generator model used to approximate the conditional distribution.
Method
Simulation data
MIMIC data
100,
48, 27
FFC AFO FO LIME
0.99
0.36
1.64
0.62
2.09
0.84
2.23
8.72
Sensitivity Analysis
0.212
0.055
Table 9: Run-time results for simulated data and MIMIC experiment.
A.5 GHG NETWORK DATA
Parameter Settings for Generator:
The settings are provided in Table 10.
Setting
value
epochs
200
RNN cell
GRU
batch size
100
Encoding size (m)
100
Loss
MSE
Gradient Algorithm
Adam (learning rate 0.0001, 81 = 0.9, Br 0.999, weight decay 0)
Table 10: Training Settings for Feature Generators for GHG Data
Parameter Settings for Black-Box: This black box regresses d 15 tracer time signals to the target synthetic GHG time series for t = 327 time points. This model is trained using a 65 train, validation, test set respectively. All features scaled to mean unit variance and the target
16

---

Under review as a conference paper at ICLR 2020
Gonerator Lous
14
12
50
Figure 11: Left: Generator Loss for ghg data. Right: (Scaled) Regresser MSE loss
is scaled time series is scaled in the range [-1, The regressor is an RNN model with the parameter settings given in Table 11. Figure 11 (a) shows the generator loss for all trained conditional
Setting
value
epochs
200
Model
RNN
batch size
100
Encoding size (m)
100
Loss
MSE
Regressor Activation
Linear
Gradient Algorithm
Adam (learning rate 0.001, 31
0.9, 82 0.999, weight decay 0)
Table 11: Training Settings for Regressor for GHG Data
(counterfactual) generators, while Figure 11 shows the training loss of black-box that was used to present feature important results in Section 4.4.
17


Reviews:
[{"review_id": "ryejqDjqtS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an extension of feature occlusion (FO) [Suresh et al., 2017] in which they sample from a pre-trained generative model for replacing the observed variables. Technically, it is in the category of saliency maps, only with possibly larger perturbations defined by the generative model. Thus, it most likely should inherit the same properties as the saliency maps.\n\nThe authors experiment on both synthetic and real-world datasets. They use log-probabilities of the generated samples as a metric for the quality of the counterfactuals; however, because not all baselines are based on counterfactuals, this approach has limited usefulness. Also, it seems that the log-probabilities are too small, indicating that most likely the authors have reported the sum log-probabilities instead of average probabilities.\n\nGiven the similarity to the saliency maps, the authors should have tested the proposed method in the sanity checks in [1]. Also, the authors should have examined the robustness of the proposed explanations given the adversarial vulnerability phenomenon.\n\nDespite sampling from a generative model, because of the univariate nature of the counterfactuals used in this paper, the process might create invalid data points. For example, increased blood sugar is usually correlated with increased blood pressure. However, this method does not account for the correlation among the features.\n\nFinally, this method should be very slow to run. The authors should have compared the run-time speed of the algorithms in the experiment section, too.\n\n[1] Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In NeurIPS.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeAAjm19r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Overall ---\n\nThis paper proposes a method for evaluating the influence of individual observations on the output of a time series prediction model by replacing each (discrete time) observation with its conditional expectation given the other observations. They evaluate this method qualitatively on synthetic, healthcare, and climate datasets. I reviewed this paper for NeurIPS and was happy to see that the authors have made substantial improvements to the presentation and evaluation of the method. With that said, I think that the methodological contribution is incremental (sampling from a conditional rather than marginal distribution), there is at least one major correctness issue that needs to be addressed, and the analysis of the experiments fails explain why the models perform differently.\n\n--- Major comments ---\n\n1. The Montecarlo approximation in Algorithm 1 does not approximate Imp(i,t). Specifically, because the averaging is done before the absolute value, Algorithm 1 approximates |F(X_{0:t}) - E[F(X_{0:-t},x_{-i,t},\\hat{x}_{i,t}]|. This is also a valid measure of feature importance and it is not clear from the paper why we should prefer one over the other.\n\n2. I think the paper needs to do a better job explaining why sampling from the conditional leads to better explanations than sampling from the marginal. The second paragraph makes an argument based on variance, but it is not clear that low variance translates to better explanations. In particular, using mean imputation has very low variance, but I would expect it to give poor explanations. I recommend using a toy example to make this point. For example, in a healthcare context, doctors are reacting to changes relative to a particular patient's baseline. A conditional model can capture this baseline but a marginal model cannot.\n\n3. In general, I thought that the experiments were well done, but stops short of explaining *why* the methods perform differently. Put differently, I think it is really important to clearly explain why certain methods fail while others succeed. For example, the authors demonstrate the sensitivity analysis fails on the synthetic data, but never explain why. I am looking for a statement of the form: \"Sensitivity analysis fails on this data because... FFC solves this weakness by doing... which is reflected in the experimental results.\"\n\n4. In 4.2.1, it is very unsurprising to me that a model that samples from an approximation of the conditional has higher likelihood under the conditional than samples from the marginal, but why should we expect this to lead to better explanations?\n\n5. I thought the idea of looking at feature importance just before clinician intervention was a very clever evaluation, but I wanted the qualitative evaluation to go one step further. That is, does bicarbonate being the most important feature just before administration norepinephrine and fluids make clinical sense? Is this picking up on a specific condition and if so what condition? A clinician could tell you what they are typically reacting to when they administer fluids or vasopressors and you can compare what they say to what the model says. I was surprised to see the top features all being lab measurements as opposed to vital signs. In particular, in a vacuum, I would expect systolic blood pressure to be the most important feature in both of these cases. Is it possible that the frequency of measurements affects which features are selected as important?\n\n6. I thought GHG experiment was *much* better and clearer in this version of the paper. Well done.\n\n7. I recommend moving the notation from the appendix to the main paper. I don't think a reader should have to reference another document to follow notation.\n\n--- Minor comments ---\n\n1. Pg. 3 \"The magnitude of our...\": I call the authors' definition of feature importance absolute not relative. I would expect a \"relative importance\" to be a ratio of some sort (e.g. relative risk). \n\n2. Pg. 5: Figure 8 --> Figure 3", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BklV1x0J5H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method computing the importance of features in time series, called Feed Forward Counterfactual (FFC).\nIn previous work, the explainability problem in time series was tackled with feature occlusion (FO), sensitivity analyses (SA) methods. However, previous counterfactual based methods do not carefully consider appropriate conditional distribution and generate out-of-distribution counterfactuals.\nThe proposed FFC method addresses this issue by leveraging a generative model which learns the underlying dynamics and generates a realistic counterfactual given the past observations. FFC is evaluated on simulated and real datasets and shows that it is better at localizing important observations over time compared to the other baselines.\nIn summary, this paper introduces a way of defining the feature importance at every time point. The main idea of this paper follows in line with [Chang et al. 2019] which address the problem of out-of-distribution counterfactuals. Although the experiment shows successes of the proposed method on several datasets, the major weakness of this paper is the lack of technical novelty and detailed analysis of the proposed method. For example,\nIf the time series is non-stationary, this could incur a different amount of the change in the model output and proposed time importance might not work. How about this?\nDid the authors consider trying out with varying size of training data or generator model?\nMinor\nOn page 2, p(\\mathbf{x_{t,i}|\\mathbf{X_{0:{t-1}}}}) -> p(x_{i,t}|\\mathbf{X_{0:{t-1}}}})\nOn page 6, affected by feature and -> affected by feature 1 and\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a definition of and an algorithm for computing the importance                                                   
of features in time series classification / regression.                                                                            
The importance is defined as a finite difference version of standard sensitivity                                                   
analysis, where the distribution over finite perturbations is given by a                                                           
learned time series model.                                                                                                         
The approach is tested on simulated and real-world data sets.                                                                      
                                                                                                                                   
The reviewers note a lack of novelty in the paper and deem the contribution                                                        
somewhat incremental, although exposition and experiments have improved compared                                                   
to previous versions of the manuscript.                                                                                            
                                                                                                                                   
I recommend to reject this paper in its current form, taking into account on the reviews and my own                                
reading, mostly due t a lack of novelty.                                                                                           
Furthermore, the authors call their method a "counterfactual" approach.                                                            
I don't agree with this terminology.                                                                                               
No attempt is made to justify is by linking it to the relevant causal literature                                                   
on counterfactuals.                                                                                                                
The authors do indeed motivate their algorithm by considering how the classifier                                                   
output would change "had an observation been different" (a counterfactual), but                                                    
mathematical in their model this the same as asking "what changes if the observation is                                            
different" (interventional query). 
Decision: Reject