Title: EXPLAINING TIME SERIES BY COUNTERFACTUALS

Abstract: We propose a method to automatically compute the importance of features at every observation in time series, by simulating counterfactual trajectories given previous observations. We define the importance of each observation as the change in the model output caused by replacing the observation with a generated one. Our method can be applied to arbitrarily complex time series models. We compare the generated feature importance to existing methods like sensitivity analyses, feature occlusion, and other explanation baselines to show that our approach generates more precise explanations and is less sensitive to noise in the input signals.

1. INTRODUCTION:

Multi-variate time series data are ubiquitous in application domains such as healthcare, finance, and others. In such high stakes applications, explaining the model outcome is crucial to build trust among end-users. Finding the features that drive the output of time series models is a challenging task due to complex non-linear temporal dependencies and cross-correlations in the data. The explainability problem is significantly exacerbated when more complex models are used. Most of the current work in time series settings focus on evaluating globally relevant features (Yang et al., 2005; Yoon et al., 2005; Hmamouche et al., 2017) . However, often global feature importance represents relevant features for the entire population, that may not characterize local explanations for individual samples. Therefore we focus our work on individualized feature importance in time series settings. In addition, besides identifying relevant features, we also identify the relevant time instances for those specific features, i.e., we identify the most relevant observations. To the best of our knowledge this is the first sample-specific feature importance explanation benchmark at observation level for time series models.

In this work, we propose a counterfactual based method to learn the importance of every observation in a multivariate time series model. We assign importance by evaluating the expected change in model prediction had an observation been different. We generate plausible counterfactual observations based on signal history, to asses temporal changes in the underlying dynamics. The choice of the counterfactual distribution affects the quality of the explanation. By generating counterfactuals based on signal history, we ensure samples are realistic under individual dynamics, giving explanations that are more reliable compared to other ad-hoc counterfactual methods. In this section we describe our method, Feed Forward Counterfactual (FFC), for generating explanation for time series models. A feature is considered important if it affects the model output the most. In time series, the dynamics of the features also change over time, which may impact model outcome. As such it is critical to also identify the precise time points of such changes.

In order to find important observations for highdimensional time series models, we propose to use a counterfactual based method. Specifically, (a) Graphical model of the conditional generator. ht represents the hidden state of an RNN cell at time t, and zt is the latent representation of the history 

Predictor Network:

(b) FFC procedure: Counterfactual Xt is generated using signal history.

We look into the difference of the original output yt and ŷ(i,t) where observation x (i,t) is replaced with a counterfactual.

Figure 2 : Components describing the proposed method.

we assign importance to an observation x i,t (feature i at time t) based on its effect on the model output at time T (> t). We replace observation x i,t with a counterfactual xi,t , to evaluate this effect. Figure 1 demonstrates how importance of a feature is characterized by replacing an observation with a counterfactual.

2.1. NOTATION:

Multi-variate time series data is available in the form of That is, the importance of an observation for feature i at time t is defined as the change in model output when the observation is replaced by a generated counterfactual. The counterfactual observation can come from any distribution, however the quality of the counterfactual random variable directly affects the quality of the explanation. We generate the counterfactual sample conditioned on signal history up to time t by sampling from the distribution p(x i,t |X 0:t-1 ). Using a conditional generator guarantees that our counterfactuals are sampled not only within domain but also specifically likely under the individual sample X (n) , as opposed to having a generator that models data across population.

X (n) ∈ R d×T (

Conditioning on the history also allows us to learn the dynamics of the signal and therefore generate a plausible counterfactual given the past observations. p(x t |X 0:t-1 ) represents the distribution at time t, if there were no change in the dynamics of the signals. The counterfactual xi,t is sampled from the marginal distribution p(bx i,t |X 0:t-1 ), obtained from p(x t |X 0:t-1 ). Let F(X 0:t-1 , x -i,t , xi,t ) be the output of the model at time T , when x i,t is replaced by the generated counterfactual xi,t . We estimate feature importance Imp(i, t) as E p(xi,t|X0:t-1) [|F(X 0:t ) -F(X 0:t-1 , x -i,t , xi,t )|], summarized in figure 2(b).

2.3. PROPERTIES:

Our proposed method has the following compelling properties in explaining the estimator F:

Time Importance (TI) For every time series, highlighting relevant time events for different features is important for actionability of the explanations. For instance in a clinical setting, just knowing a feature like heart rate is relevant, is not sufficient to intervene -it is also important to know when a deterioration had happened. With FFC, the most eventful time instances can be obtained as:

arg max t∈[T ] {Imp(i, t) ∀i ∈ [d]}

(1) We can thus rank time instances in order of importance. That is, time

t 1 t 2 , if max i∈[d] {Imp(i, t 1 )} ≥ max i∈[d] {Imp(i, t 2 )}.

Feature Importance (FI) At any time instance t, our method assigns importance to every feature of x i,t .

The magnitude of our importance function reflects relative importance. Comparing the importance values across features gives the flexibility to report a subset of important features at each time point t and also reflects the correlation between various features of the time series.

2.4. GENERATOR MODEL:

FOR CONDITIONAL DISTRIBUTION p(x t |X 0:t-1 )

We approximate the conditional distribution of p(x t |X 0:t-1 ) using a recurrent latent variable generator model G, introduced in Chung et al. (2015) . The architecture we use is provided in Figure 2(a) . The conditional generator G models p(x t |z t-1 ) where z t-1 ∈ R k is the latent representation of history of the time series up to time t. The latent representation is a continuous random variable, modeling the distribution parameters. We only use past information in the time series to reflect temporal dependencies. Using the recurrent structure allows us to model a non-stationary generative model that can also handle varying length of observations. Implementation details of the generator are in the Appendix.

Our counterfactuals are not derived by looking at future values which could be done for reliable imputation. Counterfactuals should represent the past dynamics. Note that our derived feature importance is limited by the quality of imputation that may have been utilized by the black-box risk predictor. For experimental evaluation on the effect of generator specifications on counterfactuals and the quality of explanations, see Section 4.1.

2.5. FEATURE IMPORTANCE ASSIGNMENT ALGORITHM:

The proposed procedure is summarized in Algorithm 1. We assume that we are given a trained block box model F, and the data (without labels) it had been trained on. Using the training data, we first train the generator that generates the conditional distribution, (denoted by G). In our implementation we model x as a multivariate Gaussian with full covariance to model all existing correlation between features. The counterfactual xi,t is then sampled from G and passed to the black-box model to evaluate the effect on the black-box outcome.

3. RELATED WORK:

A common method of explaining model performance, in time-series deep learning, is via visualization of activations of latent layers (Strobelt et al., 2018; Siddiqui et al., 2019; Ming et al., 2017) or via Algorithm 1 FFC Input: F: Trained Black-box predictor model, X 0:T , where T is the max time and S: Number of Monte-Carlo samples

1: Train G 2: for all t ∈ [T ] and i ∈ [d] do 3: y T = F(X 0:t ) 4: p(x t |X 0:t ) ∼ G(X 0:t-1 ) 5: for all s ∈ [S] do 6: Sample x(s) i,t ∼ p(x i,t |X 0:t-1 ) 7: ŷ(s) T = F(X 0:t , x -i,t , x(s) i,t ) 8: imp (S) T = |y T - ŷ(s) T | 9: Importance_M atrix(i, t) = Σ S s=0 imp (S) T S

10: Return Importance_M atrix sensitivity analysis (Bach et al., 2015; Yang et al., 2018) . Understanding latent representations, sensitivity and its relationship to overall model behavior is useful for model debugging. However, these but are too refined to be useful to the end users like clinicians.

Attention models (Vaswani et al., 2017; Vinayavekhin et al., 2018; Xu et al., 2018) are the most commonly known explanation mechanisms for sequential data. However, because of the complex mappings to latent space in recurrent models, attention weights cannot be directly attributed to individual observations of the time series (Guo et al., 2018) . To resolve this issue to some extent, Choi et al. (2016) propose an attention model for mortality prediction of ICU patients based on clinical visits. However attention weights may not be consistent as explanations (Jain and Wallace, 2019) .

In vision, prior works tackle explainability from the counterfactual perspective, finding regions of the image that affect model prediction the most. Fong and Vedaldi (2017) assumes higher importance for inputs that when replaced by an uninformative reference value, maximally change the classifier output. A criticism to such methods is that they may generate out-of-distribution counterfactuals, leading to unreliable explanations. Chang et al. (2019) address this issue for images using conditional generative models for inpainting regions with realistic counterfactuals.

Evaluating sample based feature importance remains largely unstudied for time series models. While more widely studied for image classification, (Bach et al., 2015; Fong and Vedaldi, 2017 ) these methods cannot be directly extended to time series models due to complex time-series dynamics. Most efforts in this domain focus on population level feature importance (Tyralis and Papacharalampous, 2017) . Suresh et al. (2017) is one of the few methods addressing sample based feature importance and use a method similar to Fong and Vedaldi (2017), called "feature occlusion". They replace each time series observation by a sample from uniform noise to evaluate its effect on model outcome to attribute feature importance. We argue that carefully choosing the counterfactual selection policy is necessary for derive reliable importances. Specifically, replacing observations with noisy out-of-domain samples can lead to arbitrary changes to model output that are not reflective of systematic behavior in the domain. Even if an observation is sampled from the domain distribution, it does not characterize temporal dynamics and dependencies well enough, potentially highlighting features that only reflect global model behavior, as opposed to sample specific feature importance. We therefore model the data-distribution in order to generate reliable counterfactuals. We demonstrate the implications of the choice of the generator (and hence the counterfactuals) on the quality of explanation.

4. EVALUATION:

We evaluate our explainability method for finding important features in time series on 2 simulated datasets and 2 real datasets. Our goal is two-fold a) comparison to existing feature importance baselines in time series and b) evaluating the choice of generators on the quality of counterfactuals and explanations.

We compare to existing feature importance baselines described below:

1. Feature Occlusion (FO) (Suresh et al., 2017) : Method introduced in Suresh et al. (2017) . This method is an ad-hoc approach for generating counterfactuals. When replacing x i,t with a random sample from the uniform distribution, the change in model output defines the importance for x i,t .

2.. Augmented feature occlusion (AFO)::

We augment the method introduced in Suresh et al. ( 2017) by sampling counterfactuals from the bootstrapped distribution over each feature. This avoids generating out-of-distribution samples.

3. Sensitivity Analysis (SA): This method evaluates the sensitivity of the output to every observation, by taking the derivative of y t with respect to x i,t , at every time point. Evaluating the quality of explanations is challenging due to the lack of a gold standard/ground truth for the explanations. Additionally, explanations are reflective of model behavior, therefore such evaluations are tightly linked to the reliability of the model itself. Therefore we created the simulated environment in order to test our method.

In this experiment, we simulate a time series data such that only one feature determines the outcome. Specifically, the outcome (label) changes to 1 as soon as a spike is observed in the relevant feature.

We keep the task fairly simple for two main reasons: 1) to ensure that the black-box classifier can indeed learn the right relation between the important feature and the outcome, which allows us to focus on evaluating the quality of the explanations without worrying about the quality of the classifier.

2) to have a gold standard for the explanations since the exact important event predictive of the outcome are known. We expect the explanations to assign importance only to the one relevant feature, at the exact time of spike, even in the presence of spikes in other non-relevant features.

To simulate these data, we generate d = 3 (independent) sequences as a standard non-linear auto-regressive moving average (NARMA) time series of the form:

x(t + 1) = 0.5x(t) + 0.5x(t) l-1 i=0 x(t -l) + 1.5u(t -(l -1))u(t) + 0.5 for t ∈ [80],

where the order is 2 and u ∼ Normal(0, 0.01). We add linear trends to the features and introduce random spikes over time for every feature. Note that since spikes are not correlated over time, no of the generators (used in FFC, AFO, FO) will learn to predict it. The important feature in this setting is feature 1. The complete procedure is described in Appendix A.2.1. We train an RNN-based black-box model on this data, resulting in AUC= 0.99 on the test set.

Figure 7 demonstrates explanations of each of the compared approaches on simulated data for 2 test samples. As shown in Figure 7 (a), Sensitivity analysis does not pick up on the importance of the spike. Feature occlusion gives false positive importance to spikes that happen in non-important signals as well as the important one. Augmented feature occlusion resolves this problem since it samples the counterfactuals from the data distribution, however, it generates noisier results as it samples from the bootstrap distribution. The proposed method (FFC) only assigns importance to the first feature at the time of spike. Hence, FFC generates fewer false relevance scores.

Note that all baseline methods provide an importance for evry sample at every time point. The true explanation should highlight feature 1 at time points of spike. Using this ground truth, we evaluate the AUROC and AUPRC of the generated explanations denoted by (exp). Table 2 

4.2. SIMULATED DATA II:

The first simulation does not necessarily evaluate feature importance under complex state dynamics as is common in applications. In this simulation, we create a dataset with complex dynamics with known ground truth explanations. The dataset consists of multivariate time series signals with 3 features. A Hidden Markov Model with 2 latent states, with linear transition matrix and multivariate normal emission probabilities is used to simulate observations. The the outcome y is a random variable, which, in state 1, is only affected by feature 1 and in state 2, only affected by feature 2. Also, we add non-stationarity to the time series by modeling the state transition probabilities as a function of time.

The ground truth explanation for output at time T is the observation x i,t where i is the feature that drives the output in the current state and t indicates when feature i became important. In a time series setting, a full explanation for outcome at t = T should include the most important feature variable as well as the time point of importance (here state change).

Figure 4 demonstrates assigned importance for a time series sample. The shaded regions indicate the top 5 important observations (x i,t ) for each method, the color indicating the corresponding feature i. AFO, FO and FFC are able to learn the state dynamics and are able to find the important feature of each state. However, the top importance values in AFO and FO do not correspond to the important time points. Only in FFC, the top important observations are indicative of state changes. Table 2 shows the performance compared to ground-truth explanations for this data.

4.2.1. EFFECT OF GENERATOR SPECIFICATION:

As mentioned earlier, the quality of explanations rely on the quality of the counterfactuals. The counterfactuals should reflect the underlying latent dynamics for an individual sample. Counterfactuals under the marginal (as used by AFO) need not be likely for a specific sample. The conditional distribution we use, on the other hand, models the underlying characteristic of an individual sample, while the marginal is an average over the population. 

4.3. MIMIC MORTALITY:

Explaining models based on feature importance is critical for clinical settings. Most of clinical data come in form of time series therefore, in is important to find critical time points in patients' trajectories along with important features. We evaluate our method on a benchmark mortality prediction task based on the Intensive Care Unit (ICU) time series data from MIMIC. The MIMIC-III dataset consists of de-identified EHRs for ∼ 40, 000 ICU patients at the Beth Isreal Deaconess Medical Center, Boston, MA. The dataset has time series measurements such as vitals and lab results over patients ICU stay (Johnson et al., 2016) . We use an RNN-based mortality predictor model as a black-box for evaluation. The model is trained on 14, 712 adult ICU admissions and reaches a classification AUC of 0.7939(0.007), using 8 vital and 20 lab measurements, as well as patient static data. More details on the model and data used are in Appendix A.3.

Following the procedure in Algorithm 1, we train a conditional generators for non-static time series features. We compare results across all four existing methods by visualizing importance scores over time. Figure 5 shows an example trajectory of a patient and the predicted outcome. We plot the importance score over time for top 3 signals, selected by each method. Shaded regions in bottom four rows indicate the most important observations, color representing the feature. As shown in Figure 5 , counterfactual based methods mostly agree and pick the same signals as important features. We further evaluate this by looking into accordance scores among methods, indicating the percentage of agreement. This analysis is provided in the Appendix A.3, and the heat map in Figure 10 demonstrates the average score across test samples. However, the methods don't agree on the exact time importance. As we observe in Figure 5 and other patient trajectories, FFC assigns importance to observations at the precise times of signal change. This is exactly as expected from the method.

The FFC counterfactual is conditioned on patient history and thus the counterfactual represents an observation assuming a patient had not change state.

4.3.1. EVALUATION USING CLINICAL ANNOTATIONS:

Since evaluation of explanations can be subjective, we also use intervention information present in patient records to evaluate clinical applicability across baselines. Clinicians intervene with a medication or procedure when there is a notable, adverse change in patient trajectory. Therefore, picking up the most relevant features before an intervention onset is indicative of clinical validity of the method. While we cannot directly associate an intervention with a specific cause (observation), we look at the overall distribution of signals considered important by each of the methods, prior to intervention onset. Figure 6 shows these histograms for a number of interventions. We see consistent 

4.4. GREENHOUSE GAS (GHG) OBSERVING NETWORK DATA SET:

This experiment evaluates the utility of our method for attributing importance to GHG tracers across different locations in California. The GHG data consists of 15 time series signals from 2921 grid cells. A target time series is a synthetic signal of GHG concentrations. We use an RNN model to estimate GHG concentrations using all tracers. Evaluating which tracers are most useful in reconstructing this synthetic signal can be posed as a feature importance problem for weather modeling over time.

In order to quantitatively evaluate the proposed method on real data, we evaluate how well the method performs at selecting globally relevant methods as a proxy. We aggregate the importance of all features over time (and training samples) and retrain the black-box by i) removing top 10 relevant features as indicated by each method ii) using top 3 relevant features only . The performance summary is provided in Table 3 suggesting that among methods that derive instance wise feature importance over time, FFC also generates reasonable global relevance of features. Results for both MIMIC-III and GHG datasets are summarized in Table 3 

4.5. SANITY CHECK TEST FOR EXPLANATIONS:

We additionally evaluate the quality of the proposed FFC method using the randomization tests proposed as 'Sanity Checks' in Adebayo et al. (2018) . Two randomization tests are designed to test for sensitivity of the explanations to i) the black-box model parameters using a model parameter randomization test, and ii) sensitivity to data labels using a using a data randomization test. We conduct this evaluation for Simulation Data II.

1. Data Parameter Randomization Test: This test evaluates how different explanations are when the black-box model is trained on permuted labels (breaking the correlation between features and output label). If explanations truly rely on the output labels, as suggested in our definition, then the explanation quality should differ significantly when a model trains on permuted labels. We evaluate the drop in the AUROC and AUPRC of the generated explanations compared to the ground truth.

2. Model Parameter Randomization Test: This test evaluates how different explanation quality is when the parameters of the model are arbitrarily shuffled. Significant differences in generated explanations suggests the proposed method is sensitive to black-box model parameters. In Adebayo et al. (2018) , these tests are conducted for saliency map methods for images by evaluating the distance between saliency maps for different randomizations. The results are included for Simulated Data II, measured with AUROC and AUPRC as ground-truth explanations are available.

The results of both tests are included in Table 4 . They indicate the drops in explanation performance for both randomization tests. The performance of the model used for model randomization test drops to 0.52 AUROC as opposed to 0.95 for the original trained model on this simulated task (Simulation Data II). For data randomization, performance of the model drops to 0.62 from 0.95 in terms of AUROC. AUROCs and AUPRCs for FFC drop the most, suggesting the FFC explanation method is sensitive to perturbations in output labels (as tested by the data randomization test) and to randomization in model parameters. Significant deterioration compared to explanation performance in Table 2 (for Simulation Data II) indicates that the proposed method passes the sanity checks.

5. DISCUSSION:

We propose a new definition for obtaining sample-based feature importance for high-dimensional time series data. We evaluate the importance of each feature at every time point, to locate highly important observations. We define important observations as those that cause the biggest change in model output had they been different from the actual observation. This counterfactual observation is generated by modeling the conditional distribution of the underlying data dynamics. We propose a generative model to sample such counterfactuals. We evaluate and compare the proposed definition and algorithm to several existing approaches. We show that our method is better at localizing important observations over time. This is one of the first methods that provides individual feature importance over time. Future extension to this work will include analysis on real datasets annotated with feature importance explanations. The method will also be extended to evaluate change in risk based on most relevant subsets of observations.

The output y t at every step is assigned using the logit in 3. Depending on the hidden state at time t, only one of the features contribute to the output and is deemed influential to the output.

p t = 1 1+e -x 1,t s t = 0 1 1+e -x 2,t s t = 1 y t ∼ Bernoulli(p t ) (3) A.2.1 DERIVATION OF p * (x i,t |X 0:t-1 )

The true conditional distribution can be derived using the forward algorithm (Bishop, 2006) as follows:

EQUATION

where,

EQUATION

where p(s t-1 |X 0:t-1 ) is estimated using the forward algorithm.

JOINT CONDITIONAL GENERATIVE MODEL:

Our generator G i is trained using an RNN (GRU). We model the latent state z t with a multivariate Gaussian with diagonal covariance and observations with a multivariate Gaussian with full covariance. Parameter setting of the generator for each of the experiments are shown in tables below. The counterfactual for observation i at time t can now be sampled by marginalizing over other features at time t. i.e, x i,t ∼ x-i p(x|X 0:t-1 ).

A.3 MIMIC-III MORTALITY EXPERIMENT:

Feature selection and data processing: For this experiment, we select adult ICU admission data from the MIMIC dataset. We use static patients' static, vital measurements and lab result for the analysis. The task is to predict 48 hour mortality based on 48 hours of clinical data, therefor we remove samples with less than 48 hours of data. Adam (learning rate = 0.0001, β 1 = 0.9, β 2 = 0.999, weight decay = 0) Accordance testing: For this test we look into how much different baselines agree on important feature assignment. As we observed from the experiments, counterfactual methods mostly agree on the most important features for individual samples. We define accordance score between 2 methods as the percentage of top n signals both identified as important. A score of 80 means on average over the test data, 80 of the assignments were similar. This is depicted in Figure 10 .

:

https://github.com/TimeSynth/TimeSynth

https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html

Figures and Tables:

Figure 1: Illustration of Proposed Method

Figure 3: Two samples showing performance on simulated data. Top row are the original sampled signals (one per subfigure). FFC and AFO assign importance at the time of spike. Additional samples are in Appendix A.1.

Figure 5: The top row shows the output risk of the prediction model (black dashed line), and normalized time series signals. Shaded regions in the top row represent specific clinical interventions. The four bottom rows correspond to importance scores generated by each of the methods. The error bars indicate the standard deviation of the importance value. Hatched regions in bottom four rows indicate the top most important observations.

General generator Setting Software used: Python 3.7.3 , Pytorch 1.0.1.post2 GPU Info: Quadro 400 CPU Info: Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz

Figure 9: Additional patient trajectories and feature importance assignment with all baseline methods

Generative Model and EstimatorGi : R d-1×m → R Conditional Generative Model sampling for feature i zt ∈ R mLatent encoding of history up to t Notation used in the paper.

Table content: <table/>

summarizes the results for simulated data. Simulated Data I & II -Explanation performance compared to ground-truth. For Simulated Data II, we also show in the third column that the log-probabilities of our counterfactuals are higher under the true distribution.

Table content: <table><tr><td/><td>Simulated Data I</td><td/><td>Simulated Data II</td><td/></tr><tr><td colspan="4">Method AUROC (exp) AUPRC (exp) AUROC (exp)</td><td>AUPRC (exp)</td><td>Log-probabilities (counterfactuals)</td></tr><tr><td>FFC</td><td>0.9995</td><td>0.8859</td><td colspan="3">0.9548 (0.0057) 0.2599 (0.0057) -5106586.33</td></tr><tr><td>AFO</td><td>0.9901</td><td>0.3768</td><td>0.724 (0.012)</td><td colspan="2">0.0374 (0.0028) -5134432.65</td></tr><tr><td>FO</td><td>0.7557</td><td>0.003</td><td>0.734 (0.0091)</td><td colspan="2">0.0376 (0.0031) -5149629.32</td></tr><tr><td>SA</td><td>0.4329</td><td>0.0052</td><td>0.7122 (0.011)</td><td colspan="2">0.0428 (0.0013) N/A</td></tr><tr><td>LIME</td><td>0.3331</td><td>0.0011</td><td colspan="3">0.4214 (0.0803) 0.0181 (0.0008) N/A</td></tr></table>

Simulated Data II: The top plot shows the time series signals and the output risk of the black-box model. States are shaded in green and yellow. Plots below show the reported importance of all features over time for baseline methods. Shaded regions in these plots represent the top 5 important observations reported by each method.

Table content: <table><tr><td>feature 0</td><td>feature 1</td><td>feature 2</td><td>Risk score</td></tr><tr><td>Figure 4:</td><td/><td/><td/></tr></table>

. Real Data -Global Importance.

Table content: <table><tr><td/><td>MIMIC</td><td/><td/><td>GHG</td></tr><tr><td/><td colspan="2">AUROC -model</td><td>AUROC -model</td><td colspan="2">MSE -model</td><td>MSE -model</td></tr><tr><td>Method</td><td colspan="2">w/o top 10</td><td>w/ top 3 only</td><td>w/o top 10</td><td>w/ top 3 only</td></tr><tr><td/><td colspan="2">(lower is better)</td><td>(higher is better)</td><td colspan="2">(higher is better)</td><td>(lower is better)</td></tr><tr><td>FFC</td><td colspan="2">0.7807 (0.0025)</td><td colspan="2">0.7213 (0.0014) 4546.460</td><td>4537.188</td></tr><tr><td>AFO</td><td colspan="2">0.7861 (0.0055)</td><td>0.7208 (0.002)</td><td>4544.163</td><td>4532.027</td></tr><tr><td>FO</td><td colspan="3">0.0.7861 (0.0026) 0.72 (0.0021)</td><td>4538.940</td><td>4537.044</td></tr><tr><td>SA</td><td colspan="2">0.7788 (0.004)</td><td colspan="2">0.6939 (0.0011) 4532.440</td><td>4552.386</td></tr><tr><td>LIME</td><td colspan="2">0.7798 (0.0031)</td><td colspan="2">0.6795 (0.0004) 4536.328</td><td>4551.874</td></tr><tr><td/><td/><td colspan="4">Data Randomization Test Model Randomization Test</td></tr><tr><td/><td colspan="3">Method ∆ AUROC ∆ AUPRC</td><td colspan="2">∆ AUROC ∆ AUPRC</td></tr><tr><td/><td>FFC</td><td>-0.2888</td><td>-0.2129</td><td>-0.2351</td><td>-0.2202</td></tr><tr><td/><td>AFO</td><td>-0.2060</td><td>-0.0184</td><td>-0.1662</td><td>-0.0174</td></tr><tr><td/><td>FO</td><td>-0.2070</td><td>-0.0176</td><td>-0.1565</td><td>-0.0172</td></tr><tr><td/><td>SA</td><td>-0.2252</td><td>-0.0258</td><td>-0.3501</td><td>-0.0253</td></tr></table>

Sanity Check Test Results for Simulated Data II. For all measures higher difference is better.

Table content: <table/>

Table 6 presents a full list of clinical measurements used in this experiment.The predictor model takes in new measurements every hour, and updates the mortality risk. We quantize the time series data to hour blocks by averaging existing measurements within each hour block. We use 2 approches for imputing missing values: 1) Mean imputatiopn for vital signals using the sklearn SimpleImputer 2 , 2) forward imputation for lab results, where we keep the value of the last

Table content: <table><tr><td>Setting</td><td>value</td></tr><tr><td>epochs</td><td>150</td></tr><tr><td>RNN cell</td><td>GRU</td></tr><tr><td>batch normalization</td><td>True</td></tr><tr><td>batch size</td><td>100</td></tr><tr><td>RNN encoding size (m)</td><td>80</td></tr><tr><td>Regressor encoding size (m)</td><td>300</td></tr><tr><td>Loss</td><td>MSE</td></tr><tr><td>Gradient Algorithm</td><td/></tr></table>

Training Settings for Feature Generators for MIMIC-III Data

Table content: <table><tr><td>BICARBONATE LACTATE MAGNESIUM PHOSPHATE</td><td>SODIUM HeartRate RespRate</td><td>Risk score vent vaso</td><td>phenylephrine colloid_bolus crystalloid_bolus</td></tr><tr><td>BICARBONATE GLUCOSE</td><td>MAGNESIUM SODIUM</td><td>WBC HeartRate</td><td>Risk score</td></tr></table>

Peer review (iclr2020):
--- Overall --- This paper proposes a method for evaluating the influence of individual observations on the output of a time series prediction model by replacing each (discrete time) observation with its conditional expectation given the other observations. They evaluate this method qualitatively on synthetic, healthcare, and climate datasets. I reviewed this paper for NeurIPS and was happy to see that the authors have made substantial improvements to the presentation and evaluation of the method. With that said, I think that the methodological contribution is incremental (sampling from a conditional rather than marginal distribution), there is at least one major correctness issue that needs to be addressed, and the analysis of the experiments fails explain why the models perform differently. --- Major comments --- 1. The Montecarlo approximation in Algorithm 1 does not approximate Imp(i,t). Specifically, because the averaging is done before the absolute value, Algorithm 1 approximates |F(X_{0:t}) - E[F(X_{0:-t},x_{-i,t},hat{x}_{i,t}]|. This is also a valid measure of feature importance and it is not clear from the paper why we should prefer one over the other. 2. I think the paper needs to do a better job explaining why sampling from the conditional leads to better explanations than sampling from the marginal. The second paragraph makes an argument based on variance, but it is not clear that low variance translates to better explanations. In particular, using mean imputation has very low variance, but I would expect it to give poor explanations. I recommend using a toy example to make this point. For example, in a healthcare context, doctors are reacting to changes relative to a particular patient*s baseline. A conditional model can capture this baseline but a marginal model cannot. 3. In general, I thought that the experiments were well done, but stops short of explaining *why* the methods perform differently. Put differently, I think it is really important to clearly explain why certain methods fail while others succeed. For example, the authors demonstrate the sensitivity analysis fails on the synthetic data, but never explain why. I am looking for a statement of the form: *Sensitivity analysis fails on this data because... FFC solves this weakness by doing... which is reflected in the experimental results.* 4. In 4.2.1, it is very unsurprising to me that a model that samples from an approximation of the conditional has higher likelihood under the conditional than samples from the marginal, but why should we expect this to lead to better explanations? 5. I thought the idea of looking at feature importance just before clinician intervention was a very clever evaluation, but I wanted the qualitative evaluation to go one step further. That is, does bicarbonate being the most important feature just before administration norepinephrine and fluids make clinical sense? Is this picking up on a specific condition and if so what condition? A clinician could tell you what they are typically reacting to when they administer fluids or vasopressors and you can compare what they say to what the model says. I was surprised to see the top features all being lab measurements as opposed to vital signs. In particular, in a vacuum, I would expect systolic blood pressure to be the most important feature in both of these cases. Is it possible that the frequency of measurements affects which features are selected as important? 6. I thought GHG experiment was *much* better and clearer in this version of the paper. Well done. 7. I recommend moving the notation from the appendix to the main paper. I don*t think a reader should have to reference another document to follow notation. --- Minor comments --- 1. Pg. 3 *The magnitude of our...*: I call the authors* definition of feature importance absolute not relative. I would expect a *relative importance* to be a ratio of some sort (e.g. relative risk). 2. Pg. 5: Figure 8 --> Figure 3