Explaining Time Series by Counterfactuals

1 INTRODUCTION . Multi-variate time series data are ubiquitous in application domains such as healthcare , finance , and others . In such high stakes applications , explaining the model outcome is crucial to build trust among end–users . Finding the features that drive the output of time series models is a challenging task due to complex non-linear temporal dependencies and cross-correlations in the data . The explainability problem is significantly exacerbated when more complex models are used . Most of the current work in time series settings focus on evaluating globally relevant features ( Yang et al. , 2005 ; Yoon et al. , 2005 ; Hmamouche et al. , 2017 ) . However , often global feature importance represents relevant features for the entire population , that may not characterize local explanations for individual samples . Therefore we focus our work on individualized feature importance in time series settings . In addition , besides identifying relevant features , we also identify the relevant time instances for those specific features , i.e. , we identify the most relevant observations . To the best of our knowledge this is the first sample–specific feature importance explanation benchmark at observation level for time series models . In this work , we propose a counterfactual based method to learn the importance of every observation in a multivariate time series model . We assign importance by evaluating the expected change in model prediction had an observation been different . We generate plausible counterfactual observations based on signal history , to asses temporal changes in the underlying dynamics . The choice of the counterfactual distribution affects the quality of the explanation . By generating counterfactuals based on signal history , we ensure samples are realistic under individual dynamics , giving explanations that are more reliable compared to other ad-hoc counterfactual methods . 2 METHOD : FEED-FORWARD COUNTERFACTUALS FOR TIME SERIES EXPLANATION b la c k -b o x o u tc o m e in p u t s ig n a l counterfactual observation 1 actual observation 1 actual outcome counterfactual outcome 1 time time actual observation 2 counterfactual outcome 2 } } feature imp 1 feature imp 2 counterfactual observation 2 we assign importance to an observation xi , t ( feature i at time t ) based on its effect on the model output at time T ( > t ) . We replace observation xi , t with a counterfactual x̂i , t , to evaluate this effect . Figure 1 demonstrates how importance of a feature is characterized by replacing an observation with a counterfactual . 2.1 NOTATION . Multi-variate time series data is available in the form of X ( n ) ∈ Rd×T ( where d is the number of features with T observations over time ) for n ∈ [ N ] samples . We are interested in black-box estimators that receive observations up to time t , xt ∈ Rd , and generate output yt at every time point t ∈ [ T ] . F denotes the target black-box model we wish to explain through the proposed approach called FFC . For exposition , throughout the paper , the index n over samples has been dropped for notational clarity . We index features with subscript i. x−i , t indicates features at time t excluding feature i . The notation used for exposition work is briefly summarized in Table 1 . 2.2 EXPOSITION . We assign importance score to each observation xi , t for t ∈ [ T ] and i ∈ [ d ] , following the definition : Definition 1 . Feature Importance : The importance of the observation i at time t , denoted by Imp ( i , t ) is defined as Ep ( x̂i , t|X0 : t−1 ) [ |F ( X0 : t ) − F ( X0 : t−1 , x−i , t , x̂i , t ) | ] , where | · | denotes the absolute value and x̂i , t is the counterfactual sample . That is , the importance of an observation for feature i at time t is defined as the change in model output when the observation is replaced by a generated counterfactual . The counterfactual observation can come from any distribution , however the quality of the counterfactual random variable directly affects the quality of the explanation . We generate the counterfactual sample conditioned on signal history up to time t by sampling from the distribution p ( xi , t|X0 : t−1 ) . Using a conditional generator guarantees that our counterfactuals are sampled not only within domain but also specifically likely under the individual sample X ( n ) , as opposed to having a generator that models data across population . Conditioning on the history also allows us to learn the dynamics of the signal and therefore generate a plausible counterfactual given the past observations . p ( xt|X0 : t−1 ) represents the distribution at time t , if there were no change in the dynamics of the signals . The counterfactual x̂i , t is sampled from the marginal distribution p ( bxi , t|X0 : t−1 ) , obtained from p ( xt|X0 : t−1 ) . Let F ( X0 : t−1 , x−i , t , x̂i , t ) be the output of the model at time T , when xi , t is replaced by the generated counterfactual x̂i , t . We estimate feature importance Imp ( i , t ) as Ep ( x̂i , t|X0 : t−1 ) [ |F ( X0 : t ) −F ( X0 : t−1 , x−i , t , x̂i , t ) | ] , summarized in figure 2 ( b ) . 2.3 PROPERTIES . Our proposed method has the following compelling properties in explaining the estimator F : Time Importance ( TI ) For every time series , highlighting relevant time events for different features is important for actionability of the explanations . For instance in a clinical setting , just knowing a feature like heart rate is relevant , is not sufficient to intervene - it is also important to know when a deterioration had happened . With FFC , the most eventful time instances can be obtained as : arg maxt∈ [ T ] { Imp ( i , t ) ∀i ∈ [ d ] } ( 1 ) We can thus rank time instances in order of importance . That is , time t1 4 t2 , if maxi∈ [ d ] { Imp ( i , t1 ) } ≥ maxi∈ [ d ] { Imp ( i , t2 ) } . Feature Importance ( FI ) At any time instance t , our method assigns importance to every feature of xi , t . The magnitude of our importance function reflects relative importance . Comparing the importance values across features gives the flexibility to report a subset of important features at each time point t and also reflects the correlation between various features of the time series . 2.4 GENERATOR MODEL FOR CONDITIONAL DISTRIBUTION p ( xt|X0 : t−1 ) We approximate the conditional distribution of p ( xt|X0 : t−1 ) using a recurrent latent variable generator model G , introduced in Chung et al . ( 2015 ) . The architecture we use is provided in Figure 2 ( a ) . The conditional generator G models p ( xt|zt−1 ) where zt−1 ∈ Rk is the latent representation of history of the time series up to time t. The latent representation is a continuous random variable , modeling the distribution parameters . We only use past information in the time series to reflect temporal dependencies . Using the recurrent structure allows us to model a non-stationary generative model that can also handle varying length of observations . Implementation details of the generator are in the Appendix . Our counterfactuals are not derived by looking at future values which could be done for reliable imputation . Counterfactuals should represent the past dynamics . Note that our derived feature importance is limited by the quality of imputation that may have been utilized by the black–box risk predictor . For experimental evaluation on the effect of generator specifications on counterfactuals and the quality of explanations , see Section 4.1 . 2.5 FEATURE IMPORTANCE ASSIGNMENT ALGORITHM . The proposed procedure is summarized in Algorithm 1 . We assume that we are given a trained block box model F , and the data ( without labels ) it had been trained on . Using the training data , we first train the generator that generates the conditional distribution , ( denoted by G ) . In our implementation we model x as a multivariate Gaussian with full covariance to model all existing correlation between features . The counterfactual x̂i , t is then sampled from G and passed to the black-box model to evaluate the effect on the black-box outcome . 3 RELATED WORK . A common method of explaining model performance , in time–series deep learning , is via visualization of activations of latent layers ( Strobelt et al. , 2018 ; Siddiqui et al. , 2019 ; Ming et al. , 2017 ) or via Algorithm 1 FFC Input : F : Trained Black-box predictor model , X0 : T , where T is the max time and S : Number of Monte-Carlo samples 1 : Train G 2 : for all t ∈ [ T ] and i ∈ [ d ] do 3 : yT = F ( X0 : t ) 4 : p ( xt|X0 : t ) ∼ G ( X0 : t−1 ) 5 : for all s ∈ [ S ] do 6 : Sample x̂ ( s ) i , t ∼ p ( xi , t|X0 : t−1 ) 7 : ŷ ( s ) T = F ( X0 : t , x−i , t , x̂ ( s ) i , t ) 8 : imp ( S ) T = |yT − ŷ ( s ) T | 9 : Importance_Matrix ( i , t ) = Σ S s=0imp ( S ) T S 10 : Return Importance_Matrix sensitivity analysis ( Bach et al. , 2015 ; Yang et al. , 2018 ) . Understanding latent representations , sensitivity and its relationship to overall model behavior is useful for model debugging . However , these but are too refined to be useful to the end users like clinicians . Attention models ( Vaswani et al. , 2017 ; Vinayavekhin et al. , 2018 ; Xu et al. , 2018 ) are the most commonly known explanation mechanisms for sequential data . However , because of the complex mappings to latent space in recurrent models , attention weights can not be directly attributed to individual observations of the time series ( Guo et al. , 2018 ) . To resolve this issue to some extent , Choi et al . ( 2016 ) propose an attention model for mortality prediction of ICU patients based on clinical visits . However attention weights may not be consistent as explanations ( Jain and Wallace , 2019 ) . In vision , prior works tackle explainability from the counterfactual perspective , finding regions of the image that affect model prediction the most . Fong and Vedaldi ( 2017 ) assumes higher importance for inputs that when replaced by an uninformative reference value , maximally change the classifier output . A criticism to such methods is that they may generate out-of-distribution counterfactuals , leading to unreliable explanations . Chang et al . ( 2019 ) address this issue for images using conditional generative models for inpainting regions with realistic counterfactuals . Evaluating sample based feature importance remains largely unstudied for time series models . While more widely studied for image classification , ( Bach et al. , 2015 ; Fong and Vedaldi , 2017 ) these methods can not be directly extended to time series models due to complex time-series dynamics . Most efforts in this domain focus on population level feature importance ( Tyralis and Papacharalampous , 2017 ) . Suresh et al . ( 2017 ) is one of the few methods addressing sample based feature importance and use a method similar to Fong and Vedaldi ( 2017 ) , called `` feature occlusion '' . They replace each time series observation by a sample from uniform noise to evaluate its effect on model outcome to attribute feature importance . We argue that carefully choosing the counterfactual selection policy is necessary for derive reliable importances . Specifically , replacing observations with noisy out-of-domain samples can lead to arbitrary changes to model output that are not reflective of systematic behavior in the domain . Even if an observation is sampled from the domain distribution , it does not characterize temporal dynamics and dependencies well enough , potentially highlighting features that only reflect global model behavior , as opposed to sample specific feature importance . We therefore model the data–distribution in order to generate reliable counterfactuals . We demonstrate the implications of the choice of the generator ( and hence the counterfactuals ) on the quality of explanation .

Summary: This paper proposes an extension of feature occlusion (FO) [Suresh et al., 2017] in which they sample from a pre-trained generative model for replacing the observed variables. Technically, it is in the category of saliency maps, only with possibly larger perturbations defined by the generative model. Thus, it most likely should inherit the same properties as the saliency maps.