TSInsight: A local-global attribution framework for interpretability in time-series data

1 INTRODUCTION . Deep learning models have been at the forefront of technology in a range of different domains including image classification ( Krizhevsky et al. , 2012 ) , object detection ( Girshick , 2015 ) , speech recognition ( Dahl et al. , 2010 ) , text recognition ( Breuel , 2008 ) , image captioning ( Karpathy & Fei-Fei , 2015 ) and pose estimation ( Cao et al. , 2018 ) . These models are particularly effective in automatically discovering useful features . However , this automated feature extraction comes at the cost of lack of transparency of the system . Therefore , despite these advances , their employment in safety-critical domains like finance ( Knight , 2017 ) , self-driving cars ( Kim et al. , 2018 ) and medicine ( Zintgraf et al. , 2017 ) is limited due to the lack of interpretability of the decision made by the network . Numerous efforts have been made for the interpretation of these black-box models . These efforts can be mainly classified into two separate directions . The first set of strategies focuses on making the network itself interpretable by trading off some performance . These strategies include SelfExplainable Neural Network ( SENN ) ( Alvarez-Melis & Jaakkola , 2018 ) and Bayesian non-parametric regression models ( Guo et al. , 2018 ) . The second set of strategies focuses on explaining a pretrained model i.e . they try to infer the reason for a particular prediction . These attribution techniques include saliency map ( Yosinski et al. , 2015 ) and layer-wise relevance propagation ( Bach et al. , 2015 ) . However , all of these methods have been particularly developed and tested for visual modalities which are directly intelligible for humans . Transferring methodologies developed for visual modalities to time-series data is difficult due to the non-intuitive nature of time-series . Therefore , only a handful of methods have been focused on explaining time-series models in the past ( Kumar et al. , 2017 ; Siddiqui et al. , 2019 ) . We approach the attribution problem in a novel way by attaching an auto-encoder on top of the classifier . The auto-encoder is fine-tuned based on the gradients from the classifier . Rather than 1Code along with the trained models will be made publicly available upon publication asking the auto-encoder to reconstruct the whole input , we ask the network to only reconstruct parts which are useful for the classifier i.e . are correlated or causal for the prediction . In order to achieve this , we introduce a sparsity inducing norm onto the output of the auto-encoder . In particular , the contributions of this paper are twofold : • A novel attribution method for time-series data which makes it much easier to interpret the decision of any deep learning model . The method also leverages dataset-level insights when explaining individual decisions in contrast to other attribution methods . • Detailed analysis of the information captured by different attribution techniques using a simple suppression test on a range of different time-series datasets . This also includes analysis of the different out of the box properties achieved by TSInsight including generic applicability , contraction in the output space and resistance against trivial adversarial noise . 2 RELATED WORK . Since the resurgence of deep learning in 2012 after a deep network comprehensively outperformed its feature engineered counterparts ( Krizhevsky et al. , 2012 ) on the ImageNet visual recognition challenge comprising of 1.2 million images ( Russakovsky et al. , 2015 ) , deep learning has been integrated into a range of different applications to gain unprecedented levels of improvement . Significant efforts have been made in the past regarding the interpretability of deep models , specifically for image modality . These methods are mainly categorized into two different streams where the first stream is focused on explaining the decisions of a pretrained network which is much more applicable in the real-world . The second stream is directed towards making models more interpretable by trading off accuracy . The first stream for explainable systems which attempts to explain pretrained models using attribution techniques has been a major focus of research in the past years . The most common strategy is to visualize the filters of the deep model ( Zeiler & Fergus , 2013 ; Simonyan et al. , 2013 ; Yosinski et al. , 2015 ; Palacio et al. , 2018 ; Bach et al. , 2015 ) . This is very effective for visual modalities since images are directly intelligible for humans . Zeiler & Fergus ( 2013 ) introduced deconvnet layer to understand the intermediate representations of the network . They not only visualized the network , but were also able to improve the network based on these visualizations to achieve state-of-the-art performance on ImageNet ( Russakovsky et al. , 2015 ) . Simonyan et al . ( 2013 ) proposed a method to visualize class-specific saliency maps . Yosinski et al . ( 2015 ) proposed a visualization framework for image based deep learning models . They tried to visualize the features that a particular filter was responding to by using regularized optimization . Instead of using first-order gradients , Bach et al . ( 2015 ) introduced a Layer-wise Relevance Propagation ( LRP ) framework which identified the relevant portions of the image by distributing the contribution to the incoming nodes . Smilkov et al . ( 2017 ) introduced the SmoothGrad method where they computed the mean gradients after adding small random noise sampled from a zero-mean Gaussian distribution to the original point . Integrated gradients method introduced by Sundararajan et al . ( 2017 ) computed the average gradient from the original point to the baseline input ( zero-image in their case ) at regular intervals . Guo et al . ( 2018 ) used Bayesian non-parametric regression mixture model with multiple elastic nets to extract generalizable insights from the trained model . Either these methods are not directly applicable to time-series data , or are inferior in terms of intelligibility for time-series data . Palacio et al . ( 2018 ) introduced yet another approach to understand a deep model by leveraging auto-encoders . After training both the classifier and the auto-encoder in isolation , they attached the auto-encoder to the head of the classifier and fine-tuned only the decoder freezing the parameters of the classifier and the encoder . This transforms the decoder to focus on features which are relevant for the network . Applying this method directly to time-series yields no interesting insights ( Fig . 1b ) into the network ’ s preference for input . Therefore , this method is strictly a special case of the TSInsight ’ s formulation . In the second stream for explainable systems , Alvarez-Melis & Jaakkola ( 2018 ) proposed SelfExplaining Neural Networks ( SENN ) where they learn two different networks . The first network is the concept encoder which encodes different concepts while the second network learns the weightings of these concepts . This transforms the system into a linear problem with a set of features making it easily interpretable for humans . SENN trade-offs accuracy in favor of interpretability . Kim et al . ( 2018 ) attached a second network ( video-to-text ) to the classifier which was responsible for the production of natural language based explanation of the decisions taken by the network using the saliency information from the classifier . This framework relies on LSTM for the generation of the descriptions adding yet another level of opaqueness making it hard to decipher whether the error originated from the classification network or from the explanation generator . Kumar et al . ( 2017 ) made the first attempt to understand deep learning models for time-series analysis where they specifically focused on financial data . They computed the input saliency based on the firstorder gradients of the network . Siddiqui et al . ( 2019 ) proposed an influence computation framework which enabled exploration of the network at the filter level by computing the per filter saliency map and filter importance again based on first-order gradients . However , both methods lack in providing useful insights due to the noise inherent to first-order gradients . Another major limitation of saliency based methods is the sole use of local information . Therefore , TSInsight significantly supersedes in the identification of the important regions of the input using a combination of both local information for that particular example along with generalizable insights extracted from the entire dataset in order to reach a particular description . Due to the use of auto-encoders , TSInsight is inherently related to sparse ( Ng et al. , 2011 ) and contractive auto-encoders ( Rifai et al. , 2011 ) . In sparse auto-encoders ( Ng et al. , 2011 ) , the sparsity is induced on the hidden representation by minimizing the KL-divergence between the average activations and a hyperparameter which defines the fraction of non-zero units . This KL-divergence is a necessity for sigmoid-based activation functions . However , in our case , the sparsity is induced directly on the output of the auto-encoder , which introduces a contraction on the input space of the classifier , and can directly be achieved by using Manhattan norm on the activations as we obtain real-valued outputs . Albeit sparsity being introduced in both cases , the sparsity in the case of sparse auto-encoders is not useful for interpretability . In the case of contractive auto-encoders ( Rifai et al. , 2011 ) , a contraction mapping is introduced by penalizing the Fobenius norm of the Jacobian of the encoder along with the reconstruction error . This makes the learned representation invariant to minor perturbations in the input . TSInsight on the other hand , induces a contraction on the input space for interpretability , thus , favoring sparsity inducing norm . 3 METHOD . We first train an auto-encoder as well as a classifier in isolation on the desired dataset . Once both the auto-encoder as well as the classifier are trained , we attach the auto-encoder to the head of the classifier . TSInsight is based on a novel loss formulation , which introduces a sparsity-inducing norm on the output of the auto-encoder along with a reconstruction and classification penalty for the optimization of the auto-encoder keeping the classifier fixed . Inducing sparsity on the auto-encoder ’ s output forces the network to only reproduce relevant regions of the input to the classifier since the auto-encoder is optimized using the gradients from the classifier . As inducing sparsity on the auto-encoder ’ s output significantly hampers the auto-encoder ’ s ability to reconstruct the input which can in turn result in fully transformed outputs , it is important to have a reconstruction penalty in place . This effect is illustrated in Fig . 2a where the auto-encoder produced a novel sparse representation of the input , which albeit being an interesting one , doesn ’ t help with the interpretability of the model . Therefore , the proposed optimization objective can be written as : ( W ′ E , W ′ D ) = arg min W∗E , W∗D 1 |X | ∑ ( x , y ) ∈X×Y [ L ( Φ ( D ( E ( x ; W∗E ) ; W∗D ) ; W∗ ) , y ) + γ ( ‖x−D ( E ( x ; W∗E ) ; W∗D ) ‖22 ) + β ( ‖D ( E ( x ; W∗E ) ; W∗D ) ‖1 ) ] + λ ( ‖W∗E‖22 + ‖W∗D‖22 ) ( 1 ) where L represents the classification loss function which is cross-entropy in our case , Φ denotes the classifier with pretrained weightsW∗ , while E and D denotes the encoder and decoder respectively with corresponding pretrained weightsW∗E andW∗D . We introduce two new hyperparameters , γ and β. γ controls the auto-encoder ’ s focus on reconstruction of the input . β on the other hand , controls the sparsity enforced on the output of the auto-encoder . Pretrained weights are obtained by training the auto-encoder as well as the classifier in isolation as previously mentioned . With this new formulation , the output of the auto-encoder is both sparse as well as aligned with the input as evident from Fig . 2b . The selection of β can significantly impact the output of the model . Performing grid search to determine this value is not possible as large values of β results in models which are more interpretable but inferior in terms of performance , therefore , presenting a trade-off between performance and interpretability which is difficult to quantify . A rudimentary way which we tested for automated selection of these hyperparameters ( β and γ ) is via feature importance measures ( Siddiqui et al. , 2019 ; Vidovic et al. , 2016 ) . The simplest candidate for this importance measure is saliency . This can be written as : I ( x ) = ∂aL ∂x where L denotes the number of layers in the classifier and aL denotes the activations of the last layer in the classifier . This computation is just based on the classifier i.e . we ignore the auto-encoder at this point . Once the values of the corresponding importance metric is evaluated , the values are scaled in the range of [ 0 , 1 ] to serve as the corresponding reconstruction weight i.e . γ . The inverted importance values serve as the corresponding sparsity weight i.e . β. I ( x ) = I ( x ) −min j I ( x ) j max j I ( x ) j −min j I ( x ) j γ∗ ( x ) = I ( x ) & β∗ ( x ) = 1.0− I ( x ) Therefore , the final term imposing sparsity on the classifier can be written as : γ ( ‖x−D ( E ( x ; W∗E ) ; W∗D ) ‖22 ) + β ( ‖D ( E ( x ; W∗E ) ; W∗D ) ‖1 ) ⇒ C × ‖D ( E ( x ; W∗E ) ; W∗D ) β∗ ( x ) ‖1 + ‖ ( x−D ( E ( x ; W∗E ) ; W∗D ) ) γ∗ ( x ) ‖22 In contrast to the instance-based value of β , we used the average saliency value in our experiments . This ensures that the activations are not sufficiently penalized so as to significantly impact the performance of the classifier . Due to the low relative magnitude of the sparsity term , we scaled it by a constant factor C ( we used C = 10 in our experiments ) . This approach despite being interesting , still results in inferior performance as compared to manual fine-tuning of hyperparameters . This needs further investigation for it to work in the future .

Summary: The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.