TSInsight: A local-global attribution framework for interpretability in time-series data
Authors: Shoaib Ahmed Siddiqui
Dominique Mercier
Andreas Dengel
Sheraz Ahmed
Venue:  (2020)
Keywords: Deep Learning
Representation Learning
Convolutional Neural Networks
Time-Series Analysis
Feature Importance
Visualization
Demystification

Abstract: With the rise in employment of deep learning methods in safety-critical scenarios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, time-series data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in a novel way by proposing TSInsight where we attach an auto-encoder with a sparsity-inducing norm on its output to the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty. The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves as a feature attribution method to boost interpretability. In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated or causal for the prediction. In contrast to most other attribution frameworks, TSInsight is capable of generating both instance-based and model-based explanations. We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy. Furthermore, we analyzed the set of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction. The obtained results advocate that TSInsight can be an effective tool for the interpretability of deep time-series models.

Under review as a conference paper at ICLR 2020
TSINSIGHT: A LOCAL-GLOBAL ATTRIBUTION FRAME- WORK FOR INTERPRETABILITY IN TIME-SERIES DATA
Anonymous authors Paper under double-blind review
ABSTRACT
With the rise in employment of deep learning methods in safety-critical scenar- ios, interpretability is more essential than ever before. Although many different directions regarding interpretability have been explored for visual modalities, time- series data has been neglected with only a handful of methods tested due to their poor intelligibility. We approach the problem of interpretability in novel way by proposing TSInsight where we attach an auto-encoder with norm on its output the classifier and fine-tune it based on the gradients from the classifier and a reconstruction penalty. The auto-encoder learns to preserve features that are important for the prediction by the classifier and suppresses the ones that are irrelevant i.e. serves feature attribution method to boost interpretability. In other words, we ask the network to only reconstruct parts which are useful for the classifier i.e. are correlated causal for the prediction. In contrast to most other attribution frameworks, TSInsight is capable of generating both instance- -based and model-basee explanations. We evaluated TSInsight along with other commonly used attribution methods on a range of different time-series datasets to validate its efficacy. Furthermore, analyzed the of properties that TSInsight achieves out of the box including adversarial robustness and output space contraction. The ob- tained results advocate that ISInsight can be an effective tool for the interpretability of deep time-series models.
INTRODUCTION
Deep learning models have been the forefront of technology in a range of different domains including image classification (Krizhevsky et al., 2012), object detection (Girshick, 2015), speech recognition (Dahl al., 2010), text recognition (Breuel, 2008), image captioning (Karpathy & Fei-Fei, 2015) and pose estimation (Cao et al., 2018). These models are particularly effective in automatically discovering useful features. However, this automated feature extraction comes at the cost of lack of transparency of the system. Therefore, despite these advances, their employment in safety-critical domains like finance (Knight, 2017), self-driving cars (Kim et al., 2018) and medicine (Zintgraf et al., 2017) is limited due the lack of interpretability of the decision made by the network.
Numerous efforts have been made for the interpretation of these black-box models. These efforts be mainly classified into two separate directions. The first set of strategies focuses on making the network itself interpretable by trading off some performance. These strategies include Self- Explainable Neural Network (SENN) (Alvarez-Melis & Jaakkola, 2018) and Bayesian non-parametric regression models (Guo et al., 2018). The second set of strategies focuses on explaining a pretrained model i.e. they try infer the reason for a particular prediction. These attribution techniques include saliency map (Yosinski et al., 2015) and layer-wise relevance propagation (Bach et al., 2015). However, all of these methods have been particularly developed and tested for visual modalities which are directly intelligible for humans. Transferring methodologies developed for visual modalities to time-series data is difficult due to the non-intuitive nature of time-series. Therefore, only a handful of methods have been focused on explaining time-series models in the past (Kumar et al., 2017; Siddiqui et al., 2019).
We approach the attribution problem in a novel way by attaching an auto-encoder top of the classifier. The auto-encoder fine-tuned based on the gradients from the classifier. Rather than
Code along with the trained models will be made publicly available upon publication

---

Under review as a conference paper at ICLR 2020
asking the auto-encoder to reconstruct the whole input, ask the network only reconstruct parts which are useful for the classifier i.c. are correlated or causal for the prediction. order to achieve this, we introduce sparsity inducing norm onto the output of the auto-encoder. In particular, the contributions of this paper are twofold:
A novel attribution method for time-series data which makes it much easier to interpret the decision of any deep learning model. The method also leverages dataset-level insights when explaining individual decisions in contrast to other attribution methods.
Detailed analysis of the information captured by different attribution techniques using simple suppression test on a range of different time-series datasets. This also includes analysis of the different out of the box properties achieved by TSInsight including generic applicability, contraction in the output space and resistance against trivial adversarial noise.
RELATED WORK
Since the resurgence of deep learning in 2012 after a deep network comprehensively outperformed its feature engineered counterparts (Krizhevsky et al., 2012) on the ImageNet visual recognition challenge comprising of 1.2 million images (Russakovsky et al., 2015), deep learning has been integrated into a range of different applications to gain unprecedented levels of improvement. Significant efforts have been made in the past regarding the interpretability of deep models, specifically for image modality. These methods are mainly categorized into two different streams where the first stream is focused on explaining the decisions of pretrained network which is much more applicable in the real-world. The second stream is directed towards making models more interpretable by trading off accuracy.
The first stream for explainable systems which attempts to explain pretrained models using attribution techniques has been a major focus of research in the past years. The most common strategy is to visualize the filters of the deep model (Zeiler & Fergus, 2013; Simonyan et al., 2013; Yosinski et al., 2015; Palacio et al., 2018; Bach et al., 2015). This is very effective for visual modalities since images are directly intelligible for humans. Zeiler & Fergus (2013) introduced deconvnet layer to understand the intermediate representations of the network. They not only visualized the network, but were also able to improve the network based on these visualizations to achieve state-of-the-arr performance on ImageNet (Russakovsky et al., 2015). Simonyan et al. (2013) proposed a method to visualize class-speciff saliency maps. Yosinski et al. (2015) proposed a visualization framework for image based deep learning models. They tried to visualize the features that a particular filter was responding to by using regularized optimization. Instead of using first-order gradients, Bach et al. (2015) introduced Layer- wise Relevance Propagation (LRP) framework which identified the relevant portions of the image by distributing the contribution to the incoming nodes. Smilkov et al. (2017) introduced the SmoothGrad method where they computed the mean gradients after adding small random noise sampled from a zero-mean Gaussian distribution to the original point. Integrated gradients method introduced by Sundararajan al. (2017) computed the average gradient from the original point to the baseline input (zero-image in their case) at regular intervals. Guo et al. (2018) used Bayesian non-parametric regression mixture model with multiple elastic nets extract generalizable insights from the trained model. Either these methods are not directly applicable to time-series data, or are inferior in terms of intelligibility for time-series data.
Palacio et al. (2018) introduced yet another approach to understand a deep model by leveraging auto-encoders. After training both the classifier and the auto-encoder in isolation, they attached the auto-encoder to the head of the classifier and fine-tuned only the decoder freezing the parameters of the classifier and the encoder. This transforms the decoder to focus on features which are relevant for the network. Applying this method directly to time-series yields no interesting insights (Fig. 1b) into the network's preference for input. Therefore, this method is strictly a special case of the TSInsight's formulation.
In the second stream for explainable systems, Alvarez-Melis & Jaakkola (2018) proposed Self- Explaining Neural Networks (SENN) where they learn two different networks. The first network is the concept encoder which encodes different concepts while the second network learns the weightings of these concepts. This transforms the system into linear problem with set of features making it easily interpretable for humans. SENN trade-offs accuracy in favor of interpretability. Kim et al. (2018) attached a second network (video-to-text) to the classifier which was responsible for the
2

---

Under review as a conference paper at ICLR 2020
(a) Auto-encoder
(b) Fine-tuned auto-encoder
Figure I: Plot at the top shows the original input while plot at the bottom indicates the reconstructed input from the raw auto-encoder or the fine-ture tne-ttned auto-encoder.
production of natural language based explanation of the decisions taken by the network using the saliency information from the classifier. This framework relies on LSTM for the generation of the descriptions adding yet another level of opaqueness making it hard to decipher whether the error originated from the classification network or from the explanation generator.
Kumar et al. (2017) made the first attempt to understand deep learning models for time-series analysis where they specifically focused on financial data. They computed the input saliency based on the first- order gradients of the network. Siddiqui al. (2019) proposed an influence computation framework which enabled exploration of the network at the filter level by computing the per filter saliency map and filter importance again based on first-order gradients. However, both methods lack in providing useful insights due to the noise inherent to first-order gradients. Another major limitation of saliency based methods the sole use of local information. Therefore, TSInsight significantly supersedes in the identification of the important regions of the input using a combination of both local information for that particular example along with generalizable insights extracted from the entire dataset in order to reach particular description.
Due to the use of auto-encoders, TSInsight inherently related to sparse (Ng et al., 2011) and contractive auto-encoders (Rifai et al., 2011). In sparse auto-encoders (Ng et al., 2011 1), the sparsity induced on the hidden representation by minimizing the KL-divergence between the average activations and hyperparameter which defines the fraction of non-zero units. This KL-divergence a necessity for sigmoid- -ggeeddbbsed activation functions. However, in our case, the sparsity is induced directly on the output of the auto-encoder, which introduces contraction on the input space of the classifier, and can directly be achieved by using Manhattan norm on the activations as we obtain real-valued outputs. Albeit sparsity being introduced in both cases, the sparsity in the case of sparse auto-encoders is not useful for interpretability. In the case of contractive auto-encoders (Rifai et al., 2011), contraction mapping introduced by penalizing the Fobenius norm of the Jacobian of the encoder along with the reconstruction error. This makes the learned representation invariant to minor perturbations the input. TSInsight on the other hand, induces a contraction on the input space for interpretability, thus, favoring sparsity inducing norm.
METHOD
We first train an auto-encoder well as a classifier in isolation on the desired dataset. Once both the auto-encoder as well as the classifier are trained, we attach the auto-encoder to the head of the classifier. TSInsight based on novel loss formulation, which introduces a sparsity-inducin norm on the output of the auto-encoder along with reconstruction and classification penalty for the optimization of the auto-encoder keeping the classifier fixed. Inducing sparsity on the auto-encoder's output forces the network to only reproduce relevant regions of the input to the classifier since the auto-encoder is optimized using the gradients from the classifier. As inducing sparsity on the auto-encoder's output significantly hampers the auto-encoder's 's ability reconstruct the input which can in turn result in fully transformed outputs, it is important to have a reconstruction penalty in place. This effect is illustrated in Fig. 2a where the auto-encoder produced novel sparse representation of
3

---

Under review as a conference paper at ICLR 2020
(a) Fine-tuned auto-encoder with Sparsity
(b) TSInsight
Figure 2: Plot at the top shows the original input while plot at the bottom indicates the reconstructed
input either from TSInsight or the auto-encoder with just sparsity inducing norm.
the input, which albeit being an interesting one, doesn't 't help with the interpretability of the model. Therefore, the proposed optimization objective can be written as:
c ((((D(EXX<W)))))))))) (x.))XXX
We.Wo) arg min Wg |
Σ
7 -1xx- (E) ( (1D( +(IMEEB++ 135133 (1)
where L represents the classification loss function which is cross-entropy in case, &b denotes the classifier with pretrained weights W while and denotes the encoder and decoder respectively with corresponding pretrained weights W and WD. We introduce two new hyperparameters, 7 and 3. controls the auto-encoder's 's focus on reconstruction of the input. 8 on the other hand, controls the sparsity enforced on the output of the auto-encoder. Pretrained weights are obtained by training the auto-encoder as well as the classifier in isolation as previously mentioned. With this new formulation, the output of the auto-encoder both sparse as well as aligned with the input as evident from Fig. 2b.
The selection of 3 can significantly impact the output of the model. Performing grid search to determine this value not possible large values of results in models which are more interpretable but inferior in terms of performance, therefore, presenting a trade-off between performance and interpretability which is difficult quantify. A rudimentary way which we tested for automated selection of these hyperparameters (3 and 7) is via feature importance measures (Siddiqui et al., 2019; Vidovic al., 2016). The simplest candidate for this importance measure is saliency. This can be written as:
ta 0x
((x)
where L denotes the number of layers in the classifier and a² denotes the activations of the last layer the classifier. This computation is just based on the classifier i.e. we ignore the auto-encoder at this point. Once the values of the corresponding importance metric is evaluated, the values are scaled in the range of [0, 1] to serve the corresponding reconstruction weight i.,, 7. The inverted importance values serve as the corresponding sparsity weight i.e. B.
T(x) min T(x);
1(x) max (xx), - min (xx);
7*(x) =(x) & B²(x) = 1.0 T(x)
4

---

Under review as a conference paper at ICLR 2020
Table 1: Dataset Details
Dataset
Train
Validation
Test
Sequence Length
Input Channels
Output Classes
Synthelic Anomaly Detection
45000
5000
10000
50
2
Electric Devices
6244
2682
7711
50
7
Character Trajectories
1383
606
869
206
20
FordA
2520
1081
1320
500
2
Forest
107110
45906
65580
2
BCG Thorax
1244
556
1965
750
42
5929
UWave Gesture
624
272
3582
946
8
Therefore, the final term imposing sparsity on the classifier can be written as:
<1Ix - D(E(x:WE)))) + (II( Cx (XI] o 8" (x)ll + ((x D-E(((((x)))))) WF): @ (
In contrast to the instance-based value of B, we used the average saliency value in our experiments. This ensures that the activations are not sufficiently penalized so significantly impact the performance of the classifier. Due to the low relative magnitude of the sparsity term, we scaled it by constant factor C (we used = 10 in our experiments). This approach despite being interesting, still results in inferior performance compared manual fine-tuning of hyperparameters. This needs further investigation for it to work in the future.
4 DATASETS
In order investigate the efficacy TSInsight, employed several different time-series datasets in this study. The summary of the datasets is available in Table 1.
Synthetic Anomaly Detection Dataset: The synthetic anomaly detection dataset (Siddiqui al., 2019) is a synthetic dataset comprising of three different channels referring to the pressure, tem- perature and torque values of a machine running in production setting where the task is detect anomalies. The dataset only contains point-anomalies. If a point-anomaly is present in sequence, the whole sequence marked as anomalous. Anomalies were intentionally never introduced on the pressure signal in order to identify the treatment of the network to that particular channel.
Electric Devices Dataset: The electric devices dataset (Hills al., 2014) a small subset of the data collected as part of the UK govermment's 's sponsored study, Powering the Nation. The aim of this study was to reduce UK's carbon footprint. The electric devices dataset is comprised of data from 251 households, sampled in two-minute intervals
month.
The character trajectories dataset contains hand- hand-written characters using a Wacom tablet. Only three dimensions kept for the final dataset which includes x, y and pen-tip force. The sampling rate was set to be 200 Hz. The data was numerically differentiated and Gaussian smoothen with 2. The task
Character Trajectories Dataset:
classify the characters into 20 different classes.
The FordA dataser² was originally used for a competition organized by IEEE the IEEE World Congress on Computational Intelligence (2008). It is binary classification problem where the task is to identify whether a certain symptom exists in the automotive subsystem. FordA dataset was collected with minimal noise contamination typical operating conditions.
FordA Dataset:
The forest cover dataset (Tan et al., 2011) 1) has been adapted from the UCI repository for the classification of forest cover type from cartographic variables. The dataset has been transformed into an anomaly detection dataset by selecting only 10 quantitative attributes out of total of 54. Instances from second class were considered be normal while instances from the
Forest Cover Dataset:
ics. uci. wtw...........mmnl...... http://www..immeeeees//t/ assi cat on.

---

Under review as a conference paper at ICLR 2020
Figure 3: Output from different attribution methods as well as the input after suppressing all the points except the top 5% highlighted by the corresponding attribution method on anomalous example from the synthetic anomaly detection dataset.
fourth class were considered to be anomalous. The ratio of the anomalies to normal data points is 0.90%. Since only two classes were considered, the rest of them were discarded.
WESAD Dataset: WESAD dataset (Schmidt et al., 2018) is a classification dataset introduced by Bosch for person's affective state classification with three different classes, namely, neutral, amusement and stress.
ECG Thorax Dataset: The non-invasive fetal ECG Thorax dataser is a classification dataset comprising of 42 classes.
UWave Gesture Dataset: The wave gesture dataset (Liu et al., 2009) contains accelcrometer data where the task is to recognize 8 different gestures.
5 MAIN RESULTS
The results we obtained with the proposed formulation were highly intelligible for the datasets we employed this study. TSInsight produced a sparse representation of the input focusing only on the salient regions. With careful tuning of the hyperparameters, TSInsight outperformed the base classifier in terms of accuracy for most of the cases. This is evident from Table 3 (Appendix G). However, it is important note that TSInsight is not designed for the purpose of performance, but rather for interpretability. Therefore, we expect that the performance will drop in many cases depending on the amount of sparsity enforced.
In order to assess the obtained reconstructions qualitatively, we visualize an anomalous example from the synthetic anomaly detection dataset in Fig. 3 along with the attributions from all the commonly employed attribution techniques (listed below) including TSInsight. Since there were only a few relevant discriminative points the case of forest cover and synthetic anomaly detection datasets, the auto-encoder suppressed most of the input making the decision directly interpretable.
simple way quantify the quality of the attribution is to just preserve parts of the input that are considered to be important by the method, and then pass the suppressed input to the classifier. If
'http:/ httt:://www... t dimeserieesll a i fi inn. com/descriptionn ph. ?Dataset- Non ivvas ve

---

Under review as a conference paper at ICLR 2020
the selected points are indeed causal for the prediction generated by the classifier, the prediction would stand. Otherwise, the prediction will flip. It is important to note that unless there a high amount of sparsity present in the signal, suppressing the signal itself will result in loss of accuracy for the classifier since there is a slight mismatch for the classifier for the inputs seen during training. We compared TSInsight with range of different saliency methods. In all of the cases, we used the absolute magnitude of the corresponding feature attribution method to preserve the most-important input features. Two methods i.e. LRP and DeepLift were shown be similar to imput @ gradient (Adebayo et al., 2018), therefore, we compare only against imput C gradient. We don't 't compute class-specific saliency. but instead, compute the saliency w.r.t. all the output classes. For all the methods computing class specific activations maps e.g. GradCAM, guided GradCAM, and occlusion sensitivity, we used the class with the maximum predicted score as our target. Following is the list of the evaluated attribution techniques:
None: None refers to the absence of any importance measure. Therefore, in this case, the complete
input is passed on to the classifier without any suppression for comparison.
Random: Random points from the input are suppressed in this case.
Input Magnitude: We treat the absolute magnitude of the input to be a proxy for the features'
importance.
We iterate over different input channels and positions and mask the corre- sponding input features with a filter size of 3 and compute the difference in the confidence score of the predicted class (i.e. the class with the maximum score on the original input). We treat this sensitivity score as the features' importance. This is a brute-force measure of feature importance and employed commonly in prior literature served as strong baseline in experiments (Zeiler & Fergus, 2013).
Occlusion sensitivity:
TSInsight: We treat the absolute magnitude of the output from the auto-encoder of TSInsight as features' importance.
Similar TSInsight, we use the absolute magnitude of the auto-encoder's output as the features' importance (Palacio et al., 2018).
Palacio et al.:
Gradient: We use the absolute value of the raw gradient of the classifier w.r.t. to all of the classes as the features' importance (Siddiqui et al., 2019; Kumar et al., 2017)
Gradient @ Input: We compute the Hadamard delement-wise, product between the gradient and the input, and use its absolute magnitude as the features' importance (Sundararajan et al., 2017).
Integrated Gradients: We use absolute value of the integrated gradient with 100 discrete steps between the input and the baseline (which was zero in our case) as the features' importance (Sun- dararajan al., 2017).
SmoothGrad: We use the absolute value of the smoothened gradient computed by using 100 different random noise vector sampled from Gaussian distribution with zero mean, and a variance of 2/ 2/(max,,xx min;xz) where x was the input as the features' importance measure (Smilkov et al., 2017).
Guided Backpropagation: We use the absolute value of the gradient provided by guided backpropa- gation (Springenberg et al., 2014). In this case, all the ReLU layers were replaced with guided ReLU layers which masks negative gradients, hence filtering out negative influences for particular class to improve visualization.
We use the absolute value of Gradient- Class Activation Map (GradCAM) (Sel- varaju et al., 2016) as our feature importance measure. GradCAM computes the importance of the different filters present in the input in order to come up with a metric to score the overall output. Since GradCAM visualizes a class activation map, we used the predicted class as the target for visualization.
GradCAM:
Guided GradCAM: Guided GradCAM Selvaraju et al. (2016) is a guided variant of GradCAM which performs a Hadamard product (pointwise) the signal from guided backpropagation and GradCAM to obtain guided GradCAM. We again the absolute value of the guided GradCAM output as importance measure.
7

---

Under review as a conference paper at ICLR 2020
(a) WESAD
(b) Character Trajectories
(c) ECG Thorax
(d) Electric Devices
(e) FordA
(f) Forest Cover
UWave Gesture
(h) Anomaly
Figure 4: Suppression results against a large number of baseline methods averaged over random runs. Enlarged version of the plots is available in Appendix H.
The results with different amount of suppression are visualized in Fig. 4 which are averaged over 5 random runs. Since the datasets were picked to maximize diversity in terms the features, there no perfect method which can perfectly generalize to all the datasets. The different attribution techniques along with the corresponding suppressed input is visualized in Fig. for the synthetic anomaly detection datasets. TSInsight produced the most plausible looking explanations along with being the most competitive saliency estimator on average in comparison to other attribution techniques. Alongside the numbers, TSInsight was also able produce the most plausible explanations.
PROPERTIES OF TSINSIGHT
6.1 GENERIC APPLICABILITY
TSInsight is compatible with any base model. We tested our method with two prominent architectural choices in time-series data i.c. CNN and LSTM. The results highlight that TSInsight was capable of extracting the salient regions of the input regardless of the underlying architecture. was interesting to note that since LSTM uses memory cells to remember past states, the last point was found to be the most salient. For CNN the other hand, the network had access to the complete information resulting in equal distribution of the saliency. visual example is presented in Appendix E.
6.2 MODEL-BASED VS INSTANCE-BASE BASED EXPLANATIONS
Since TSInsight poses the attribution problem itself as an optimization objective, the data based on which this optimization problem is solved defines the explanation scope. If the optimization problem solved for the complete dataset, this tunes the auto-encoder to be generic feature extractor, enabling extraction of model/datuset-leeel insights using the attribution. In contrary, if the optimization problem solved for a particular input, the auto-encoder discovers an instance's attribution. This is contrary most other attribution techniques which only instance specific.
AUTO-ENCODER S JACOBIAN SPECTRUM ANALYSIS
Fig. 5 visualizes the histogram of singular values of the average Jacobian test set of the forest cover dataset. We compare the spectrum the formulation from Palacio et al. (2018) and TSInsight. is evident from the figure that most of the singular values for TSInsight close to zero, indicating
8

---

Under review as a conference paper at ICLR 2020
Derape
Figure 5: Spectrum analysis of the auto-encoder's average Jacobian computed over the entire test set of the forest cover dataset. Large singular values corresponds to directions which the network retained. The sharp decrease in the spectrum for TSInsight suggests that the network was successful in inducing contraction of the input space.
nasgh
(a) Character Trajectories Datasct
(b) ECG Thorax Dataset
(c) UWave Gesture Dataset
Figure 6: Accuracy of the adversarial examples curated using FGSM attack with varying max c.
contraction being induced in those directions. This is similar to the contraction induced in contractive auto-encoders (Rifai et al., 2011) without explicitly regularizing the Jacobian of the encoder.
6.4 ADVERSARIAL ROBUSTNESS
We used the fastest and the simplest attack i.e. Fast-Gradient Sign Method (Goodfellow al., 2014) as dummy check. The accuracy with increasing values of are plotted to provide a hint regarding the possibility of attaining higher level robustness since the setup is similar to high-level representation interpretability instead of robustness (Liao et al., 2018). Recent
guided denoising with emphasis studies have indicated that the two objectives i.e. interpretability and robustness are complementary to each other (Engstrom et al., 2019). Fig. 6 indicates that TSInsight achieved high level of immunity against adversarial noise in comparison to the base classifier which is better or on par with Palacio al. (2018). It is important note that this is not a proper adversarial evaluation, but rather, only speculation which needs further investigation in the future.
7 CONCLUSION
We presented a novel method to discover the salient features of the input for the prediction by using the global context. With the obtained results, it evident that the features highlighted by TSInsight are intelligible as well as reliable at the same time. In addition to interpretability, TSInsight also offers off the shelf properties which are suitable in a wide range of problems. Interpretability essential in many domains, and we believe that our method opens up new research direction for interpretability of deep models for time-series analysis. We would like further investigate the automated selection of hyperparameters in the future which is primitive for the wide-scale applicability of TSInsight along with its impact on adversarial robustness.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-9515,

---

Under review as a conference paper at ICLR 2020
2018.
David Alvarez-Melis and Tommi S. Jaakkola. Towards robust interpretability with self-explaining neural networks. CoRR, fbs//806.07538, 2018. URL httt:::/aaxx..ogg///o /1866. 07538.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Roberr Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, (00))440 2015. doi: 10.137
URL wwtp::ttp//.. /ttp doi. org/ 10 371 journal. pone. 0130140
Thomas M. Breuel. The ocropus open source ocr system, 2008. URL https://doi.org/// 1117/12. 783598.
Zhe Cao. Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint 181.0008,,.08008, 2018.
George Dahl, Marc' aurelio Ranzato, Abdel rahman Mohamed, and Geoffrey E Hin- ton. Phone recognition with the mean-covariance restricted boltzmann machine. In D. Lafferty, C. lotta (eds.), Advances in Neural Information Processing Systems 23, pp. 469- 477. Curran Associates, Inc., 2010. URL http:/ http://papers... nips wttpsesiiipsp..pppppeeeeeeeeee cc/paper/ 1169-phone-1 reconneioonniiinn-neeoo woon-witttthee mean covariancee restricteddeee pdf.
Williams, J. Shawe-Taylor, R. S. Zemel, and A. Cu-
machine..
Logan Engstrom, Andrew llyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Learning perceptually-aligned representations via adversarial robustness. arXiv preprint 10w.:19000009455. 2019,
Ross Girshick. Fast r-cnn. Proceedings of the IEEE international conference on computer vision, pp. 1440-1448, 2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014.
Wenbo Guo, Sui Huang, Yunzhe Tao, Xinyu Xing, and Lin Lin. Explaining deep learning models-a bayesian parameeeitiic approach. In Advances in Neural Information Processing Systems, pp. 4519-4529, 2018.
Jon Hills, Jason Lines, Edgaras Baranauskas, James Mapp, and Anthony Bagnall. Classification of time series by shapelet transformation. Data Mining and Knowledge Discovery, 28(4):851-881, 2014.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-31 -137, 2015.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John F. Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. In ECCV (2), volume 1 1206 of Lecture Notes in Computer Science, pp. 577-593. Springer, 2018.
Will Knight. Mit technology review: The financial world wants open ai's black boxes, 2017. URL https: /tttp:://ww..r wttp://wwwtccooooo.eeeeieee.eee wwwp://www.o///eeeeeeeeeeeeeeett com/s//00/2////////
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012. URL http: http::://papp.s..eee wwtaapers.niss.... wwwpss.cc/pppp... 4824-imagenet classification. pdf.
10

---

Under review as a conference paper at ICLR 2020
D. Kumar, G. W Taylor, and A. Wong. Opening the Black Box of Financial AI with CLEAR- Trade: A CLass- LLaass-Ehhaned Attentive Response Approach for Explaining and Visualizing Deep Learning-Drin Stock Market Prediction. ArXiv e-prints, September 2017.
Hao Li, Zheng Xu, Gavin Taylor, and Tom Goldstein. Visualizing the loss landscape of neural nets. CoRR, abs/ 0tbs//712.099/) 2017. URL http: http/arxvvv//ca/aa//aaa http:::://axxi..rg/abb..... 1712.1111111 09913.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang. Xiaolin Hu, and Jun Zhu, Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1778-1787, 2018.
Jiayang Liu, Lin Zhong, Jehan Wickramasuriya,, and Venu Vasudevan. uwave: Accelerometer-busee personalized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6): 657 - 675, 2009. ISSN 1574-1192. doi: //www. sciencedirect http:///eiiieeeerrrra.eeaa////// Com 2009.
URL http:
Per-
Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes, 72(2011))1-19) 2011.
Sebastian Palacio, Joachim Folz, Jörn Hees, Federico Raue, Damian Borth, and Andreas Dengel. What do deep networks like see? In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 3108-3117, 2018.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learming, pp. 833-840. Omnipress, 2011.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-255) 2015. doi: 10.1007/sl 1263-015-0816->:
Philip Schmidt, Attila Reiss, Robert Duerichen, Claus Marberger, and Kristof Van Laerhoven. Introducing wesad, a multimodal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM International Conference on Multimodal Interaction, ICMI '18, pp. 400-408. New York, NY, USA, 2018. ACM. ISBN 978- 1 4503-5692333 doi: 10. 10.114553344999932449999 URL http::tp/// /toi. acm. www.orgg///oooo..oooooooo /1/./1..000 1100/1115/// 32 426.9. 3242985.
Ramprasaath Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? visual explanations from deep networks via gradient- based localization. CoRR, abs/1610.02391,, 2016. URL 1610.02339 02391.
owg.ziivoorg//aa. abs/
Shoaib Ahmed Siddiqui, Dominique Mercier, Mohsin Munir, Andreas Dengel, and Sheraz Ahmed. Tsviz: Demystification of deep learning models for time-serics analysis. IEEE Access, pp. 1-1, 2019. ISSN 2169-3536. doi: 10. L09////CESSSS2019.22900
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013. URL http:// rxii. www.wvvw./.....o........oooo..rt abs /1/12.1111 6034.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1ttv:/706.0330 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint
2014.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 3319-3328. JMLR. org, 2017.
Swee Chuan Tan, Kai Ming Ting, and Tony Fei Liu. Fast anomaly detection for streaming data. In IJCAI cerpediinng-nneeaationnn Joint Conference on Artificial Intelligence, volume 22, pp. 1511, 2011.
11

---

Under review as a conference paper at ICLR 2020
R.
Salerey Compuratin
Santly
(1
Train Autto Eocoder
(MS55)
04.
Sa
Munno
Tran Comuster (GSMGF))
Thai Modds
Figure 7: System Pipeline
Marina M-C Vidovic, Nico Görnitz, Klaus-Robert Müller, and Marius Kloft. Feature importance measure for non-linear learning algorithms. arXiv preprint
2016.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. CoRR, abs/1311.2901,, 2013. URL http::/aaxx....r/aaaaatt..
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 172.:1702.0459 2017.
A APPENDIX
B SYSTEM PIPELINE
The system overview pipeline is visualized in Fig. 7.
12

---

Under review as a conference paper at ICLR 2020
C HYPERPARAMETERS
Table 2: Hyperparameters
Hyperparameter
Value
Initial learning rate
0.0001
Learning rate reduction factor
0.9
Learning rate reduction tolerance
Activation regularization (L1)
0.0001
Reconstruction weight
4.0
Max epochs
Batch size
256
Early stopping patience
10
D LOSS LANDSCAPE
We analyze the loss landscape in order asses the impact of stacking the auto-encoder on top of the original network on the overall optimization problem. We follow the scheme suggested by Li et first perform filter normalization using the norm of the filters.
al. (2018) Li et al. (2017) where This allows the network to be scale invariant. We then sample two random directions (5 and 17) and linear combination of these directions to identify the loss landscape. We keep the values of the classifier in the combined model intact since we treat those parameters as fixed. The function representing the manifold be written
(0, D(O" + aδ BB)
Va, E 1-1.0,-0.95. 1005.-0.90 -0.90, .... 0.90, 0.95, 1.0] (2)
Once the loss function is evaluated for all the values of and 3 (4000 different combinations), we plot the resulting function as a 3D surface. This loss landscape for the model trained on forest cover dataset is visualized in Fig. 8. The surface at the bottom (mostly blue) signifies the loss landscape for the classifier. The landscape was nearly convex. The surface on the top is from the model coupled with the auto-encoder. can be seen that the loss landscape has a kink at the optimal position but remains flat otherwise with significantly higher loss value. This indicates that problem of optimizing the auto-encoder using gradients from the classifier is significantly harder one to solve. This is consistent with our observation where the network failed to converge in many cases. Similar observations have been made by Palacio et al. Palacio et al. (2018) where they failed to fine-tune the complete auto-encoder, resorting to only fine-tuning of the decoder to make the problem tractable. The results were very similar when tested on other datasets.
E GENERIC APPLICABILITY
example of TSInsight trained with CNN and an LSTM presented in Fig. 9.
HARDWARE
All the models were trained on single V-100 (16 GB) using the DGX-1 system. The system is equipped with 512 GB Ram and two Xeon E5-2698 v4 processors (2.20GHz).
G CLASSIFICATION ACCURACY
The results from the classifier training highlighting the obtained accuracies are presented Table It is evident from the table that attaching TSInsight had no statistically significant impact the classification performance. Although the accuracy itself went up after attaching the auto-encoder, we consider it to be coincidence rather than feature of TSInsight.
13

---

Under review as a conference paper at ICLR 2020
Figure 8: Loss landscape where the bottom surface indicates the manifold for the classifier while the surface on the top indicates the manifold for the auto-encoder attached to the classifier.
Table 3: Results for the different datasets in terms of accuracy for both the classifier well as TSInsight.
Dataset
Model
9
Accuracy
Synthetic Anomaly
Classifier TSInsight
98.01 %
Detection
1.0
0.001
98. 13
WESAD
Classifeer
99.94 %
TSInsight
Character Trajectories
Classiferr TSInsight
97.01 15
0.25
0.0001
97.24
Classifier TSInsight
2.0
0.0001
93.26 %
Forest Cover
Classifer TSInsight
95.79 18:
4.0
0.0001
96.26
Electric Devices
Classfier TSInsight
65.14 56
ECG Thorux
Classiier TSInsight
86.01 15
0.0001
84.07
U'Wvve
Classifier TSInsight
4.0
0.0005
92.29 %
14

---

Under review as a conference paper at ICLR 2020
(a) CNN Model
(b) LSTM Model
Figure 9: Auto-encoder training with different base models (CNN and LSTM). TSInsight was able to discover salient regions of the input regardless of the employed classifier.
H ATTRIBUTION RESULTS
The enlarged version of the attribution plots (Fig. 4) are presented in four different figures i.c. Fig. 10, Fig. 11, Fig. 12 and Fig. 13.
15

---

Under review as a conference paper at ICLR 2020
Sussraston Oeluoet
Guiced CrudCrr
Palaci
Smouthorse
15mmight
Taxas nou tnuaaa proveed
(a) WESAD
Suppression Patt Cho note" Tipecttr Detaset
GINEAM
QuiicE Dec-gronnpoon
Yoar sighisss
DV voorrayy
15iugh)
06
66
10
(b) Character Trajectories
Figure 10: Enlarged suppression plots (a). Copy of Fig, 4.
16

---

Under review as a conference paper at ICLR 2020
Suppresu un 220 Thenor Orlond
64-
sakded
Guiced CrudCir
Magr
mnaderion
Palacii
Smouthorse
15mmight
Taxe nou tndara proveed
ECG Thorax
Suppression Mott for BiecDic Devices Datiseet
GINEAA
Quiicc Dac-gronppoon
Yoar sighisss
Pragrited
De voorrayy
Handow
06
66
10
(b) Electric Devices
Figure 11: Enlarged suppression plots (b). Copy of Fig. 4.
17

---

Under review as a conference paper at ICLR 2020
Superesaten
sabded
Quiced CrudCrr
Palaci et
Smoothorse
15mmight
Taxcslage nout tndara proveed
(a) FordA
Supp espin Hot for Firest Cover Datoset
62
04
GINEAM
Quiict Dec-gronppion
Yoar saghinsss
Pragrited
Handow
06
66
10
Forest Cover
Figure 12: Enlarged suppression plots (c). Copy of Fig, 4.
18

---

Under review as a conference paper at ICLR 2020
Supprooo Cebtars
sabded
Quieed CrudCrrs
Lie mrderaa
Palaci
Senothorse
15mmight
Taxal nout tndana proveed
(a) UWave Gesture
Superesson Pot Syrthet Arumaly Detection Dateseet
GIEEAM
Quiict Dac-gronepann
0.20
Poar saghiniss
Pragrited
82
06
66
32
(b) Anomaly
Figure 13: Enlarged suppression plots (d). Copy of Fig. 4.
19


Reviews:
[{"review_id": "SJl2U01TFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new approach for improving the interpretability of deep learning methods used for time series. The is mainly concerned with classification tasks for time series. First, the classifier is learned in a usual way. Subsequently, a sparse auto-encoder is used that encodes the last layer of the classifier. For training the auto-encoder the classifier is fixed and there is a decoding loss as well as a sparsity loss. The sparse encoding of the last layer is supposed to increase the interpretability of the classification as it indicates which features are important for the classification.\n\nIn general, I think this paper needs to be considerably improved in order to justify a publication. I am not convinced about the interpretability of the sparse extracted feature vector. It is not clear to me why this should be more interpretable then other methods. The paper contains many plots where the compare to other attributes methods, but it is not clear why the presented results should be any better as other methods (for example Fig 3). The paper is missing also a lot of motivation, the results are not well explained (e.g. Fig 3) and it needs to be improved in terms of writing. Equation 1 is not motivated (which is the main part of the paper) and it is not clear how Figure 2a has been generated and why this represented should be \" an interesting one, doesn’t help with the interpretability of the model\". The authors have to improve the motivation part as well as the discussion of the results.\n\nMore comments below:\n- The method seems to suffer from a severe parameter tuning problem, which makes it hard to use in practise.\n- It is unclear to me why the discriminator is fixed during training the encoder and decoder. Shouldnt it improve performance to also adapt the discriminator to the new representation.\n- Why can we not just add a layer with a sparsity constraint one layer before the \"encoded\" layer such that we have the same architecture  and optimize that end to end? At least comparison to such an approach would be needed to justify something more complex. \n- The plots need to be better explained. While the comparisons seems to be exhaustive, it is already too many plots and it is very easy to get lost. Also, the quality of the plots need to be improved (e.g. font size)\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgYPG-3YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed an algorithm for identifying important inputs for the time-series data as an explanation of the model's output.\nGiven a fixed model, the authors proposed to put an auto-encoder to the input of the model, so that the input data is first transformed through the auto-encoder, and the transformed input is then fed to the model.\nIn the proposed algorithm, the auto-encoder is trained so that (i) the prediction loss on the model's output to be small, (ii) the reconstruction loss of the auto-encoder to be small, and (iii) the transformed input of the auto-encoder to be sufficiently sparse (i.e. it has many zeros).\n\nI am not very sure if the proposed algorithm can generate reasonable explanation, for the following two reasons.\n\nFirst, the auto-encoder transforms the input into sparse, which can completely differ from any of the \"natural\" data, as shown in Fig1(b).\nI am not very sure whether studying the performance of the model for such an \"outlying\" input is informative.\n\nSecond, it seems the authors implicitly assumed that zero input is irrelevant to the output of the model.\nHowever, zero input can have a certain meaning to the model, and thus naively introducing the sparsity to the model input may bias the model's output.\n\nI think the paper lacks deeper considerations on the use of sparsity.\nThus, to me, the soundness of the proposed approach is not very clear to me.\n\n\n### Updated after author response ###\nThrough the communication with the author, I found that the author seems to be confusing the two different approaches *sparsifying the feature importance* and *sparsifying the input data*.\nThis paper considers the latter approach, which can be biased as I mentioned above.\nThe authors tried to justify the approach by raising related studies, which is however the mix of the two different approaches.\nI expect the author to clarify the distinction between the two approaches, and provide strong evidences that the sparsity is not a harm (if there is any).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lLPJJ0KS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.\n\n* Pros\n\t* The work provides extensive comparison to a battery of other methods for model prediction interpretation. \n\t* The method is conceptually simple and is easy to implement. It is also general and can be applied to any prediction model (though this is more property of the sparse auto-encoder).\n\t* Despite its simplicity and generality, the method is shown to perform well on average, though it sometimes performs significantly worse than simple baselines.\n\n* Cons\n\t* The method itself is not explained very well. The authors use language such as “attach the auto encoder to the classifier”, which is a bit vague and could mean a number of things. It would be helpful if they provided either a formal definition of the model or a architectural diagram.\n\t* Though the quantitative evaluation is not entirely flattering, the authors should not be punished for providing experiments on many datasets. That said, if their contribution is then rather one of technical novelty, i.e. a sparse-autoencoder-based framework for time series interpretability, it would be helpful for them to \n\t\t* More formally define their framework / class of solutions\n\t\t* Provide a more in depth study of possible variants of the method (this is elaborated on in the “Questions” section)\n\t\t* More strongly argue the novelty of their method\n\t* The authors provide a discussion on automatic hyper-parameter tuning that seems a bit out of place in the main method section, since it is not mentioned much thereafter and is claimed to not bring benefits.\n\t* The qualitative evaluation made by authors is rather vague:\n\t\t* \"Alongside the numbers, TSInsight was also able to produce the most plausible explanations”\n\t\n* Additional Remarks\n\t* Why not train things jointly? Does this have to be done post-hoc? The authors state that they “should expect a drop in performance since the input distribution changes” -> so why not at least try fine-tune and study the effect of training the classifier with sparse representations end-to-end? Exploring whether things can be trained jointly, or in other configurations, might allow the authors to frame their work as more of a general technical contribution.\n\t* It would be nice to have the simple baseline of a classifier with a sparsity constraint, i.e. \n\t\t* I.e. ablate the reconstruction loss\n\nI’ve given a reject because 1) the explanation of the method is not very precise and could be greatly improved, 2) the quantitative evaluation is not sufficiently convincing, given the lack of technical novelty), and 3) the qualitative evaluation is hand-wavy. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #2 summarizes it well:

The aim of this work is to improve interpretability in time series prediction. To do so, they propose to use a relatively post-hoc procedure which learns a sparse representation informed by gradients of the prediction objective under a trained model. In particular, given a trained next-step classifier, they propose to train a sparse autoencoder with a combined objective of reconstruction and classification performance (while keeping the classifier fixed), so as to expose which features are useful for time series prediction.  Sparsity, and sparse auto-encoders, have been widely used for the end of interpretability. In this sense, the crux of the approach is very well motivated by the literature.

--

Discussion:

All reviews had difficulties understanding the significance and novelty, which appears to have in large part arisen from the original submission not having sufficiently contextualized the motivation and strengths of the approach (especially for readers not already specialized in this exact subarea).

--

Recommendation and justification:

The reviews are uniformly low, probably due to the above factors, and while the authors' revisions during the rebuttal period have improved the objections, there are so many strong submissions that it would be difficult to justify override the very low reviewer scores.
Decision: Reject