Objective Evaluation of Deep Visual Interpretations on Time Series Data
Authors: Christoffer Löffler
Wei-Cheng Lai
Lukas M Schmidt
Dario Zanca
Bjoern Eskofier
Christopher Mutschler
Venue: ICLR 2022 Submitted (2022)
Keywords: explainable ai
deep learning
time series
visual interpretation
evaluation metrics
classification
segmentation

Abstract: The correct interpretation and understanding of deep learning models is essential in many applications.
(Explanatory) visual interpretation approaches for image and natural language processing allow domain experts to validate and understand almost any deep learning model. However, they fall short when generalizing to arbitrary time series data that is less intuitive and more diverse. Whether a visualization explains the true reasoning or captures the real features is more difficult to judge. Hence, instead of blind trust we need an objective evaluation to obtain reliable quality metrics. This paper proposes a framework of six orthogonal quality metrics for gradient- or perturbation-based post-hoc visual interpretation methods designed for time series classification and segmentation tasks. This comprehensive set is either based on "human perception" or on "functional properties". An extensive experimental study includes commonly used neural network architectures for time series and nine visual interpretation methods. We evaluate the visual interpretation methods with diverse datasets from the UCR repository as well another complex real-world dataset. We show that none of the existing methods consistently outperforms any of the others on all metrics while some of them are ahead in either functional or human-based metrics. Our results allow experts to make an informed choice of suitable visualization techniques for the model and task at hand.

Under review as a conference paper at ICLR 2022
OBJECTIVE EVALUATION OF DEEP VISUAL INTERPRETATIONS ON TIME SERIES DATA
Anonymous authors
Paper under double-blind review
ABSTRACT
The correct interpretation and understanding of deep learning models is essential in many applications. (Explanatory) visual interpretation approaches for image and natural language processing allow domain experts to validate and understand almost any deep learning model. However, they fall short when generalizing to arbitrary time series data that is less intuitive and more diverse. Whether a visual- ization explains the true reasoning or captures the real features is more difficult to judge. Hence, instead of blind trust we need an objective evaluation to obtain re- liable quality metrics. This paper proposes framework of six orthogonal quality metrics for gradient- or perturbation- based post-hoc visual interpretation methods designed for time series classification and segmentation tasks. This comprehen- sive set is either based on "human perception" or on "functional properties". An extensive experimental study includes commonly used neural network architec- tures for time series and nine visual interpretation methods. We evaluate the visual interpretation methods with diverse datasets from the UCR repository and another complex real-world dataset. We show that none of the methods consistently out- performs any of the others on all metrics while some of them are ahead in either functional or human- based metrics, Our results allow experts to make an informed choice of suitable visualization techniques for the model and task at hand.
INTRODUCTION
Due to its high performance complex multi-modal data, deep learning (DL) becomes increas- ingly popular in many real-world applications that process time series data (Fawaz et al., 20096). While we fundamentally rely their classification accuracy in many salety-relevant applica- tions (Berkenkamp et al. 2017) they remain difficult to interpret. Typical applications are the mon- itoring of industrial processes (Löffler al., 2021). the support of health care and sports (Dorschky et al. 2020), or safety in autonomous driving (Schmidt et al. 2021). The need for improved model understanding (Carvalho et al., 2019), along with regulatory guidelines (Goodman & Flaxman 2017). led to a myriad of new approaches to the visual interpretation problem (Zhang & Zhu 2018)
Post-hoe visual interpretation allows domain expert to validate and understand how (almost) arbi- trary deep learning models operate. Their central idea lies in highlighting features on the input that are "relevant" for the prediction of learned model (Adebayo et al. 2018). Many these techniques do not require a modification of the original model (Simonyan et al. 2014]] Ribeiro et al., 2016) and are compatible with different architectures, which makes them useful as a general-purposs validation tool for neural networks across different tasks (Arrieta et al. 2020).
However, while visual interpretation yields intuitive and correct explanations for images (Samek et al. 202T), the application of these methods on time series data is still an unsolved problem (Rojat et al. 2021) Time series inherently are more diverse (Rojat et al. (202T) (because they may originate from variety of different sensors and processes), and often do not allow obvious patch- or texture-based localization of critical features for human observers. This makes the application and the evaluation of visual interpretability methods difficult. A domain expert cannot easily judge if explanations are correct in (i) explaining the reason for the decision process in the DL model and in (ii) capturing the real features in the dataset that lead correct classification.
Hence, it is important not to blindly apply different visualization methods. This requires quality met- rics that evaluate visual interpretations and that enable an expert select suitable visualization for

---

Under review as a conference paper at ICLR 2022
given model and task at hand. However, both of-te-of-the-rr visualization techniques and metrics that evaluate visual interpretations (e.g. Pixel Flipping (Samek et al. (017), Sanity Check (Adebayo et al. 2018), and sensitivity checks (Rebuthi et al. 2020 have far only been examined im- ages (Rojat et al., 2021) or on NL.P tasks (Arras et al. (017). This lack of objective and subjective evaluation seriously limits the application and utility of them for time series.
In this paper, we propose to evaluate visual interpretations for time series combining six orthogonal metrics: "sanity" (Adebayo al., 2018). "faithfulness" (Alvarez Melis & Jaakkola 2018), "sensi- tivity" (Rebutti et al. (020), "robustness" (Yeh et al. 2019). "stability" (Fel & Vigouroux, 2020 Li
al., 2021). ) and novel metric based on human preferences: "localization" " These metries rate and validate distinct qualities of saliency. Our metrics are both based on the functional perspective, i.e.,
based on the model-specific operation, and on how well they represent annotators' semanties.
In an extensive evaluation, we train four different architectures on two different types of tasks: U- Time model (Perslev et al., (019)) and bidirectional Long Short-Term Memory (bi-LSTM) (Schuster & Paliwal 1997) segmentation tasks, and Fully Convolutional Network (FCN) (Long al. 2015) and Temporal Convolutional Network (TCN) (Bai et al. 2018) on classification tasks. We use di- verse datasets from the UCR repository (Dau et al. 2018) (GunPointAgeSpan. FordA, FordB, Blec- trieDevices, MelbournePedesstrian, NATOPS) and for segmentation the more complex real-world tool tracking dataset (Löffler et al. (02T). The experiments show the necessity of all categories to create an objective rating for methods, models and tasks, and allow domain experts to understand, rate, and validate saliency for time series in safety-critical applications.
The rest of the paper is structured as follows. Section 2 discusses background and related work. Section 3 introduces extended and novel metries for both the classification and segmentation task. Section discusses the experimental results and Section 5 proposes recommendations.
BACKGROUND AND RELATED WORK
Interpretation methods for DL models can be divided into ante-hoc methods, i.e., methods that are inherently part of the model, and post-hoc methods, i.e., methods that provide the interpretation after training (Rojat et al. 2021). We focus on post-hoe methods and divide them into madieet-basee and
perturbation- based methods (Li et al. 2021 Warnecke et al. 2020
Ismail al. 2020p.
Gradient-base methods compute the relevance for all input features by passing gradients backwards through the neural network (Ancona et al. 2018). Gradient (Simonyan et al.) 2014) computes class c's saliency map M using the derivative of the class score p of the model with respect to the input ap The gradient indicates the importance of points in the input sequence sample x. as M M=xx) = Oz
for predictions. The advantage of gradient-based methods lies in their computational efficiency, as they use only a small number of backward passes to compute (z).
Certurbation-based methods perturb known input samples and measure the effects of specific per- turbations on the predicted class via a forward pass through the network. For instance, Local Interpretable Model-Agnostic Explanations (LIME) (Ribeiro et al.] (016)) fits local surrogate model (e.g., a linear regression) as an explanation, and calculates relevance based on this surrogate. Perturbation-base methods are computationally expensive as they require multiple forward passes per sample. However, they do not need gradient information and work with black-box models.
2.1 METRICS FOR SALIENCY ON TIME SERIES
Most interpretation methods were originally designed for image or text data. Understanding and comparing visual interpretation intuitive on image data, compared to more abstract time series. Furthermore, the diversity of interpretation methods complicates
objective choice for the model and task (Rojat et al., 2021). For example, when Wang et al. (2017) 7) and Fawaz et al. 2019b) apply Class Activation Maps (CAM) (Zhou et al. 2016) well-known UCR datasets (Dau et al. 2018). they notice a qualitative difference of CAM interpretations between network architectures. Similarly, other work relies on domain experts that perform (costly) qualitative evaluation (Strodthoff & [Strodthoff], (2019] Fawaz et al.) 2019at Oviedo et al.) 2019). Hence, there is an increasing need for objective evaluation metrics make the interpretations measurable and comparable.
2

---

Under review as a conference paper at ICLR 2022
Acaroryy
goad
omt/ dxtance
Speat
opsr
Syord
509
130
(a) Sanity
(b) Faithfulness
(c) Sensitivity
(d) Robustness
(e) Stability
(l) Localization
Figure I: We show examples for a good (green) and bad (red) score for each metric. sane saliency depends on network parameters, tested by randomizing weights and biases. (b) Faithful saliency correlates with predictive accuracy, tested by perturbing the input sequence.
sensitive
saliency of the predicted class in one sample is different from others. { for robust saliency small changes to input data cause only small effects. (e) stable salieney of all class samples has low variance and standard deviation. (f) saliency should be localized on the predicted segment.
Metrics. There exists a large variety of different metrics. First, the stability of saliency maps can be evaluated with respect to the sensitivity of the majority class (Yeh al. 2019). This metric evaluates the change of interpretation when the input samples are attacked by adversarial noise. It is spe- cial form of perturbation, that was so far only applied image data. Second, Rebuffi al. (2020) measure class sensitivity by comparing saliency maps of min- and max-class. Third, Cho et al.) (2020) perturb the unimportant input samples and preserve only the important inputs. If the predic- tions are stable compared to the unperturbed input, the interpretation is robust. Similarly, Ates et al. (202T) evaluate the local robusiness (i.e., similar samples should lead to similar interpretations) of the interpretation by measuring the ratio of change of the interpretation compared to the amount of input perturbation.
faithfulnls a perturbation basiodd----dddd--- metric that aims to measure the rela- tionship between saliency and predictive features (Alvarez Melis & Jaakkola 2018). Schlegel et al. (019) focus on continuous univariate time series. Sub-sequences with high relevance values are perturbed either by swapping their samples replacing them with mean value. Similarly, [Imail et al. (2020) propose input perturbations to evaluate models on synthetic time series. As they perturb inputs point-wise instead of sub-sequences their metric does not consider time dependency between consecutive points. Our faithfulness metric perturbs sub-sequences of high saliency (Schlegel et al., 2019). We adapt each type of metric into our set to provide a visual interpretation of time series.
Categories of metrics. Metrics can be divided into different categories. depending on the ques- tion they answer. Doshi- Velez & Kim (2017) propose a distinction between human- -ronanng and functional metrics. The former involve human perception and intuition with the goal of generating qualitative, intuitive visualizations, e.g., bounding boxes for image object detection for testing the localization of saliency maps (Jianming et al. 2016), or questionnaires to indicate testers' opinions on the quality of explanations (Li et al. 2021). Functional metrics provide statistical measures, e.g., to aggregate performance metrics automatically (Rojat et al. 2021). and make use of proxy tasks to generate a quantitative evaluation. Another categorization (Li et al. 2021) identifies multiple broad categories for image data, that may transfer to time series, i.e., the faithfulness of salient features, the class sensitivity of an explanation, and the stability of explanations given noisy inputs.
This paper proposes framework of metrics with orthogonal categories, specifically time series. We adapt and extend metrics to (multivariate) time series and propose an intra-class stability metric and a concept of relevance localization: we build on the pointing game (Jianming al., 2016) and combine it with the precision and recall for time series framework (Tatbul et al. 2018). To our knowledge we are the first to apply and evaluate sanity check (Adebayo et al. 2018 on time series.
3 SCORING CATEGORIES
We propose a set of six distinct categories (sanity, faithfulness, sensitivity, robustness, stability, and localization) to assess visual interpretation methods and to determine their performance and trustworthiness in classification or segmentation tasks on time series. For each of them we propose a metric that enables a comparative evaluation of diverse types of visual interpretation methods.
3

---

Under review as a conference paper at ICLR 2022
Why do need six scoring categories? It seems tempting to rely on a single metric or on a single aggregated score across multiple metrics to assess the quality of visual interpretation. However, we show that the six presented categories are inher- ently orthogonal (see Fig. 2 and capture distinct qualities. In- terpretations depend on model parameters (sanity check (Adebayo ct al., 2018), predictive features (faithfulnees (Alvarez Melis & Jaakkola 2018)), coherence of class predictions (intra-class stabil- ity (Fel & Vigouroux 2020), robustness against noise (max sensi- tivity with adversarial noise (Yeh et al., 2019)), specific sequences (or even points) in a time series like by Tatbul et al. (2018) (novel localization metric), and the relevance map's s specificity (inter- class sensitivity (Rebufi et al., 2020). It is important to assess if a given interpretation accurately captures these dependencies.
62
4:37
Ful Me 2.V
974
831
4.24
-22
0.00
C424
Figure 2: Pairwise Pearson correlations of scores (also plotted in Fig. 7b). Every met- ric is independent of all others.
We give a brief overview of our frameworks' functional (Fig. Tafe) and human- grounded metrics (Fig. Tf) that we evaluate with time series data. (a) sane saliency depends network parameters and is structurally different after randomizing the network's weights p; for layers [1. 2. 3| (Adebayo
eet al. 2018). ( faithful saliency correlates with predictive accuracy and perturbing percentage
of the input sequence with high saliency decreases accuracy (Alvarez Melis & Jaakkola 2018). (c)
saliency is sensitive when the predicted (max) class in one sample sufficiently different from any other (min) class (Rebuffi et al. 2020). (d) saliency is robust, if small changes to the input cause only small changes the saliency (Ych et al. 2019). (c) saliency is stable if has a low variance and standard deviation for all samples of class, with respect to suitable distance metric (Fel & Vigouroux 2020). f saliency should be localized on the predicted classes segment (to to t1). We will define those categories on time series after introducing a unified notation.
Notation. We follow the notation adapted from Fawaz et al. (011b): A multivariate time series is defined by X = [X- ....., "x. X#,, where H is the number of input channels, X = (ai...., x.y)) E R is an ordered set of real values, and T denotes the number of time steps. For H equal to 1, we consider univariate time series, otherwise we consider multivariate time series. Time series often include complex temporal dependencies, i.e., distinct points are not independent. Time series classification defines mapping X + y that minimizes the error where N is the number of data samples, X E D is a time series, y; E RC denotes the one-hot vector of class label the input belongs to, and C is the number of classes. In time series segmen- tation, we search X + Y that maps an input sample to a dense classification Y = [u,,-.., yr E RCTT (Perslev al., 2019), i.c., class label is predicted for each time step. Post-hoc visual in- terpretation methods compute a relevance map M% E RØ M (X) [R] ..... R"I, where rf),, representing the importance of each input feature with respect to the class e and model m, for each time step. We use M as a function to produce the saliency map. For clarity, we will omit the dependency on m, i.e., M = Ms, is not explicitly required. An evaluation metric for visual interpretation methods defines a score S-mmic() that rates the quality of the relevance map M at a sample X given a model m and optional parameters. We provide a unified view, so that for all scores, a higher score corresponds to better visualization according to the perspective.
dataset D = {(X,, V11 vi..... (XN UN }
R' =
3.1 SANITY
Intuitively, if the weights and biases of trained networks models were re-initialized with random values, the networks predictions and generated saliency maps should also be different from the original maps. However, this is not always the case. Despite drop in model accuracy, saliency may remain stable. Hence, we test sanity using a variant of sanity check (Adebayo et al. (2018). that performs layer-wise cascading randomization of the network's 's weights and biases, starting from the output to the input. In contrast to independent randomization, the cascading approach results in a mostly continuous performance degradation of predictions, see Fig. Ia for an illustration. Network accuracy should increasingly resemble random guessing. Following Adebayo et al.] (2018). we compare saliency using the structural similarity index measure (SSIM) (Wang al. 2004), that compares the distribution of sub-sequences of M We define the sanity score as
SSanity (M, D,m) - EXED
SSIM(MS, (X).MS (X))
L
4

---

Under review as a conference paper at ICLR 2022
where [D] = .,, enumerates the L layers of whose parameters are randomized. M (2) is the saliency map after randomizing layer i of m. We average the SSIM over layers and compute the average over all samples in D.
3.2 FAITHFULNESS
relevance measure is faithful, if input features with a high relevance (w.r.t. M99 have high influ- ence on the model prediction, see Fig. [b] an example. Alvarez Melis & Jaakkola (2018) propose a perturbation-eet based metric that evaluates the faithfulness of predictions. The metric measures the correlation between input (features) with high saliency on the one hand and predictive accuracy on the other hand. We choose a sub-sequence of the input to perturb by ranking each time step of Xr E E according to the relevance of its saliency map. Next, we perturb connected sub-sequences similar to Schlegel et al. (019)). We extend the metric to multivariate data X/I: Based on the rele- vance т, we select time points & E {I,...,TT T', and features the maximum relevance ri and set as the middle point the sub sequence. Therefore, we select and reverse the sequence Frev (x) ...., ..... 2
{I, H.. We select t according to
) to break the temporal correlation. Next,
we insert it back into copy of X that we denote as X'. With this perturbation we compute the mean faithfulness score over the whole dataset as
Spathfulnesss (M, D. m)
ExeD (X) mx')).
where m² R R the softmax prediction of the target class c. m mx() (X') is the gap score between the softmax prediction of the original and the perturbed sample.
3.3 INTER-CLASS SENSITIVITY
In multi-class prediction tasks the classifier needs to identify relevant features for each of the classes make a correct prediction. Hence, the relevance map M should identify different salient features for those different classes (Li et al. 2021), see Fig. Ic If method is not sensitive to the class, the saliency could be misleading, especially the classifier fails learn correct features for some classes. Inter-class sensitivity (Rebuffi et al., 2020) measures class specific sensitivity of the gen- crated relevance map with respect to the most (Cmax) and least (Cmin) likely class according to the model. We compute the mean inter-class sensitivity score
Sinter-Class Sensitivity ( (M, D) - -XXED sim CM Ciza (X), M (X)).
(3)
We compute similarity of two saliency maps as sim Fia Fime (I), Smir (I)) where .im is a simi- larity function (e.g. a cosine similarity) that easy to interpret via its geometric interpretation. is defined as the angle between two non-zero vectors that measures the similarity between their inner product space (Han al.] (2012). Similarity of M in binary classification would result in a negative cosine similarity, meaning nearly inverted saliency maps for max- and min-classes.
3.4 ROBUSTNESS
We evaluate method's robustness with respect to adversarial noise (Yeh et al., 2019) via its sensi- tivity of the most likely class (Cmax) see Fig. [T]] Intuitively, even with noisy inputs the saliency map of a model should not change significantly (Alvarez-Melis & Jaakkola 2018) and saliency maps with high sensitivity are less reliable. Yeh et al. (2019) define the sensitivity of the saliency map derived from the gradient as
(X)) lim M=(tee)))-MxX)
(
for any j E (1,..,, H TI}. where EXT is the j-th coordinate vector and the j-th entry is one while the others are zero. We use Monte Carlo sampling of eej, where <eyl<a (a a user-specified radius), to generate different X = X + eej. We compare X with the original X to compute
SMux Sensitivity (M', D,a)= - max x x MM M=(X)I-
(5)
5

---

Under review as a conference paper at ICLR 2022
3.5 ANTRA-CLASS STABILITY
A method is stable if similar input samples from the same class produce similar interpretations (Fel
& Vigouroux 2020). We implement this concept test the intra-class stability of saliency maps Mc for a given dataset D using distance statistics, see Fig. Ic for an example. To this end, we compute pairwise distances between saliency maps M for different samples X,, D. Then, we aggregate these distances for all samples from one class:
Sintr-Claass Stability (M . D) = -E EJE(HINN) das (M..).).MM(..)))
(6)
N-(N-1)
We use Dynamic Time Warping Viintyykk 1968) as the distance function dprw. The score compares each classes' sample's M with all other samples' saliency maps in the dataset, using the distance dprw. summary, the mean distance for class should be small and the variance low.
3.6 LOCALIZATION
The temporal location of class-specific features with high relevance in time series segmentation task should be situated within (or close-by) labeled segment (temporal sub-sequence), see Fig. [T Due to its connection to the annotation task call it human- human--gouudde grounded metric. Specifically, propose a novel metric that adapts Pointing Game (Jianming et al.) (016) used for object detection to time series using range-based metries (Tatbul et al.) 2018). We argue that Pointing Game's ' original hit-and-miss accuracy is inadequate for time-series methods, as Segmentation is point-wise exact. Thus, replace it with sange-based testing, yielding Localization. Furthermore, some saliency methods can show strong temporal biases towards the beginning, middle or end of segmented class. We would expect high saliency for features within the predicted segment, and low saliency otherwise, to accurately capture the model's predictive behavior. Hence, Localization measures agreement between annotators, models' predictions and saliency, and can discover issues like temporal biases and imprecise margins.
observed that
First, we filter out non- non-relevant predictions (according to the relevance map) as follows. prediction time I, denoted with (1), is relevant if there exists an i E {1, M) such that rf > max((r)) 0. We select the model's predicted class at time t if the prediction is relevant, otherwise we set the class to none. The resulting relevancy-fiterred prediction is denoted Y', Finally, we can compare Y' with the ground truth Y and evaluate how well high saliency features lie within the annotated sebbsequences (Tatbul et al. 2018).
We compute the mean localization score for the whole dataset D. We compare any of the Nuub existing labeled sub-sequence Yub. R from D each with its temporally co-located prediction Y
Stocalization (Y, "' Ein
Nut (aa, Nus
For each pair (Ysub,. Y we calculate range-based recall score based on the point-wise comparison proposed by Tatbul et al. (2018)
Sieaal (Ysab: Y')
existence(Y.u. Y") (1-a) overlap(Y Y"),
where weighs the two reward terms for "existence" and "overlap". Existence is 1 if any time point was correctly predicted within the labeled region, and otherwise. The parameterized "overlap" function determines the finer properties of cardinality, size and position. The cardinality parameter discounts the score if the prediction is an interrupted, fragment range instead of being continuous. The overall size of the overlap of predicted and label ranges depends on a positional bias. It may favor "front", "middle", or "back" overlap. Practically, for some applications an early detection is preferable over late detection. For further details see Appendix or Tatbul et al. (2018).
EXPERIMENTS
Our evaluation compares visual interpretation methods on set of network architectures and diverse classification datasets from the UCR repository, and on a more complex segmentation dataset. In total, aggregate the results from 540 experiments. This section is divided into two parts. We first discuss the new localization metric for the segmentation task. Next, we discuss the metrics for the classification task (faithfulness, sensitivity, stability, robustness, sanity).

---

Under review as a conference paper at ICLR 2022
4.1 EXPERIMENTAL SETUP
We evaluate nine visual interpretation methods: Gradient (Simonyan al. 2014), Integrated Gradi-
ent (Sundararajan et al. 2017). SmoothGrad (Smilkov et al. 2017). Guided Backpropagation (Sprin-
genberg et al. | 2013)) GradCAM [Selvaraju et al. (017)) 7). Guided GradCAM (Selvaraju et al. 2017).
Layer- Wise Relevance Propagation (LRP) (Bach et al. 2015), LIME (Ribeiro et al.) 2016) and Ker-
nel SHAP (Lundberg & Lee
2017). See Appendix B.2 for method-specific hyper parameters.
For the segmentation task, consider two models: U-time (Perslev al.. 2019)) (derived for time
series from U-Net (Ronneberger et al., 2015)), and a bi-LSTM (Schuster & Paliwal 1997). We pro- vide details on each architecture in Appendix [33] We evaluate them the tool tracking (Löffeer itt al.) (222I) dataset, a complex multivariate, multi-class time series from a 9-D magneto-inerial sensor, and use the electric screwdriver's data. For the classification task, we select two com- monly used model architectures, i.e, a Fully Convolutional Network (FCN) (Long et al. 2013) and a Temporal Convolutional Network (TCN) (Bai et al., al., 2018). We do not focus on simpler architec- tures such as Multilayer Perceptrons and LSTMs (Hochreiter & Schmidhuber 1997) due to noisy or vanishing saliency in preliminary studies. Our dataset selection follows related work (Fawaz) ee al. 2019b [Wang et al. [017]] Schlegel et al. 2019; Ates et al. (0011) that uses large, multivari- atc, multi-class datasets from diverse domains (GunPointAgeSpun, FordA, FordB, ElectricDevices, Melbournceedestrian and NATOPS from the UCR repository (Dau al. 2018)).
4.2 EVALUATION ON SEGMENTATION TASK
This section presents the key findings from our evaluation when applying the localization metric on saliency maps generated from U-time (results for bi-LSTM are in Appendix C.T), trained for segmentation task on the tool tracking dataset. Fig. Bal shows the localization metrics' positional biases "classic", "front", "middle" and "back" for each visualization method. Fig. 3h shows an exemplary saliency map for each method.
GradCAM clearly outperforms the other methods on the localization metric. It's saliency maps are smoother, with high relevance located on the annotated seg- ments. This may result from its coarser relevance, as its gradients do not flow to the input but stop at the last convolution layer. In contrast, LRP and Integrated Gradient perform worst. As Fig. 3b shows, the saliency maps highlight fewer relevant input features for the segmenta- tion task, but instead features within the labeled sub-sequence that may be impor- tant for classification. Saliency maps are located better in the "middle" of the annotated segments, compared "front"
Irtegrited
Sregein
Viogreied Chuckers
Guises
Gallec GrandcM
Kamel
100
354
0.0
Time [mis]
(a) U-time localization.
(b) U-time examples.
Figure 3: Results for the localization metric on the tool tracking dataset and with all methods on U-time. We show detailed values for the localization metric in (a) and examples saliency maps in
and "back", see Fig. 3a This is also visible in the example in Fig. 3b where the relevance values start decreasing below the threshold of 0.5 already before the end of the labeled segment at 100 ms.
This points towards a biased labeling process with longer sub-sequences assigned to the classes than necessary, and can help identifying faulty labels.
Furthermore, the high variance of most results (except for GradCAM) reveals that saliency maps for time series segmentation are noisy. This especially the case for Kernel SHAP, see Fig. (bb where relevance is assigned scemingly at random. In summary, GradCAM performs best with respect to the numan-groundee low variance in the results and focus the "middle" segment, while other methods hardly align with annotated relevant segments in the input features or their saliency maps are noisy.
metric of localization, compared to all other methods in this study. exhibits
Conversely, no method works reliably the bi-LSTM. We identify lack of useful visualization methods for LSTMs, as CAM variants (GradCAM, Guided GradCAM. and Guided Backprop) are not applicable, see Fig. 6a and 6b in Appendix C.1 for detailed results.
7

---

Under review as a conference paper at ICLR 2022
4.3 EVALUATION ON CLASSIHICATION TASK
In this section, we discuss the five visual interpretation quality metrics for the classification task. Fig. 1as.4ajsnn summarizess the scores for each metric over model architectures (FCN, TCN) and datasets.
Sanity. Across all datasets, we find that Gradient, Integrated Gradients, LIME, and Kemel-SHAP consistently achieve high sanity scores. In contrast, LRP, Guided GradCAM and Guided Backprop- agation often assign similar saliency even after randomizing network parameters. This replicates sanity results by Adebayo et al. 2018). Low sanity methods highlight features in the input that may also be randomly extracted but do not necessarily predict classes.
Faithfulness. The scores differ widely across datasets. While Gradient, Integrated Gradients, LRP and Guided Backprop have an above-average faithfulness, Guided Gradcam and GradCAM are able to identify critical features for a prediction. Especially scores for FordA, FordB and NATOPS are generally very low across all visualization methods. Perturbating connected sub-sequences in- stead of random samples may be less suitable for these datasets.
Sensitivity. GradCAM shows the highest sensitivity for classes the salieney maps for the most and least likely classes differ largely, On the other hand, Integrated Gradient and Guided Back- propagation produce similar saliency maps, independent of sample's class. This holds across all datasets.
Robustness. As found by Smilkov et al. (017) and us, Gradient is less robust to (adversarial) noisy. LIME and, interestingly, SmoothGrads also have a low average robustness. The additive noise of SmoothGrads may compound and lead to diverging saliency maps. The metric's weakness is computational cost (TCN, Flec,,ElectricDevvice, timed out after 12 days.
Intra-class Stability. This metric is highly dependent on the dataset, e.g., centered samples as in GunPointAgeSpan. However, when accounting for variance introduced by the dataset (see 7bg Appendix C.2p. SmoothGrads, LIME, and Kernel-SHAP show unstable saliency maps for the same classes. Guided GradCAM is able to produce highly stable saliency maps across all datasets.
Dataset Influence. First, datasets heavily impact scores, see the outliers in Fig. 4a Plotting the categories over each dataset for all (model architectures, methods} (Fig. 4b) emphasizes this finding. The choice of visual interpretation method depends, for the most part, on the task. Still, after normalizing for dataset bias, the relative scores do not diverge significantly but confirm the initial scores. We report the relative scores in Fig. 76 in Appendix C.2 Second, the model architecture (FCN or TCN) matters little to the scores, see Fig. [7] Appendix C.2] The scores for FCN and TCN diverge slightly more for faithfulness. This may be due to the higher capacity of the TCN, which learns more features of lower relevance, so that accuracy is not as affected by perturbations as for the smaller FCN. A larger perturbation percentage may resolve this issue.
Santy
Specticly
Rebudress
Casc Salll
lat Br3ppt
(a) Scores for all nine methods aggregated over (model architectures, datasets).
Seratiity
Feus Clas Baxily
(b) Datasets' impact scores, aggregated over {model architectures, methods].
Figure 4: (ap: shows aggregated results for TCN and FCN over all classification datasets without normalization as we found that model architectures generally lead to similar scores); (b): shows the influence of datasets on the scores, when we aggregate for each dataset over {model architectures, methods). Similarity metrics for saliency maps yield different results in different domains.
8

---

Under review as a conference paper at ICLR 2022
5 RECOMMENDATIONS
The quality of visualizations differs wildly between methods. No method passes the tests for all categories on all six datasets. This emphasizes the need to evaluate all metrics for every visual inter- pretation. We recommend use a summary as in Fig. 5 to judge visualizations on every category, and propose the following guidelines for relative ranking. The absolute scores may be understood in comparison with a random baseline, similar to shuffled AUC (Borji et al. 2013).
First: Ensure Faithfulness and Sanity. The gen- eral purpose of interpretability methods is to pro- vide insights into model behavior. We propose to use the Faithfulness and Sanity scores to ascertain that a saliency map represents the model behavior. Faithfulness ensures that the saliency matches the model's 's predictive features. Sanity checks confirm that saliency maps are sensitive to model parameters. This is important avoid finding highly salient fea- tures, like edges in images (Adebayo et al., 2018). while being insensitive to model parameters. We avoid saliency maps with low scores in either metric. See Fig- for an example: Gradient, IG and LIME achieve the three highest scores. Random saliency maps are dissimilar, hence the high Sanity score. However, its Faithfulness hints at low reliability of the other methods. Guided Backprop. Guided GradCAM and LRP fail the Sanity Check, while Kerne-SHHAP performs poorly in Faithfulness.
0.21
0:22
0.74
6:75
0.24
6:25
9.04
0.57
0.55
0.00
6:67
2.46
0.50
2.20
0.33
6:90
0.50
6:90
6.52
6.77
9.30
6:24
IC
0.24
0:95
0.70
0:23
0.52
0.55
9.00
Figure 5: Scores as a heatmap for TCN on FordB. Columns with high relative scores (bright squares) indicate good visualizations. Dark squares show the shortcomings.
Second: Check Sensitivity and Robustness. Once Faithfulness and Sanity are established, we propose to look at Sensitivity and Robustness of the generated saliency maps. A low Inter-Class Sensitivity can indicate that the saliency maps only focus the predicted class and underestimate the importance of features that do not belong to this class. A low Robustness score suggests that the visualization method is susceptible to adversarial examples and small perturbations in the input. Given that the model predictions are robust (Zhang et al. 2019), and the saliency is faithful, a low Robustness implies that the saliency method cannot be trusted, and may even be manipulated (Dom- browski et al. 2019). For non-robust models we recommend Faithfulness. According to Fig. 5 we keep Gradient and LIME, but disregard IG, due its lower Sensitivity.
Detailed Analysis: Analyze Intra-class Stability and Localization. Intra-class stability measures how much saliency maps for one class agree between different samples. The localization metric, like other Pointing Game-like metrics, measures semantic precision based on human annotations. These human-grounded metrics allow an expert to filter visualizations that are more intuitive and understandable. Note that it is crucial to ensure the Faithfulness of method before relying on this metric. If the method is not faithful, stable and localized visualizations are visually pleasing, but do not reflect model behavior. This can hide issues like spurious correlations (Arjovsky et al. 2019) or the shortcut learning problem (Geirhos et al. 2020) behind a good score.
6 CONCLUSION
This paper proposes a framework of six orthogonal metrics for the objective evaluation of visual interpretation methods for time series classification or segmentation. All these metrics should be evaluated for every visual interpretation to prevent a reliance on interesting, but spurious results. We also show that each perspective emphasizes different strengths and weaknesses of visualization methods. We show empirically for six datasets and different model architectures that none of nine state-of-the--rt visual interpretability methods passes all tests. We propose to use our framework to guide the selection of visual interpretations and to understand their weaknesses. Further, we found that datasets highly influence the quality of visualizations. Our extended and novel metrics provide important information, like the localization metric for time series. Interestingly, on the segmentation task only one visual interpretation method passes our novel evaluation metric. None of the saliency maps for LSTMs produce satisfactory scores. Future work combines range-based and functional metrics, or optimizes models specifically to achieve higher interpretability scores.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
This paper includes supplemental materials to improve reproducibility. Most importantly we pub- lish the code for model training and dataset loaders, for generating visual interpretations and of all evaluation metries. Furthermore, all datasets are publicly available.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in Neural Information Processing Systems, 2018.
David Alvarez-Melis and Tommi S. Jaakkola. On the robustness of interpretability methods. In Proceedings of the 2018 ICML Workshop in Human Interpretability Machine Learning, Stock- holm, Sweden, 2018.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez- Paz. Invariant risk minimization.
arXiv preprint arXiv: /907.022990000 2019.
Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Explaining recur- rent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 159-168, Copenhagen, Denmark, September 2017.
Alejandro Barredo Arrieta, Natalia Diaz- Rodriguz, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Gareía, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunitics and challenges to- ward responsible ai. Information Fusion, 58:82-115 113.111,, 2020.
Emre Ates, Burak Aksar, Vitus Leung, and Ayse K. Coskun. Counterfactual explanations for mul- tivariate time series. In 2021 International Conference on Applied Artificial Intelligence (ICAPAI), pp. 1-8, Halden, Norway, May 2021.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robae Müller, and Wojeiech Samek. On wixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation. PLOS ONE, 10(7):1-46, 07 2015. doi: 10. 10.o37//j pon. .01 0130140,
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprini arXiv: 1803.01271, 2018.
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. In Proc. of the 31st Intl. Conf. on Neural Infor- mation Processing Systems, pp. 908-919, 2017.
Ali Borji, Hamed R. Tavakoli, Dicky N. Sihite, and Laurent Itti. Analysis of scores, datasets, and models in visual saliency prediction. In Proceedings of the IEEE International Conference Computer Vision (ICCV), December 2013.
Diogo V. Carvalho, Eduardo M. Pereira, and Jaime S. Cardoso. Machine learning interpretability: A survey on methods and metries. Electronics, 888):33,, Jul 2019. ISSN 2079-9292.
Sohee Cho, Ginkyeng Lee, Wonjoon Chang, and Jaesik Choi. Interpretation of deep tempo- ral representations by selective visualization of internally activated nodes. arXiv preprint arXiv:2004.1255 1253, 2020.
H. Dau, A. Bagnall, K. Kamgar, M. Yeh, Zhu, S. Gharghabi, and C. Ratanamahatana. The UCR time series archive. ArXiv e-prints, 110.0:1810.07758. 2018.
10

---

Under review as a conference paper at ICLR 2022
Ann- KathKKn Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, 2019.
Eva Dorschky, Marlies Nitschke, Christine Martindale, Antonie J. van den Bogert, Anne Koelewijn, and Bjöm Eskofier. CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data. Frontiers in Bioengineering and Biotechnol- ogy. 8:1-14, 2020.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint
2017.
Hassan Ismail Fawaz, G. Forestier, Jonathan Weber, L. Idoumghar, and Pierre-Alain Muller. Accu- rate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks. International Journal of Computer Assisted Radiology and Surgery, 14:1611- 1617, 2019a.
Ismael Hassan Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification; review. Data Mining and Knowledge Discovery, 33, 07 2019b.
Thomas Fel and David Vigouroux. Representativity and consistency measures for deep neural net- work explanations. arXiv preprint arXiv:2009.0452/, 2020.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-677 2020.
Bryce Goodman and Seth Flaxman. European union regulations on algorithmic decision making and a "right to explanation". Al magazine, (8(3):50-57, 2017.
Jiawei Han, Micheline Kamber, and Jian Pei. 2 getting to know your data. In Data Mining (Third Edition), The Morgan Kaufmann Series in Data Management Systems, pp. 39-82 Boston, 2012.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):
1735-1780, 1997.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by re- ducing internal covariate shift. In Proc. of the 32nd Intl. Conf. on Machine Learning, pp. 448-456, 2015.
Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. Benchmark- ing deep learning interpretability in time series predictions. In Advances in Neural Information Processing Systems, volume 33. pp. 6441-6452, 2020.
Zhang Jianming, Lin Zhe, Brandt Jonathan, Shen Xiaohui, and Sclaroff Stan. Top-down neural attention by excitation backprop. In European Conference on Computer Vision(ECCVV, 2016.
Xiao-Hui Li, Yuhan Shi, Haoyang Li, Wei Bai, Caleb Chen Cao, and Lei Chen. An experimental study of quantitative evaluations on saliency methods. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 3200-3208 2021.
Christoffer Löffler, Christian Nickel, Christopher Sobel, Daniel Dzibela, Jonathan Braat, Benjamin Gruhler, Philipp Woller, Nicolas Witt, and Christopher Mutschler. Automated quality assurance for held tools via embedded classification and automl. In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track, pp. 532-535, 2021.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems, volume 30, 2017.
11

---

Under review as a conference paper at ICLR 2022
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 1(10):e3, 2016.
Felipe Oviedo, Zekun Ren, Shijing Sun, Charles Settens, Zhe Liu, Noor Titan Putri Hartono, Savitha Ramasamy, Brian L DeCost, Siyu IP Tian, Giuseppe Romano, et al. Fast and interpretable clas- sification of small x-ray diffraction datasets using data augmentation and deep neural networks. npj Computational Materials, 5(1):1-9, 2019.
Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jø rgen Jennum, and Christian Igel. U-time: A fully convolutional network for time series segmentation applied to sleep staging, In Advances in Neural Information Processing Systems 32, pp. 4415 4426. Curran Associates, Inc., 2019.
Sylvestre-Alvise Rebuffi, Ruth Fong, Xu Ji, and Andrea Vedaldi. There and back again: Revisiting backpropagation saliency methods. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proc. of the 22nd ACM SIGKDD Insl. Conf. on Knowledge Discovery and Data Mining, pp. 1135-1144, 2016.
Thomas Rojat, Raphaël Puget, David Filliat, Javier Del Ser, Rodolphe Gelin, and Natalia Díaz- Rodríguez. Explainable artificial intelligence (xai) on timeseries data: A survey. arXiv preprint arXiv:2/04.00950, 2021.
O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image seg- mentation. In Medical Image Computing and Compuler-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234-241. Springer, 2015.
Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. JEEE Transac- tions on Neural Networks and Learning Systems, 28:2660-208 2673, 11 2017. doi: 10. 10.N1SSS 2599.20
Wojeiech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher Anders, and Klaus- Robert Müller. Explaining deep neural networks and beyond: A review of methods and applica- tions. Proc. of the IEEE, 109(3)2277-77,, 2021.
Udo Schlegel, Hiba Arnout, Mennatallah El-Assady, D. Oclke, and D. Keim. Towards a rigorous evaluation of xai methods on time series. IEEE/CVF Intl. Conf. on Computer Vision Workshop (ICCVW), pp. 4197-4201, 2019.
Lukas Michael Schmidt, Georgios Kontes, Axel Plinge, and Christopher Mutschler. Can you trust your autonomous car? interpretable and verifiably safe reinforcement learning. In IEEE Intelli- gent Vehicles Symposium (IV), pp. 171-178, 2021.
M. Schuster and K.K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(1 45611)12633-2611 1997.
Ramprasaath Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based based local- ization. In IEEE Intl. Conf. on Computer Vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:/312.603: 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 002177766..82.. 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. Striving for simplicity: The all convolutional net. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Nils Strodthoff and Claas Strodthoff. Detecting and interpreting myocardial infarction using fully convolutional neural networks. Physiological Measurement, 40(1 40(1):01 5001)0050011, jan 2019.
12

---

Under review as a conference paper at ICLR 2022
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pp. 3319-3328, PMLR, 2017.
Nesime Tatbul, Tae Jun Lee, Stan Zdonik, Mejbah Alam, and Justin Gottschlich. Precision and recall for time series. In Advances in Neural Information Processing Systems, volume 31, 2018.
Taras K Vintsyuk. Speech discrimination by dynamic programming. Cybernetics, 4(1):52-57, 1968.
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In Intl. Joint Conf. on Neural Networks, pp. 1578-1585, 2017.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4):600-612, 2004.
Alexander Warnecke, Daniel Arp, Christian Wressnegger, and Konrad Rieck. Evaluating expla- nation methods for deep learning in security. In JEEE European Symposium on Security and Privacy, EuroS&P, pp. 174, 2020.
Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in)fidelity and sensitivity of explanations. In Advances in Neural Information Processing Systems, volume 32, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael Jor- dan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7472 7482. PMLR, 2019.
Quanshi Zhang and Song-Chun Zhu. Visual interpretability for deep learning: a survey, Frontiers Inf. Technol. Electron. Eng.. 19(1):27-39, 2018
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
A LOCALIZATION METRIC POSITIONAL BIASES
This section explains the "existence" and "overlap" terms of our localization metric. Note again the recall score
Specat (Ysub; Y') existence(Y,us Y") + (1 aa) overlap((ysbb Y'),
(9)
The "existence" and "overlap" terms are defined by Tatbul et al. 2018)) as follows. First, existence is defined as a correct prediction of one sample point at index the correct class within the labeled region:
existence(Y..b, Y") [1, if E Yuu n nYjI 0, otherwise
(10)
The overlap determines the finer properties cardinality 7(), size w() and position 6().
overlap(Ysut. Y") cardinality (Y.b. Y'). Σ WYaub, Yub n YY,8)
(11)
1, cardinality(Y Y') = (Y Y") otherwise
Yub overlaps with at most one Y E y
(12)
The term a (0 < < 1) weights existence and the following qualitative measures overlap.
The three user-defined functions return values of 0 7() < 1, 0 w() < and 8() 2 The
13

---

Under review as a conference paper at ICLR 2022
cardinality 7( function weighs the prediction of continuous range Y' covering the whole labeled range Yub versus interrupted ranges a fragment manner. The overall size w() of the agreement of predicted and label ranges depends the positional bias 6() and can favor "front", "middle", or "back". Practically, for some applications an early detection is preferable to a late detection. For the four functions, selected is equal to 0, y is equal to 1. depends the position bias and for ó we have three variants, "front", "middle" or "back", proposed by Tatbul et al. (2018).
B EXPERIMENTAL SETUP
We describe each visual interpretation method shortly in Sec. B.I. then the hyper parameters for each visual interpretation method Sec. B.2 and for the optimizer and networks Sec. B.3
B. CATEGORIZATION OF METHODS
This Section provides a short introduction the methods that we use in our experiments.
Gradient-based methods. Gradient (Simonyan et al., 2014) computes class c's saliency map M using the derivative of the class score p of model with respect to the input sample 2, as ar (=)x)= = 3x However, Gradient suffers from the saturation problem (a feature may have global importance, but its local derivative small (Smilkov et al., (017)). local sensitivity, and noisy saliency maps (due to (sharp) local variations in the gradients (Smilkov et al., (017p). Follow-up work smooths the gradients to reduce noise (Smilkov et al. (017), applies special propagation rules instead of propagating gradient (Arras et al. 2017), or propagates only up to specific intermediate layer (Selvaraju et al. 2017).
Perturbation-basel methods. Local Interpretable (LIME) (Ribeiro et al., 2016) fits a local surrogate model (e.g., linear regression) as an ex- planation and uses this surrogate to calculate relevance. Kernel SHAP (Lundberg & Lee (017) builds on LIME, but calculates Shapley values that measure the contribution of individual features to the input more accurately.
Model-Agnostic Explanations
B.2 VISUALIZATION METHODS HYPER PARAMETERS
We set the hyper parameters for each visual interpretation method according to their recommen- dations from literature. We provide the reasoning behind the selected parameters in the following paragraphs.
Gradient computes the derivative the target class score with respect the input sample and returns the saliency map of the input sample at the end. Integrated Gradient uses the linear path method to compute gradient along the path from a baseline x'. We use zero vector for the baseline. As suggested by Smilkov et al. (2017). the number of steps for the path should be selected between 20 and 300. Hence, we use steps N 60, meaning that it takes steps from baseline x' to the original input sample x, according to x = x' (=-x') n. is the current step.
N
SmoothGrad also computes the derivative of the target class score with respect to the input sample. However, it adds Gaussian noise NN0 o²) to the input sample multiple times and computes the gradients from the perturbed samples (ar + oO) o²)). The number of iteration adding noise 0.2.
chosen N 60 and the standard deviation of Gaussian noise is chosen
For LRP, we select the e-propagation rule for every DI. model with = le Due to the residual block in TCN model the propagated relevances should therefore be added together (R R₁ R2).
GradCAM and Guided-GradCAM are designed for models with convolutional layers, i.c. FCN, TCN and U-time. GradCAM focuses on the last convolutional layer, which produces feature maps, whose shapes usually are smaller than the shape of input samples. Therefore, we use interpolation to up-sample the saliency maps from the last convolution layer match their shape to the shape of the input samples, which allows us to visualize them in the input space. Because of ReLU functions, GradCAM returns only positive relevances.
14

---

Under review as a conference paper at ICLR 2022
For LIME, we use the cosine distance function as the kernel function with width w 5.0 weight the perturbed samples, and perform 1000 iterations. For the perturbation, we consider that neighbors along time dimension should have similar relevance to reduce the computational time, number of features along time dimension 50 for GunPointAgeSpan. This means that saliency maps for samples with length 150 in GunPointAgeSpan have same relevance for every 3 neighhors and saliency maps for dense labeling samples in tool tracking have same relevance for every neighbors.
set the
For Kernel SHAP, we use 1000 iterations and set the number of features along time dimension to 50. Furthermore, the sampling of feature perturbation in Kernel SHAP is based on the distribution
p(f) (F-1) [F}}3 where f is the number of selected features and F is the total number of features
in interpretation space.
B.3 NETWORK ARCHITECTURES
We use the Adam optimizer with a learning rate of 0.002. We train every dataset for 600 epochs (with early stopping after 80 epochs). We use Cross Entropy loss for time series classification and a Generalized Dice loss with Cross Entropy function for time series segmentation.
For the time series segmentation task on the tool tracking dataset, we report an accuracy of 83% for U-time and 85% for the bi-LSTM. Table 1 shows the classification results for FCN and TCN with all classification datasets.
model architectures
datasets
FCN
TCN
GunPointAgeSpan
98.73
97.15
FordA
89.77
91.67
FordB
79.01
80.37
MelbournePedestee
90.13
95.17
NATOPS
97.22
95.56
ElectrieDevices 69.51
69.47
Table 1: The test accuracy in % of the classification task.
B.3.1 FULLY CONVOLUTION NETWORK
We use a slightly modified FCN, similar to Wang et al. (0117). Ours contains four convolution blocks with a convolutional layer, a batch normalization layer (loffe & Szegedy 2013) and a ReLU layer in each block. The kernel shapes and numbers of filter for convolution layers are {7, 5, 3, 3} and {16, 32, 32, 16). Therefore, there are four convolutional layers. Each convolutional layer has unit stride and padding, which means the time sequence will be reduced continuously by the blocks. The final convolution block, which is behind the four convolution blocks, does not have a ReLU layer and contains Ixl convolutional layer. The Ix1 convolutional layer serves as projection layer. It can not only reduce the channel size of feature maps but also keep their salient features. Finally, we apply Global Max Pooling on the features maps, before the softmax operation.
B.3.2 TEMPORAL CONVOLUTION NETWORK
For the TCN, that was first proposed by Bai et al. 2018), we use a global pooling layer for the prediction. In our architecture, TCN has the convolution filters {16, 32, 32, 32} and kernel shapes for convolution layers {7,5, 5,5,5 5} in four residual blocks. Therefore, the total number of layers is 8.
B.3.3 BIDIRECTIONAL LONG SHORT-TERM MEMORY
We use standard, single-layer bi-LSTM to predict dense labels for the segmentation task, as imple- mented in PyTorch, with 512 hidden units. There is a dense layer behind the LSTM model to fit the hidden units 512 to the number of classes. Also, the dropout rate is set to 0.2 to prevent overfitting.
15

---

Under review as a conference paper at ICLR 2022
B.3.4 U-TIME
We use the U-time (Perslev et al., 2019) architecture with the following configuration. In each convolution block, there are two dilated convolution layers with dilation 3, followed by ReLU layer, batch normalization layer, and a Max-Pooling layer at the end. The number of filters for convolution layers in four convolution blocks are {16, 32, 64, 128} and the pooling window sizes are all Two additional convolutions with filter numbers (256, 256} follow after four convolution blocks. In each transposed convolution block, nearest-neighhor up-sampling (Odena et al. 2016) of its input is implemented, followed by dilated convolution layer with dilation and a batch normalization layer. The number of filters for convolution layers in four transposed convolution blocks are in the reverse order of the encoder {128, 64, 32, 16). The kernel size of convolution layers in both encoder and decoder is 7.
ReL. layer
C ADDITIONAL RESULTS
We report additional results to supplement our discussion for bi-LSTM architecture on the seg- mentation task in Sec. C.I and for the classification task in Sec.
C.I EVALUATION SEGMENTATION TASK: BI-LSTM
Bront
Gradient
Wiltle
Back
Inteorated Gradient
Signal
Smooth
Gradient
lazeled
Intograted Gradient
83
LRP
salleno
SmoothGGrd
LSP
SHAP
860
SHAP
0
50
100
150
time liel
(a) Localization
(b) Exemplary saliency maps
Figure 6: We show the localization metric results for all methods on the tool tracking dataset and with the bi-LSTM model. (a) shows the results for the localization metric. (b shows exemplary saliency maps.
The results for bi-LSTM on the segmentation task for tool tracking in Fig, show that none of the visual interpretation methods, that compatible with the model's architecture, produce satisfactory saliency maps.
C.2 CLASSIFICATION TASK
We show the variability of each visualization method with respect each dataset and metric cat- egory for the TCN in Fig. 7a It is noteworthy that some datasets are more problematic for the methods, especially for faithfulness and intra-class stability, than others. We also show that model has only a small influence. For this result, we aggregate all datasets and visualization meth- ods separately for FCN and TCN in Fig. 7c
In order to compare methods independent of datasets, control for their bias by normalizing the scores of all methods for each (category, dataset) so that their mean and variance is 1. With this,
correctly assess the "relative" performance of the visual interpretation methods across different datasets. The plot in Fig. 7b shows these relative (or marginal) scores that a method can achieve, compared other methods on the same datasets.
We prove the orthogonality of metrics in Figure g No combination of our metric scores shows a high correlation. This shows that each metric measures quality independent of all other metrics.
16

---

Under review as a conference paper at ICLR 2022
We show examples for saliency of the TCN architecture the FordB dataset for the visual interpre- tation methods for class in Figure lndd class in Figure I0 We argue that the choice of a suitable method to generate saliency maps should be guided by our framework.
(a) Separated datasets.
Grddat
(b) Relative scores that method can achieve compared to other methods on the same datasets.
8.5
16.5 6.
04
8.
00
0.2
4.8
18 0.8
(c) Results for different model architectures, aggregated datasets and visual interpretability methods. Model architecture has comparatively small influence on the performance, which speaks their generaliza- tion capabilities.
Figure 7: (a) shows scores separately over all {model, method combinations. removes the datasets' bias in order to assess the performance of individual visual interpretability metrics inde- pendent from datasets. (c) shows results for different model architectures, aggregated datasets and visual interpretability methods. Model architecture has a comparatively small influence on the performance, which speaks their generalization capabilities.
17

---

Under review as a conference paper at ICLR 2022
Du aod Baciproo
Surre
-5.0
-2.5
SubvMrees
10r Clos 500 B7
So
Serstbity
Figure 8: Pairplots of the correlations of metric scores with each other. No combination of metrics has a meaningful correlation with each other, proving that they provide independently useful signals. This analysis is based scores that were normalized for dataset bias, but the results also hold when this normalization not done.
18

---

Under review as a conference paper at ICLR 2022
Figure Examples of saliency maps for the TCN architecture on the FordB dataset for class
19

---

Under review as a conference paper at ICLR 2022
Figure 10: Examples of saliency maps for the TCN architecture on the FordB dataset for class
20


Reviews:
[{"review_id": "rhgtUVdI9D", "reviewer": "Reviewer_aJfy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written overall\n- Experiments are well designed where the six metrics can be compared for different tasks\n- The paper demonstrates why it is necessary to evaluate all six metrics for every visual interpretation\n- The paper compares 9 different interpretation methods\n- The paper shows how these metrics can vary across datasets and for different models\n\nWeaknesses:\n- The metrics are derived from existing literature\n- Figure 5 caption needs to be fixed\n", "summary_of_the_paper": "The paper proposes six quantitative metrics for evaluating post-hoc visual interpretation methods on time series. It demonstrates the efficacy of nine visual interpretation methods using these six metrics over some common neural network architectures. ", "summary_of_the_review": "The paper combines multiple interpretation approaches and evaluation metrics for those interpretation schemes. Thus, it demonstrates the need for multiple interpretation evaluation schemes to truly understand which ones are optimum. It also shows how these can vary across different datasets and models. So, although the paper is not novel, the experiments performed and the results obtained is very useful for researches looking for a way to perform post-hoc visualization. As a result, I consider it as a borderline paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YD4zE9gnz6V", "reviewer": "Reviewer_oySW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies deep neural network (DNN) explainability methods in the context of time-series data. Several metrics exist for evaluating the validity of DNN explainability methods on computer vision tasks. However, it is not clear whether these metrics are reliable for DNNs on time-series data. This paper conducts experiments to compare 6 different metrics on time-series classification and segmentation tasks. \n\n\nStrengths:\n1. The paper is generally well organized. (ALTHOUGH there are minor typos scattered all over so I urge the authors to proof read their work). \n2. The work is really well motivated. It is important to validate explainability methods before deploying them, and to that end it is important to design good validation criteria. \n3. I found the experiments and the accompanying figures to be really well designed and insightful, thank you! In particular I liked figures 1, 4, and 5. The appendix also has insightful experiments and figures!\n\nWeaknesses:\n1. Can you explain how Fig. 2 is obtained? I think the observation in this figure is interesting in its own right. It is worrying that 6 different methods can have very low correlations. \n2. I think the discussion in section 4.3 is hard to read and not very well organized. But I also understand that there are a lot of variables (data, task, model, explainability method, and metrics) which makes it difficult to fully cover all of them in text. (I am not penalizing the paper for this). \n\nGeneral questions (no need to run more experiments just wondering): \n1. Some metrics are exhibit high variability on particular datasets (e.g. Ford). I wonder what you think would happen if you tried transfer learning. For instance pretrain on a dataset where all metrics show relatively low variability, then finetune on FordA dataset?  ", "summary_of_the_paper": "This paper studies deep neural network (DNN) explainability methods in the context of time-series data. Several metrics exist for evaluating the validity of DNN explainability methods on computer vision tasks. However, it is not clear whether these metrics are reliable when applied to DNN explainability methods on time-series data. This paper conducts experiments to compare 6 of those metrics on time-series classification and segmentation tasks. ", "summary_of_the_review": "The paper is well motivated, the experiments are well designed, and the insights are helpful to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "BsxZCocyEuB", "reviewer": "Reviewer_cLNa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of this paper is that it provides a comprehensive evaluation of nine well-known interpretation methods for NNs with multiple architectures using multiple data sets. It is considered to be a very good survey paper to understand the characteristics of each method. The technical contribution of this study is mainly in the evaluation metric called localization, which is described in 3.6. This metric itself is based on a reasonable idea as an interpretive method. On the other hand, most of the considerations are derived from experimental results, which is a bit insufficient in terms of proposing new ideas based on deep technical insights.", "summary_of_the_paper": "This study deals with the issue of interpretability in the analysis of time series data using deep learning models. Six metrics for evaluating interpretation methods are introduced and nine existing interpretation methods are evaluated. One of these metrics is proposed by the author. One of the metrics is based on an experimental evaluation, which shows that none of the methods is superior in all the metrics.", "summary_of_the_review": "The paper is excellent as a survey paper focusing on performance evaluation, but somewhat inadequate as a research paper pursuing technical originality.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-s8oQjt-Hjx", "reviewer": "Reviewer_9Qic", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) **Unclear objective / no novelty or insight**: The objective of this paper is very unclear, and there does not seem to be any novelty in the methods it introduces. Majority of the saliency evaluation methods it uses are known in literature, and these evaluation methods do not seem to change for time series classification. The paper simply proposes to use these metrics to evaluate saliency methods. Now, it is understandable that different saliency metrics are not aligned, but the paper proposes no solution for this. If each saliency method has different strengths and weakness, how can one use this \"framework\" to guide selection of saliency methods? This lack of a solid objective within this paper makes it unsuitable for publication at this time.\n2) **Incorrect to use robustness of saliency map as a metric**: As a general remark on saliency metrics, please note that the only job of an evaluation metric should be to judge how well a saliency method reflects model behaviour. Metrics such as robustness or intra-class stability are thus contingent on such model behaviour. For example, if the underlying model is non-robust, there is no reason to expect robustness from saliency methods, and if a saliency method is indeed robust in such scenarios it does not reflect model behaviour and thus can be misleading. Thus usage of these metrics for evaluating saliency maps must be avoided.\n3) **Incorrect to use localization of saliency map as a metric**: A similar critique holds for the localisation metric (including the pointing game and other metrics in literature). The implicit assumption here is that model holds the same parts of the input to be important as judged by a human evaluator. However models can be erroneous and can sometimes have spurious correlations and might assign a different portion of the input to be important. In such a case, we would want the saliency map to reflect modelling errors, which would necessarily perform poorly in the localisation metric. Hence localization is a not a good saliency evaluation metric.", "summary_of_the_paper": "This paper proposes to use six evaluation metrics to evaluate saliency methods for time series classification. The experiments perform this evaluation for segmentation tasks, and find that few metrics achieve high scores with low variance.", "summary_of_the_review": "The paper does not present either a novel method or any novel insight. For this reason, I would recommend rejection.\n\nThe post-rebuttal discussions did not change my view of the paper, but I am improving my score in light of author updates.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper evaluates interpretation methods of neural networks on time series data. The reviewers find some values in this work, but were also consistently concerned with the main theme and novelty of this work. The authors have actively responded to reviewer comments, but the reviewers were not convinced with the major contributions and novelty. Thus the work is not ready for ICLR.
Decision: Reject